Virtual and augmented reality live or die on frame rate. Miss the headset's refresh and the wearer gets sick, so every millisecond you spend drawing the left eye is a millisecond you do not have for the right. We have been putting real 3D engines, not canned demos, into headsets since the Oculus DK2, and underneath the changing hardware the problem stays the same: draw a large, real scene twice or more, fast enough that nobody notices the machine straining.
At I/ITSEC 2015 we demonstrated OpenSceneGraph and osgEarth running on an early Oculus DK2, real terrain and a real scene graph in a headset at a time when most VR was still spinning cubes.
Rendering two eyes without dropping frames
A stereo headset renders the scene at least twice, once per eye, and modern devices push that further with more views and higher refresh rates. Doing that the naive way, one eye after the other, wastes the traversal. We wrote a Multiview extension for OpenSceneGraph that renders the views in parallel: the scene graph is traversed once and the eyes are drawn together. On headsets with eye tracking we have done foveated rendering, spending full resolution only where the eye is actually looking and dropping detail in the periphery, which is where the frame-time budget on a device like the Varjo actually comes from.
(Some of the) Headsets we have worked on
Oculus DK2 through the consumer Rift, the HTC Vive (we worked on the MakeVR port to it), Varjo's high-resolution eye-tracked headsets, and a defense program on the Meta Quest 2 and Quest Pro, plus the Microsoft HoloLens 2, which we reached by porting OpenSceneGraph to it through ANGLE for the U.S. Army's IVAS program. The device changes every couple of years. The job of feeding it a real scene at full rate does not.
Virtual data in, not just pixels out
XR is an input problem as much as a display problem, and our history there runs back to the 1990s, when we built a serial interface for the Nintendo Power Glove using a PIC microcontroller. Since then we have integrated glove and spatial controllers with OpenSceneGraph, including the P5 Data Glove and the Razer Hydra, so a tracked hand or wand drives the same scene graph that renders the view.
Augmented reality in the real world
We worked on a maritime augmented reality trainer for Rockwell Collins, a land-based system that uses AR headsets and greenscreen to overlay navigation and situational data the way a crew would see it at sea, over top of a fully functional simulated tangible control console. We do airborne AR too: for Precog's Skyline, an iPad glass-cockpit device, we drew synthetic-vision terrain over the pilot's view and integrated a Stratux ADS-B and GPS receiver to capture the aircraft's pose accurately for the overlay, work that sits squarely between mobile and safety-critical avionics.
Mobile, autostereo, and holographic
Our XR work goes back to phone-based viewers: Google Cardboard and the viewmaster-style holders that turned a smartphone into a stereo display, including the VR app we built to present a special VR movie tied to the film Allegiant (our part was the app, not the film). We have also gone past the headset entirely. We have built custom multiheaded rendering farms, banks of multi-output graphics cards, to drive horizontal-segmented, lenticular-style autostereoscopic and holographic displays, and we have worked with glasses-free devices such as the Leia Lumepad tablet.
How it connects
XR is where several of our disciplines meet. The engine and the Multiview rendering are real-time 3D graphics. The training and mission content is visual simulation. The tracking that keeps augmented reality glued to the world is computer vision. One team that has built all of it is worth more than three vendors who each own a slice.
Who works on it
AlphaPixel is a US-owned small business, founded in 2004, with senior developers who each have 25-plus years in real-time 3D. DLA DD2345 / ITAR registered. We take closed-source defense work and open source alike, and we do not sell a headset, so our advice is not a sales pitch for one.
Frequently asked questions
Which headsets and XR platforms do you develop for?
We have shipped real 3D engines into headsets since the Oculus DK2, through the consumer Rift, the HTC Vive, Varjo's high-resolution eye-tracked headsets, the Meta Quest 2 and Quest Pro, and the Microsoft HoloLens 2, which we reached by porting OpenSceneGraph to it through ANGLE. We also work past the headset, on phone-based viewers and glasses-free autostereoscopic and holographic displays.
How do you keep VR frame rates high enough to avoid motion sickness?
A stereo headset draws the scene at least twice, so wasted work costs you double. We wrote a Multiview extension for OpenSceneGraph that traverses the scene once and renders both (or four views with foveated) eyes together, and on eye-tracked devices we use foveated rendering to spend full resolution only where the eye is looking. The goal is to draw a large, real scene fast enough that nobody notices the machine straining.
Can you put my existing 3D engine or scene graph into a headset?
Yes, that is the core of our XR work. We put real terrain and real scene graphs into headsets, including OpenSceneGraph and osgEarth, rather than building canned demos.
Do you build augmented reality that stays locked to the real world?
Yes. We worked on a maritime AR bridge trainer project that overlays navigation and situational data over a simulated console, and an airborne iPad glass-cockpit device that draws synthetic-vision terrain over the pilot's view, locked to aircraft pose with an ADS-B and GPS receiver. Keeping the overlay glued to the world is a computer-vision and tracking problem as much as a rendering one.
Do you work on autostereoscopic or holographic (glasses-free) displays?
Yes. We have built custom multiheaded rendering farms, banks of multi-output graphics cards, to drive horizontal-segmented lenticular autostereoscopic and holographic displays, and worked with glasses-free devices such as the Leia Lumepad.
Do you handle input and tracking, not just rendering?
Yes. XR is an input problem as much as a display one. Our history there runs all the way back to a serial Nintendo Power Glove interface in the 1990s, and since then we have wired glove and spatial controllers, including the P5 Data Glove and Razer Hydra, into the same scene graph that renders the view. Modern XR systems usually have controllers integrated into them, and even have equipment free hand tracking, which we've worked with.
Do you take defense or closed-source XR work?
Yes. AlphaPixel is a US-owned small business, DLA DD2345 and ITAR registered, and we take closed-source defense work and open source alike.
How is working with you different from a VR studio?
We are graphics and simulation engineers, not a content studio, and we do not sell a headset. We handle the engine, the multiview rendering, the tracking, and the mission content together, so the parts that usually fall between separate vendors are one team's responsibility.
Talk to us
Tell us what you’re building and where it’s stuck. We’ll tell you straight whether it’s something we can help with, and how we’d approach it.
Or see how we run consulting and software development engagements.
