Infrared virtual reality headset

Context

Lynred's application lab looks for new, sometimes playful uses of infrared: introduce the technology to a wide audience and show how easily its sensors integrate. Data Science Experts worked with Lynred on this project through one of its engineers: me, from December 2023 to June 2024.

The goal: a virtual reality headset that shows the world in thermal, in real time. The main constraint is latency: the slightest lag between head movement and image brings nausea and dizziness. The experience had to feel as smooth as a commercial VR headset.

Architecture

Three parts: Lynred's ATI320 camera (320 × 240 pixels, 7 g without its housing, shutterless, so it can film continuously), an NVIDIA Jetson Nano board for computing, and an Android smartphone inside the headset for display.

As soon as it is powered, the Jetson opens a USB tethering link with the phone: the board has neither Wi-Fi nor Bluetooth, and this isolated local link gets a new IP address on every connection. A Docker container then starts a multithreaded Flask server: the app fetches frames with GET requests and sends its settings with POST. A first approach, the Android Debug Bridge (ADB), was dropped: it could not keep up in real time.

Image processing

Frames arrive continuously through Lynred's SDK and are processed with OpenCV. Correction first relied on a custom chain; to gain speed, it now uses the camera's built-in shutterless correction. A colour map is then applied (greyscale by default, changeable from the app).

With a single camera, there is no true depth: I built a fake stereoscopy. The frame is duplicated, each copy is cropped by 40 pixels on opposite sides, then both views are placed side by side, which is enough to give a sense of depth inside the headset.

Optionally, human detection draws boxes around people in the image: a MobileNetV2 model trained on infrared pedestrians, run by the DetectNet module of jetson-inference, optimised for the board. To keep things smooth, detection runs on one frame in five, and the boxes are reused in between.

The mobile app

The app is written in React Native. After a first 2D prototype, it was rebuilt in 3D with ViroReact: once the phone is in the headset, the touch screen is out of reach, so everything is driven with the head, using the gyroscope and a reticle that hovers and selects.

It finds the Jetson on its own by scanning the local network, welcomes the user in a 3D space with videos of industrial infrared applications, shows the stereo stream, and offers settings applied live on the board: spacing between the two images, colour map, human detection, camera calibration.

A second screen for trade shows

For demos, an HTML page served by Flask shows the infrared stream on a large screen plugged into the Jetson, alongside the headset, or on its own when no phone is connected. A script detects the screen at start-up, then keeps checking for it.

The headset, with three students

In parallel, I was project lead for three ENSE3 students (Grenoble INP), Louis-Stéphane Assi, Jocelyn Ribes and Louis Métay-Peault, in charge of the mechanical design. The brief: survive a fall, protect the camera, run about four hours on a swappable battery, and fit every face.

Starting from an open-source headset adapted in SolidWorks, they designed a ventilated housing for the camera, held by brackets, and a mount for the battery and the board.

With 702 g of components, when a complete commercial headset weighs about 600 g, wearing tests settled between three positions for the battery and board block (447 g): top, side or back. The top won, for weight distribution and cabling.

The parts were printed in ABS and HIPS on Zortrax printers. Printing was the project's main hurdle: poor precision, parts lifting off the bed, parts broken and reprinted up to three times, then filed and drilled by hand.

For battery life, a Python script logged the board's power and temperature live: the 10,000 mAh battery lasted 3 h 47 min, at steady power, 55 °C on average and 60 °C at most, below the materials' 80 °C deflection point.

Outcome

Working software that shows infrared in the headset in real time with minimal latency, and a first complete mechanical prototype. The technical report delivered in June 2024 sets out what comes next: a dedicated display instead of the phone, true stereoscopy with two cameras, mini-games that play with thermal vision, a fully autonomous headset and user testing.