The debate over passthrough camera access is creating quite a stir in the XR community nowadays. While we have clear strategies from Meta, Apple, and Pico, the mystery remains about Google’s approach to Android XR. After having a direct chat with Google, I can reveal they plan to offer a similar solution to what we see on smartphones. Read on to find out more!
The camera access dilemma
Before I dive deep, let’s backtrack a bit for those not completely familiar with the subject. Recent standalone VR headsets have evolved into MR headsets, offering users a color passthrough view of their surroundings via front cameras. This feature is pivotal for supporting engaging mixed reality applications like Cubism, Starship Home, and Pencil, among others.
The operating system relies on camera frames to showcase passthrough visuals. As developers, having access to these frames would allow us to analyze the user’s environment using AI and computer vision, enhancing their reality significantly. In a previous piece, I voiced my belief that true mixed reality needs camera access. This capability can make applications smarter and more aware of their surroundings. Using a workaround for camera access on Quest, I developed a prototype AI+MR application that aids in home interior design—a feat impossible without such access.
This all sounds exciting, but it’s not without its complications. The significant issue here is privacy. Should a rogue developer manage to access camera frames, they could potentially harvest images of users’ environments surreptitiously, identifying personal data from national IDs or bank cards lying around. Moreover, it could lead to unauthorized captures of faces or personal appearances.
It’s a balancing act: protecting user privacy while unlocking the power of mixed reality.
The behavior of the XR companies
Initially, full camera access was available with little regard for privacy concerns. For those who’ve followed me for some time, you might recall the experiments with camera textures on the Vive Focus by my team at NTW around 2019. Our projects spanned from diminished reality and Aruco marker tracking to sound reactivity.
As mixed reality gained traction, companies opted for caution, denying developers access to camera frames to avoid potential privacy violations—Meta, Pico, HTC, Apple, all included. This restriction became the norm until XR developers, recognizing the necessity of this feature, began urging manufacturers to grant camera access. Advocates like Cix Liv, Michael Gschwandtner, and myself led the charge, proposing a transparent method for accessing frames, contingent on user agreement, to implement features like object recognition.
The comparison to smartphones arose, arguing why XR devices couldn’t offer similar camera permissions as mobile phones do. This push catalyzed movement, with Meta promising the release of its “Passthrough API” earlier this year. This prompts the question: What stance will Google’s Android XR take?
Android XR to treat the headset like a phone
Android rules the roost as the operating system for most smartphones globally. For Android app developers, obtaining camera streams is as simple as a user permission request. Once granted, developers can access camera feeds by specifying the camera ID, with 0 typically indicating the rear camera. This straightforward process allows developers to harness camera frames as needed, using existing APIs.
Following trends, Google intends to make Android XR compatible with standard Android apps. Although previously unverified, I reached out to a Google representative, receiving confirmation of their approach to camera access for Android XR:
"Similar to any Android app, a developer can use existing camera frames with user permission for XR…"
Developers can use the conventional camera APIs, Camera2 and CameraX, to request access to standard camera streams. To access the world-facing camera feed (akin to a phone’s rear camera), applications need the same sort of permissions they use on phones. However, for the front-facing or selfie-camera stream, developers receive an avatar image stream generated by apps on the headset using OpenXR APIs for data like head and eye movements.
This arrangement lets Android developers utilize familiar classes to manage camera streams on Android XR headsets, just as they do with smartphones. Since these classes facilitate video capturing and machine learning processes, this should extend seamlessly to XR headsets and glasses. It’s indeed great news!
Yet, while access to the world-facing camera is straightforward, the selfie-cam is more about offering a reconstructed avatar, similar to Apple’s Vision Pro vision. This design balances Android XR functionality with Android phone operations, providing avatars for user-facing streams matching the “selfie camera” setup.
Google’s strategy ensures compatibility for Android apps on Android XR, a wise choice to maintain coherence in camera permissions across device types.
However, some questions linger. Firstly, raw camera stream access remains restricted:
"At the moment, we’re not providing a path for applications to access non-standard sensor data."
This suggests non-traditional camera streams, other than the front or back cameras, are still unavailable. There’s hope for potential openness in the future, particularly for enterprise-level users.
Regarding Unity development, Camera2 and CameraX are native Android classes. Unity developers could attempt using the WebcamTexture class to manage camera frames. If issues arise, a workaround might involve creating a native wrapper for the CameraX functionality in Unity.
A little caveat about Android XR
Bear in mind, Android XR is still in a preview phase, and no official headsets are yet released. This means changes could occur before its official launch—something to keep in mind, although unlikely.
The opening up of camera access
With both Google and Meta easing restrictions on camera access, it seems likely other companies will follow. This makes 2025 poised to be a breakthrough year for mixed reality innovation. I’m eagerly anticipating the incredible creations from the developer community!
(Header image inspired by a Samsung image)
Disclosure: This blog is funded through advertisements and affiliate links. Clicking an affiliate link means I may earn a small commission from your purchase. You can view my comprehensive disclosure here.
