In February this year, reports suggested that Apple had completed the production tests for its AR/VR headsets, which are expected to be launched by the end of 2022, or early 2023. Apple has been actively hiring for the Mixed Reality Headset team. Now, with 2023 just around the corner, we can expect Apple to get mass production running and make an announcement soon.
The headsets are expected to be branded as “Reality One”, “Reality Pro”, and “Reality Processor”, based on the new trademark filings by Apple for ‘RealityOS’ earlier this year, suggesting their entry into a new product category.
Apple CEO Tim Cook in discussing the role of AR technology in the future said, “I think AR is a profound technology that will affect everything.” Cook also had reservations about the excessive focus on the term metaverse stating that he is not sure if an average person can really say what the term metaverse is. It makes sense why Apple has been away from the VR game altogether, which is what metaverse is—a virtual place (separate from reality) where users can create, interact, develop and reside with each other.
We find Robert Scoble, chief strategy officer at Infinite Retina, saying, “Apple is about to kill Zuckerberg’s VR toys.” The reason for this, he believes, is that, unlike Meta, Apple is equipped with an AI inferencing engine. Here is what Scoble said referring to the potential Apple has because of having its own AI inference engine.
Motion Capture
The motion capture technology captures the movement of a person in front of the camera in real time. Movement is measured as a series of joints and bones, keeping track of body position and movement which can again act as an input to the AR experience, which will be indicated later in the article. In some additional improvements, motion capture also tracks left and right ears, improving the overall body pose detection.
Additionally, it also detects up to 100 images at a time, and automatically gets an estimation of the physical size of the object in the image.
The body tracking feature estimates leg movement even when the user is fully clothed and carries high value for applications like fitness, training, game development, and much more. The technology also enables people occlusion where AR content is placed in front and behind people in the real world, paving the way for an immersive experience which facilitates a green screen-type effect in all environments in the real world.
Face tracking
Apple’s face tracking system is already in the works, and currently offers five different AR visualisations which users could switch between. It includes tracking the face through its ARKit coordinating system which overlays x/y/z axes onto the face, attaching or interacting with 3D virtual content placed on the user’s real face, and animating facial expression using ARKit’s animation blend shape.
ARKit 6, for instance, also improves interactivity between the user and the world by simultaneously accessing the front camera, and AR content in the back camera. Hrafan Thorrison, co-founder & CEO of Aldin, believes that compared to other platforms, Apple’s avatars are well received, and combined with the motion capture system, we can expect these avatars to have legs, when overlaid on real user faces.
Scene geometry
The Scene Geometry tool, which Thorisson deems to be “years ahead of the leading VR platform holders”, creates a topographical mapping of any space, for example your room, automatically identifying the objects inside, labelling and measuring their dimensions. Thus, it will automatically categorise the objects as ‘sofa’, ‘chair’, ‘refrigerator’, ‘window’, etc. thereby unlocking various potential such as object occlusion, by which virtual objects can be integrated with ease in any physical environment. Apple has been able to leverage its AR technology to apply it to a variety of applications such as e-commerce, gaming, architecture, and much more.
3D object capture
In this year’s WWDC event, Apple hinted towards some of the developments in the Object Capture front. Object Capture, initially launched as RealityKit API on macOS, allows the conversion of an image of a real-world object taken using an iPhone, iPad, or a DSLR, into a 3D model. Using the Photogrammetry API, the input image can be transformed into a 3D model that includes both geometric mesh as well as material maps, including textures, automatically applied to the model.
The combined efforts of LiDAR scanner and 3D mesh data generated by Scene Geometry takes in depth information about the surrounding environment and makes virtual object occlusion even more realistic, allowing for seamless placement of virtual objects in a physical environment.
Apple had already mentioned several use cases of Object Capture, especially in e-commerce where users, for instance, can try out the fit of shoes, or a preview of how a certain plant or furniture will look in their living space.
Final thoughts
Apple’s USP has been the end-to-end development of the product, which includes interoperability between its various devices—AR technology has been one such offering. Apple, since 2020, has been working on a private federated learning model, which allows for on-device machine learning. The AI algorithm runs on the device locally by analysing the data stored on the device, instead of an AI training model run on data collected in a centralised system.
Apple’s strategy has been to play the long game well, and they are certainly about to make its ecosystem more robust with the inclusion of AR headsets, and future devices running on AR.
- OpenAI Says AGI is Here. But Not Everyone is Buying It
- Why Oracle is Bringing Post-Quantum Security to Java
- What Really Happens When You Turn Your Selfie Into a 1980s AI Pic?
- AI Agents Are Making Data Security A Bigger Enterprise Problem
- OpenAI is Getting Nervous About Reinforcement Learning
- Meta’s Bold Open-Weight Bet is Meant to Muse Developers