This is how Apple could turn the iPhone into a 3D controller

For VR -Games should apparently serve the iPhone as an input device. Another patent in this area uses the smartphone as a display for a mixed reality headset.

In the last few days the US patent office has again granted the iPhone manufacturer a number of patents in the field of virtual reality (VR). The iPhone plays a key role in two documents: on the one hand as a display, on the other hand as a controller. According to rumors, Apple plans to enter this market with its first AR / VR headset in 2022. It is currently dominated by devices from the Facebook subsidiary Oculus and smartphone manufacturers such as HTC (Vive Pro) and Samsung as well as Sony with a proprietary solution for the Playstation.

The telephones should serve as an interface to the VR environment. Apple is thus giving customers an input device that they already know. It also saves them having to buy and learn new devices. So far, the problem has been that VR glasses cannot calculate the position of cell phones as precisely. This is where the patent comes in and lists several implementations. In one case, a marking on the physical display of the iPhone provides orientation for the VR glasses. The system uses this marker to calculate and track the position in the virtual space. It also recognizes the rotation of the “controller”. The patent says that the control signal enables “the second device” – meaning the iPhone – to be used as a three-dimensional control, a 3D pointer or a user interface input device.

It is the 16th patent in which Apple engineers describe a VR headset that can accommodate an iPhone. They call this a “display device with an optical combiner”. You mention that the device can also be mounted on the head. The combination is intended to make computer-generated and real images viewable at the same time and belongs to the area of ​​mixed reality. According to the text, the optical combiner also shows the real environment – so it looks like glass glasses, not like a display that only shows images of the real environment. A reflector should also bring the synthetic models of the cell phone into the picture. Apple lists several ways to achieve this. In one arrangement, an optical component overlaps the front camera and serves as a partial mirror. Another works with different layers that are applied to the display and achieve the desired effects. The patent also mentions various sensors and cameras that scan the environment and record the direction in which the user is looking. As always, you don’t know whether Apple will even consider these patents in future product development.

You might also be interested in