The US Patent & Trademark Office has published a patent application from Apple related to developing a mixed reality headset that uses advanced RGB-D sensors to produce accurate 3D images for virtual reality, augmented reality and mixed reality applications. The patent covers devices, systems, and methods that implement simultaneous localization and mapping for RGB-D sensors, such as RGB-D cameras.
One possible implementation involves the device receiving multiple frames of a real-world scene within a field of view of a camera at multiple times, the frames including color values and depth values for pixels for the field of view of the camera. The device selects keyframes from the multiple frames of the real-world scene within the field of view of the camera. The keyframes are associated with camera poses defined in a three-dimensional (3D) coordinate system.
The device receives a current frame of the real-world scene currently within the field of view of the camera. The current frame includes current color values and current depth values for the pixels for the field of view of the camera at a current time. The device determines a current camera pose of the camera in the 3D coordinate system based on the current frame. The device then provides a virtual representation of the current frame based on the current camera pose of the camera and two or more of the keyframes.
RELATED: Apple patent places biometric sensors in earbuds
The virtual representation is provided based on the color values and the depth values of the two or more of the keyframes.
Apple’s use of RBG-D sensors is not entirely new. Those sensors have been used in Microsoft's Kinect, which in the early stages used PrimeSense technology now owned by Apple. The Asus Xtion Pro Live uses RBG-D sensors as well.