Prophesee, a provider of event-based neuromorphic vision sensor technology, announced a partnership with Qualcomm Technologies at the massive Mobile World Congress event in Barcelona, Spain, this week, a collaboration that will see Prophesee’s Metavision sensors optimized for use with Qualcomm’s Snapdragon platform to bring neuromorphic-enabled image capture capabilities to mobile devices.
Event-based neuromorphic vision sensor technology could be a game changer for camera performance, with neuromorphic capabilities processing movement and moments closer to the way the human brain processes them–with less blurring and more clarity than a frame-based camera can manage. Specifically, the technology allows cameras to perform better while capturing fast movements and scenes in low lighting in their photos and videos. For the most part, the consumer marketplace is still waiting to get their hands on devices with these capabilities, but the new partnership means that wait is growing shorter.
Later this year, the Prophesee plans to release a development kit to support the integration of the Metavision sensor technology for use with devices that contain next-generation Snapdragon platforms. After that, it will not be too much longer before consumers can experience the benefits of event-based neuromorphic vision sensor technology themselves.
Luca Verre, co-founder and CEO of Prophesee, told Fierce Electronics, “We expect phones with this feature/capability to be in the market by 2024. It will likely appear in ‘flagship’ models first.”
When the technology arrives, it is not expected to replace traditional frame-based sensors, but instead work in tandem with them, with much of the camera performance improvement coming through Prophesee’s event-based continuous and asynchronous pixel sensing approach which will help in the “de-blurring of images” and the highlighting of focal points that otherwise might become lost where low lighting intrudes on a captured moment.
That could make consumers much happier about the quality of the pictures they take on their mobile devices, although there is a good chance they may not even know they will have neuromorphic sensors to thank for the improvement, as they probably will not have to think about switching into a different photo capture mode to take advantage of neuromorphic sensing.
“It’s unlikely that smartphones would have a ‘neuromorphic mode,’ but instead would work seamlessly with the existing image capture capabilities - but, in theory, that could be something the OEM could consider,” Verre said. “Note that using an event based camera actually reduces the amount of data processed, capturing only things in a scene that move, which are often ‘invisible’ to traditional cameras, so it is largely an augmentation of traditional frame based cameras (and other sensors, such as lidar in a car), not a replacement, especially in consumer applications.”
These sensors already are used in other kinds of applications, including business and industrial use cases such as security cameras, surveillance, preventative maintenance, vibration monitoring, high speed counting, and others where event cameras can work “as a standalone machine vision option,” Verre said, adding, “There is vast potential in the idea of sensor fusion, combining event-based sensors with other types of sensors, like frame-based cameras.”
The Qualcomm partnership comes almost a year after Prophesee announced a partnership with Sony that revolved around enabling improved integration of event-based sensing technology into devices, and Verre said the migration of the technology to mobile phones likely will be smoother as a result of the earlier partnership. Working with Sony, a leading CMOS sensor provider for the mobile industry, helped make the sensors “more applicable for mobile (smaller size, lower power, etc.) with 3D stacking manufacturing techniques,” he said.
Prophesee also sees the technology as having potential in other mobile and wearable device applications, such as augmented reality headsets. Prophesee already is talking to OEMs about moving in that direction, and Verre said the company believes that “by enabling a paradigm shift in sensing modalities with this approach, there are countless applications that can benefit.”