Whether driving at high speeds on the German Autobahn or on California’s Highway 1 as it twists and turns along the Big Sur coast, drivers cannot afford to take their hands off the wheel nor take their eyes off the road. Distracted driving is out of the question, which is exactly why auto manufactures are designing in features to keep the driver’s eyes on the road and not on that large center console display.
Prominent human-machine interface (HMI) technologies include heads up displays, improved speech recognition, and gesture control. Applications currently using simple swiping hand gesture control include changing the radio channel, answering a phone call, playing the next song on your playlist, menu scrolling, navigation screen zooming, sunroof open and close, and overhead dome light control. Gesture control and the sensors that enable the feature are available now mainly in high end cars.
A market study conducted in 2018 by Global Market Insights forecasts the automotive gesture recognition market size to grow from $1.1 million in 2017 to $13.6 billion by 2024. Hand gesture recognition market share is projected to be greater than $8.6 billion by 2024. Compounded annual growth between 2018 and 2024 for both America and Europe is projected to be greater than 40%. The report states that safety regulations defined by various governments is improving the product penetration owing to advantages offered such as compatibility of vehicle’s electronic systems and improved maneuvering control. However, prohibitive cost, complexity, and integration may challenge the automotive gesture recognition market from 2017 to 2024.
Optical Gesture Systems
Optically based sensors like Vishay’s VCNL4035 are used for detecting swiping hand motions. Basically, an integrated proximity sensor with the ability to pulse up to three emitters in quick succession, one after the other, is combined with externally connected infrared emitters to create a zone in which a gesture can be performed.
For simple left and right swipe and zoom in and out motions, only two emitters are needed. In figure 1, the internal LED driver will drive the left emitter during which a proximity reading is made. This value is converted into a digital count, a number between 0 and ~65k as the analog to digital converter has a 16-bit resolution. The multiplexer will then switch from the output connected to the left emitter to the one connected to the right emitter, in turn drive the right emitter, while taking another proximity reading. Both readings are saved in separate data registers within the sensor, so that they can be read by a host microcontroller individually. The microcontroller will then pick up these measurements and compare the two readings.
If the hand is closer to the left emitter the output count of the sensor will initially be higher for the left reading versus the right reading. As the hand continues with its right swipe, the output counts from the sensor will become approximately equal as the hand reaches the center between the two emitters, and then the output associated with the right emitter will be higher than the left as the hand completes the left-to-right swipe.
This scenario could be rotated ninety degrees to enable up and down gesture control. The zooming function is enabled by measuring the output of the sensor as it is moved closer or further away from the sensor. These algorithms are predicated on having a threshold proximity value that is exceeded by the initial presence of the hand which should be higher than the offset that the sensor reads when nothing is in front of it. There are more robust and complex algorithms where the signal of each emitters is recorded in a continuous stream, and a measurement frame is overlaid such that the shape of the signal can be analyzed.
The percentage overlay of one emitter’s signal about the other can then be used to denote a time difference, which shows that a gesture has been made. Adding a third emitter will improve the resolution of the simple gestures described above and could possibly be used to recognize more complicated gestures. There are lots of variables to account for: size of object if not a hand, distance from object to sensor and emitter, speed of the moving object, reflectivity of moving object, distance between emitters, thresholds levels to avoid false detects and accounting for ambient lighting and possible optical disturbances.
The flexibility of placing discrete emitters is important to design engineers as this is used to control the size of the gesture and the sensitivity zone. Some sensors integrate emitters within a single package which eliminates this flexibility. The typical sensing range of an infrared emitter and sensor systems is 20 centimeters or 8 inches.
While the previous system used infrared emitters and light-to-digital proximity sensors, other systems offer millimeter granularity by combining a time-of-flight sensor with a 3D camera. This is clearly a more expensive system, but it does allow the measurement of ‘small’ finger gestures versus large swiping motions. As first-generation applications have launched in vehicles utilizing large motions, second generation systems are looking to identify multiple finger gestures. Imagine the motion your fingers make when enlarging a photo on a smartphone screen. Automotive manufacturers want to recognize this gesture in the air. This will necessarily require 3D camera systems.
As Murray Slovick recently pointed out, this capability is like the player motion detection used in game systems. For example, Microsoft’s Kinect system for the Xbox game console detects motion from distances up to approximately 10 feet. Gaming systems track the entire body motion of the player, but their automotive counterparts will aspire only to track the driver’s hand gestures. The more complex the system for measuring gestures becomes, the more complex the middleware becomes to convert the output of the sensor into a definable action.
Long-term Outlook
Companies like LG and Sony are joining more typical Tier-1 automotive manufacturers like Delphi Automotive PLC, Harman International, Continental AG, and BHTC in developing automotive gesture recognition systems. Customer comfort, multimedia, navigation and infotainment applications are taking on a larger priority which dovetails with development of driver assist and driverless cars. When and if that becomes a reality, avoiding driver distractions won’t be an issue because only passengers will be in the vehicle.
Further reading: Gesture Recognition, Proximity Sensors Drive Advances in Automotive Infotainment
About the author
Jim Toal is Senior Manager, Product Marketing for Vishay Optoelectronics. Vishay is a major supplier of infrared emitters, photo detectors and optical sensors to automotive, consumer and industrial manufacturers.