Gesture Control in Automotive Infotainment

Gesture Control in Automotive Infotainment

Sensors Insight by Jim Toal

Whether driving at high speeds on the German Autobahn or on California’s Highway 1 as it twists and turns along the Big Sur coast, drivers cannot afford to take their hands off the wheel nor take their eyes off the road.  Distracted driving is out of the question, which is exactly why auto manufactures are designing in features to keep the driver’s eyes on the road and not on that large center console display. 

Prominent human-machine interface (HMI) technologies include heads up displays, improved speech recognition, and gesture control.  Applications currently using simple swiping hand gesture control include changing the radio channel, answering a phone call, playing the next song on your playlist, menu scrolling, navigation screen zooming, sunroof open and close, and overhead dome light control. Gesture control and the sensors that enable the feature are available now mainly in high end cars. 

A market study conducted in 2018 by Global Market Insights forecasts the automotive gesture recognition market size to grow from $1.1 million in 2017 to $13.6 billion by 2024.  Hand gesture recognition market share is projected to be greater than $8.6 billion by 2024.  Compounded annual growth between 2018 and 2024 for both America and Europe is projected to be greater than 40%.  The report states that safety regulations defined by various governments is improving the product penetration owing to advantages offered such as compatibility of vehicle’s electronic systems and improved maneuvering control. However, prohibitive cost, complexity, and integration may challenge the automotive gesture recognition market from 2017 to 2024.

Sponsored by Infosys

In Conversation with Antonio Neri, President & CEO – Hewlett Packard Enterprise & Salil Parekh, CEO – Infosys

Hear the CEOs of Infosys & HPE discuss the current crisis and how it has accelerated the need for digital transformation for their clients. Connectivity at the Edge, right mix of hybrid cloud, ability to extract data faster than ever before… these are just some of the contributions that HPE and Infosys make to our clients’ digital transformation journey.


Optical Gesture Systems

Optically based sensors like Vishay’s VCNL4035 are used for detecting swiping hand motions.  Basically, an integrated proximity sensor with the ability to pulse up to three emitters in quick succession, one after the other, is combined with externally connected infrared emitters to create a zone in which a gesture can be performed. 

For simple left and right swipe and zoom in and out motions, only two emitters are needed.  In figure 1, the internal LED driver will drive the left emitter during which a proximity reading is made. This value is converted into a digital count, a number between 0 and ~65k as the analog to digital converter has a 16-bit resolution.  The multiplexer will then switch from the output connected to the left emitter to the one connected to the right emitter, in turn drive the right emitter, while taking another proximity reading. Both readings are saved in separate data registers within the sensor, so that they can be read by a host microcontroller individually. The microcontroller will then pick up these measurements and compare the two readings. 

Figure 1

If the hand is closer to the left emitter the output count of the sensor will initially be higher for the left reading versus the right reading.  As the hand continues with its right swipe, the output counts from the sensor will become approximately equal as the hand reaches the center between the two emitters, and then the output associated with the right emitter will be higher than the left as the hand completes the left-to-right swipe. 

This scenario could be rotated ninety degrees to enable up and down gesture control.  The zooming function is enabled by measuring the output of the sensor as it is moved closer or further away from the sensor.  These algorithms are predicated on having a threshold proximity value that is exceeded by the initial presence of the hand which should be higher than the offset that the sensor reads when nothing is in front of it.  There are more robust and complex algorithms where the signal of each emitters is recorded in a continuous stream, and a measurement frame is overlaid such that the shape of the signal can be analyzed. 

The percentage overlay of one emitter’s signal about the other can then be used to denote a time difference, which shows that a gesture has been made.  Adding a third emitter will improve the resolution of the simple gestures described above and could possibly be used to recognize more complicated gestures. There are lots of variables to account for:  size of object if not a hand, distance from object to sensor and emitter, speed of the moving object, reflectivity of moving object, distance between emitters, thresholds levels to avoid false detects and accounting for ambient lighting and possible optical disturbances. 

The flexibility of placing discrete emitters is important to design engineers as this is used to control the size of the gesture and the sensitivity zone.  Some sensors integrate emitters within a single package which eliminates this flexibility.  The typical sensing range of an infrared emitter and sensor systems is 20 centimeters or 8 inches. 

While the previous system used infrared emitters and light-to-digital proximity sensors, other systems offer millimeter granularity by combining a time-of-flight sensor with a 3D camera.  This is clearly a more expensive system, but it does allow the measurement of ‘small’ finger gestures versus large swiping motions.  As first-generation applications have launched in vehicles utilizing large motions, second generation systems are looking to identify multiple finger gestures.  Imagine the motion your fingers make when enlarging a photo on a smartphone screen.  Automotive manufacturers want to recognize this gesture in the air.  This will necessarily require 3D camera systems.

As Murray Slovick recently pointed out, this capability is like the player motion detection used in game systems. For example, Microsoft’s Kinect system for the Xbox game console detects motion from distances up to approximately 10 feet. Gaming systems track the entire body motion of the player, but their automotive counterparts will aspire only to track the driver’s hand gestures. The more complex the system for measuring gestures becomes, the more complex the middleware becomes to convert the output of the sensor into a definable action.


Long-term Outlook

Companies like LG and Sony are joining more typical Tier-1 automotive manufacturers like Delphi Automotive PLC, Harman International, Continental AG, and BHTC in developing automotive gesture recognition systems.  Customer comfort, multimedia, navigation and infotainment applications are taking on a larger priority which dovetails with development of driver assist and driverless cars.  When and if that becomes a reality, avoiding driver distractions won’t be an issue because only passengers will be in the vehicle.

Further reading: Gesture Recognition, Proximity Sensors Drive Advances in Automotive Infotainment


About the author

Jim Toal is Senior Manager, Product Marketing for Vishay Optoelectronics.  Vishay is a major supplier of infrared emitters, photo detectors and optical sensors to automotive, consumer and industrial manufacturers.

Suggested Articles

Originally a 1960s memory manufacturer, Intel wants out of NAND following the market decline in 2018

Process could be used to apply sensors detect symptoms for COVID-19, and the sensors could be reusable

From Sky Water Technology to Intel the message is the same: domestic chip production protects national security and jobs