FitByte, a device that attaches to your eyeglasses and tracks all phases of food intake, is just one example of how researchers at Carnegie Mellon University are exploring ways to leverage generic sensors found on consumer devices to develop new and innovative health monitoring applications.
To overcome some of the challenges with existing approaches, FitByte employs a multi-modal sensing approach to more precisely capture data in the typically noisy, unconstrained environments in which people consume food.
An onboard camera captures images of the area around the mouth, while gyroscopes monitor jaw motion, high-speed accelerometers located near the earpiece of the glasses detect throat vibrations when swallowing, and a proximity sensor detects hand-to-mouth gestures.
All the sensors used in the device are of the kind commonly found in smartphones and wearables like the FitBit. Extending the capabilities of these generic sensors, said researcher Mayank Goel, an assistant professor in the Institute for Software Research and the Human-Computer Interaction Institute, is key to the commercialization and scalability of tomorrow’s health monitoring devices and diagnostic tools.
“Smartphones are ubiquitous, and if you can find ways to extend the existing technology there are huge opportunities to provide richer interactions and improve the quality of life,” said Goel. “By leveraging what is already there, you also avoid the complications of developing entirely new sensors for an application, for which the barriers to entry are quite high. It also can be a tough sell to convince manufacturers of these general-purpose computing platforms to add more sensors that are too specific for an application.”
So the philosophy at CMU’s SmaSh l(Smart Sensing for Humans Lab), which Goel heads up, is to develop innovative sensing systems by extending the core functionality of everyday sensors, which are common across multiple manufacturers’ platforms.
Goel points to the example of SpiroSmart, an application that uses the microphone capability to convert a phone into a medical device. The device is able to measure the lung function of a patient by sensing sound and pressure. The data is sent to the cloud, where machine learning algorithms convert the data into standard lung function measurements.
The low-cost and portability of this smartphone-based solution, which has been tested in clinics in resource-constrained developing countries such as Bangladesh and Kenya, hold the promise of lower-cost access to healthcare.
Other applications SmaSH researchers are exploring include the relationship between sleep patterns and mental health issues and the use of a smartphone app to assess neonatal jaundice. In all cases, Goel said, the trick is to develop new information from what is already there—both by taking a multi-modal sensing approach and applying machine learning algorithms.
One of the challenges of this work, said Goel, is that it is completely dependent on the sensors featured on existing platforms. (Which, of course, is also one big advantage.) “Every year we see things like phones get upgraded, for example the camera might get changed. So, in a sense you would need to play that catch-up game with every new version of the platform, at the very least it means we would need to collect some data and calibrate the new sensor.”
Stressing the importance of working directly with the manufacturers when at all possible during the development phase, Goel said that Google was an ideal partner for the neonatal jaundice and spirometry applications, as they both use phone sensors and can be potentially built into the Android phone. Apple has also shown interest in many of the applications Goel and his collaborators are developing.
As for the FitByte, the next step is to add more noninvasive sensors to detect blood glucose levels and other key physiological measures.
“By collecting data on how people are behaving and what they consume, we can begin to connect that information with how their body reacts, for example, by monitoring their blood glucose levels. What we can look for is patterns that could be useful in their overall health management,” said Goel.
Editor’s Note: Mayank Goel will be speaking about FitByte and the research at Carnegie Mellon on wearables and medical devices at MedTech Innovation Week, a digital event series taking place October 19-22, 2021. For more information and to register for your free pass click here.