Last week, HP and Shell announced their collaboration on a seismic sensing system that will couple many thousands of wireless sensors together in a bid to create a better tool for oil and gas exploration. And although this is the latest in a series of projects aiming to create very large systems of networked sensors, I haven't seen anywhere near as much discussion about what to do with the vast quantities of data that will ensue and, specifically, how to handle them.
Steve Lohr in his New York Times blog entry, "Smarter Sensors Start Going to Work" said of such research projects from HP and Intel: "Their promise, writ large, is to help link the digital world of computing to the physical world as never before. The payoff would be to bring data-rich measurement, more intelligence and higher levels of optimization to all sorts of fields—including energy, traffic management, food distribution and health care."
We know this. If you've been involved in the sensor industry for any length of time you've already seen how more and more sensors are being used, in manufacturing and in end products (and in every industry I can think of). You've seen the number of types of sensors increase, you've seen their sophistication increase, and you've seen their applications diversify. But central to any sensor's utility is the requirement that the data be useful, which generally means communicating it to a DA or control system or a computer—some entity that will analyze the data and use it to optimize a process or warn of a machine failure or say which sensor is out of calibration or any of a million other tasks.
I know people are working on the data management end of the equation: there are scholarly papers, there are conferences and workshops on the topic, and I'm certain that HP, Intel, and IBM—all of them companies with serious experience in the computing and data management—have a good grasp of how to scale up to (and manage and use) networks involving millions of devices. Groups are trying to hammer out how to make disparate types of data Web-accessible in some uniform manner to facilitate sharing and use of the data. But what I want to know is, where does the field stand in terms of non-academic, real-world findings? If you know, I'd love to hear from you.