Ever wanted to be able to manipulate images on a computer the way Tom Cruise did in 2002 sci-fi film Minority Report? In the film, Tom Cruise plays a cop who can operate computers simply by waving his hands in the air. Well, now it is possible. A new Media Lab invention, SixthSense, lets you do just that as it allows users to manipulate digital information with hand gestures. The gadget works by cobbling together a web camera, a tiny projector and an internet-connected smart phone into a device that can be worn like jewellery. Tiny sensors then respond to hand movements—allowing wearers to conjure internet information seemingly from the air.
Pranav Mistry, an MIT, USA, scholar and his adviser, Pattie Maes, have contrived this fascinating gesture-controlled computer interface that recognises and responds to hand signals and conducts appropriate searches. Though the gadget is some time away from being available, it can turn any surface into a computer touch-screen.
This gadget’s prototype is comprised of a pocket projector, a mirror and a camera. The hardware components are coupled in a pendant like mobile wearable device. Both the projector and the camera are connected to the mobile computing device in the user’s pocket. The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognises and tracks user’s hand gestures and physical objects using computer-vision based techniques. The software programme processes the video stream data captured by the camera and tracks the locations of the coloured markers (visual tracking fiducials) at the tip of the user’s fingers using simple computer-vision techniques. The movements and arrangements of these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces. The maximum number of tracked fingers is only constrained by the number of unique fiducials, thus this device also supports multi-touch and multi-user interaction.
When outside conditions aren’t enough to prompt the internet search, this device makes use of gestures. Tracing a circle on your wrist tells the device to project a digital clock on your arm. And users can take photographs simply by framing an object with their fingers. In this early stage, Mistry wraps coloured tape around his thumbs and pointer fingers, markers that make them easier for the web cam to spot.
So far, there are just a handful of online databases that this device can fluidly mine for pieces of what Mistry calls “instant wisdom”. In addition to grocery store inventories, Mistry has thus programmed the gadget that by simply waving their hands over an aircraft ticket, the gadget can let users know whether the flight is on time. Or it can recognise books in a bookstore and project reviews or author information from the internet onto the pages in front of it and project video clips about current events onto newspapers.
As slick and automatic as these capabilities might appear, they yet to offer a significant upgrade over an internet-enabled cellphone. But observers are already envisioning future improvements to the gadgets which could result in some startling possibilities. “Its current representation is a pretty fun parlor trick that has the roots of being a transformative capability down the road,” says Jonas Lamis, founder of the advanced technology research and consulting firm SciVestor in Austin, Texas, USA.