In this blog post, we’ll explore the principles and possibilities demonstrated by Pranav Mistry’s “Sixth Sense” technology, which sought to connect the physical and digital worlds using hand gestures, projectors, and cameras.
From Movie Scenes to Real-World Technology
In the 2002 film ‘Minority Report’, the protagonist, Tom Cruise, operates a computer system using hand gestures and other body movements. This film serves as a prime example, and similar scenes have frequently appeared in various other movies depicting future societies.
Technologies that apply information from the physical world to the digital world have been steadily advancing for a long time. For example, technologies such as operating portable electronic devices via touch and having computers recognize hand gestures have been continuously developed. However, since these technologies were primarily focused on bringing information from the physical world into the digital world, they had limitations in bridging the gap between the two worlds. Therefore, a new form of interaction was needed to connect the physical and digital worlds, allowing users to move naturally between them using gestures. The name “SixthSense” was used to describe this concept; it should be understood not as the actual addition of a sixth sense to the human body, but rather as a way to connect existing senses with digital information to enable a new form of interaction.
Bringing the Digital World into the Physical World
Amid this trend, a technology capable of revolutionizing conventional thinking was unveiled at TED in 2009. Developed by Pranav Mistry during his research at the MIT Media Lab, SixthSense departed from previous approaches by asking, “What if we brought the digital world into the physical world?” This technology was unveiled at TED in 2009, and the MIT Media Lab introduced it as a wearable interface that uses a projector and a camera to project digital information onto physical surfaces and manipulate that information through hand gestures.
It began with a relatively simple idea: combining a projector—a typical device for projecting information from the digital world onto the physical world—with a camera, which brings information from the physical world into the digital world. Previously, it was difficult to integrate the two worlds to create new value. One reason was the lack of ways to directly interact with the physical world from the digital world, and another was the spatial constraint posed by the need for a screen and input devices. For example, using a smartphone required input processes—such as pointing the camera at an object or touching the screen—as well as an output device, namely the smartphone screen, to display the processed information. SixthSense sought to reduce these constraints by using a camera and a projector together. It utilized a camera—which functions like the human eye to recognize objects—and a projector to display digital information in the physical world; initially, it tracked the position and movement of the hand using colored markers attached to the fingers. The goal was to expand the bidirectional interaction between the physical and digital worlds.
Manipulating Digital Information with Fingertip Gestures
This technology, which operates by having a built-in camera recognize markers and gestures on the fingertips, was capable of performing a variety of tasks. For example, it could project a map onto a wall or a hand and then zoom in on it, or allow users to draw on the projected screen. Additionally, the system could capture a photo of a scene by forming a rectangular frame with the fingers—just as we typically do when taking a picture. This feature demonstrates that SixthSense is not merely a technology for viewing screens but one that utilizes hand movements themselves as a means of input.
After taking a photo, users could also edit or organize the photos using finger gestures. It was also possible to have the camera recognize items or books held in one’s hand to retrieve information related to those objects. When reading a newspaper article or examining a map, if the camera recognized the content, it could display related information—such as news videos or regional weather forecasts—onto the physical object via the projector. The demonstrations at the time showcased how digital information could be projected onto real-world objects like books or newspapers, linking the physical object with digital information.
Of course, it was also possible to transfer digital data being manipulated via gestures to other computing environments, and the concept of bringing information from the physical world—such as text printed on paper—into the digital world through gestures, as if plucking it up with one’s hands, could also be implemented. The most intriguing aspect of this technology was that these tasks were achieved using cameras, projectors, mobile computing devices, and hand gestures, rather than relying solely on traditional screens and input devices.
What legacy did SixthSense leave behind?
At the time the original article was written, there were expectations that SixthSense would be commercialized in the near future; however, to date, SixthSense itself has not been widely commercialized as a consumer product in the form demonstrated back then. Instead, SixthSense remains a representative research prototype that demonstrated the potential of wearable interfaces and gesture-based interaction for naturally connecting the physical world with digital information. Furthermore, plans were announced to release the related software as open source, and SixthSense is featured as an archived research project on Pranav Mistry’s project page.
The key significance of SixthSense lay in the possibility that digital information could enter the physical environment in which people live and interact naturally, rather than humans having to adapt their behavior to digital devices. The concept of connecting the physical and digital worlds and manipulating digital information through natural actions, such as hand movements, was quite novel at the time. Therefore, the “sixth sense” referred to in SixthSense should be understood not as the addition of an actual new biological sense to humans, but rather as the creation of new interactive experiences by connecting humans’ existing senses and behaviors with digital information.
This technology is particularly significant in that it demonstrates new ways of accessing information and interacting with devices for people who have difficulty using conventional methods, including those who cannot speak or hear. If technology is designed to understand human natural behavior and respond accordingly—rather than requiring humans to learn how to use machines—the relationship between humans and machines can also change. Although the current SixthSense has not been commercialized in the form originally envisioned, it remains an intriguing technology in that it encourages us to envision technology that naturally adapts to human behavior and the environment, rather than humans having to adapt to machines and act according to their rules.
