Even though forms of gestural devices can vary wildly—from massive touchscreens to invisible overlays onto environments—every device or environment that employs gestures to control it has at least three general parts: a sensor, a comparator, and an actuator. These three parts can be a single physical component, or, more typically, multiple components of any gestural system, such as a motion detector (a sensor), a computer (the comparator), and a motor (the actuator).
A sensor is typically an electrical or electronic component whose job is to detect changes in the environment. These changes can be any number of things, depending on the type of sensor, of which there are many.[11] The most common types currently used for interactive gestures are:
- Pressure
To detect whether something is being pressed or stood on. This is often mechanical in nature.
- Light
To detect the presence of light sources (also called a photodetector). This is used mostly in environments, especially in lighting systems.
- Proximity
To detect the presence of an object in space. This can be done in any number of ways, from infrared sensors to motion and acoustic sensors.
- Acoustic
To detect the presence of sound. Typically, this is done with small microphones.
- Tilt
To detect angle, slope, and elevation. Tilt sensors generate an artificial horizon and then measure the incline with respect to that horizon.
- Motion
To detect movement and speed. Some common sensors use microwave or ultrasonic pulses that measure when a pulse bounces off a moving object (which is how radar guns catch you speeding).
- Orientation
To detect position and direction. These are often used in navigation systems currently, but position within environments could become increasingly important and would need to be captured by cameras, triangulating proximity sensors, or even GPSes in the case of large-scale use.
It's no exaggeration to state that the type of sensor you employ entirely determines the types of gestural interactions that are possible. If the system can't detect what a user is doing, those gestures might as well not be happening. I can wave at my laptop as much as I want, but if it doesn't have any way to detect my motion, I simply look like an idiot.
It is crucially important to calibrate the sensitivity of the sensor (or the moderation of the comparator). A sensor that is too sensitive will trigger too often and, perhaps, too rapidly for humans to react to. A sensor that is too dull will not respond quickly enough, and the system will seem sluggish or nonresponsive.
The size (for touchscreens) or coverage area of the sensor is also very important, as it determines what kinds of gestures (broad or small, one or two hands, etc.) are appropriate or even possible to have.
Note
The larger the sensor coverage area is, the broader the gesture possible.
Often in more complex systems, multiple sensors will work together to allow for more nuanced movement and complicated gesture combinations. (To have 3D gestures, multiple sensors are a necessity to get the correct depth.) Many Apple products (including Apple laptops) have accelerometers to detect speed and motion built into them, as do Wii controllers. But accelerometers are tuned to themselves, not to the environment, so they alone can't determine the user's position in the room or the direction the user is facing, only whether the device is moving and the direction and speed at which it is moving. For orientation within an environment or sophisticated detection of angles, other sensors need to be deployed. The Wii, for instance, deploys both accelerometers and gyroscopes within its controllers for tilt and motion detection, and an infrared sensor that communicates to the "sensor bar" to indicate orientation[12] for a much wider range of possible gestures.
Once a sensor detects its target, it passes the information on to what is known in systems theory as a comparator. The comparator compares the current state to the previous state or the goal of the system and then makes a judgment. For many gestural interfaces, the comparator is a microprocessor running software, which decides what to do about the data coming into the system via the sensor. Certainly, there are all-mechanical systems that work with gestures, but they tend toward cruder on/off scenarios, such as lights that come on when someone walks past them. Only with a microprocessor can you design a system with much nuance, one that can make more sophisticated decisions.
Those decisions get passed on to an actuator in the form of a command. Actuators can be analog or mechanical, similar to the way the machinery of The Clapper turns lights on; or they can be digital, similar to the way tilting the iPhone changes its screen orientation from portrait to landscape. With mechanical systems, the actuator is frequently a small electric motor that powers a physical object, such as a motor to open an automatic door. As with the comparator's decision making, for digital systems, it is software that drives the actuator. It is software that determines what happens when a user touches the screen or extends an arm.
Of course, software doesn't design and code itself, and sensors and motors and the like aren't attached randomly to systems. They need to be designed.
[11] http://en.wikipedia.org/wiki/Sensor has a more complete list, including many sensors that are currently used only for scientific or industrial applications.
[12] See "At the Heart of the Wii Micron-Size Machines," by Michel Marriott, in The New York Times, December 21, 2006; http://www.nytimes.com/2006/12/21/technology/21howw.html?partner=permalink&exprod=permalink.
[13] For a more detailed view of FTIR and DI systems, read "Getting Started with MultiTouch" at http://nuigroup.com/forums/viewthread/1982/.
Get Designing Gestural Interfaces now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
