O'Reilly logo

Designing Gestural Interfaces by Dan Saffer

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 4. Patterns for Free-Form Interactive Gestures

"Sense, sure, you have, Else you could not have motion."

William Shakespeare, Hamlet, Act III, Scene IV

The following are patterns for gestures that are typically done in space, not via a touchscreen or other interactive surface. Of course, these patterns don't have to be mutually exclusive; free-form and touchscreen patterns can exist on the same device. For instance, a touchscreen kiosk may use Proximity Activates/Deactivates as a way to turn the kiosk on and off.

PROXIMITY ACTIVATES/DEACTIVATES

WHAT

This pattern is for performing the simplest of all gestures, requiring only the presence of a person without any direct body contact (except perhaps to a pressure sensor). When a body is detected, an action triggers.

USE WHEN

Use Proximity Activates/Deactivates to trigger simple on/off settings, such as lighting, display changes, sound, and other environmental controls.

WHY

Environmental and cost-saving concerns have driven the adoption of this pattern. For instance, if lights come on only when a person is in the room, it saves energy (and money).

HOW

The presence of a person can be detected with a variety of means: camera, motion detector, infrared "tripwire," pressure sensor, or microphone.

Of almost all the gestures, this is the one most likely to be triggered accidentally, so care should be taken when calibrating the sensor.

It may be beneficial in certain instances (for privacy or environmental controls) to be able to detect multiple people in one location, which requires considerably more nuance in processing the data.

A variation on this is to change the action based on other (somewhat) detectable characteristics, such as race, age, and gender.[27] For instance, a store advertisement that utilizes a camera could change its display based on the gender of a passerby.

EXAMPLES

Glowing Places combines plastic seating with LED strips and pressure sensors to measure the presence of people over time. Both the number of people sitting and the length of time they stay create lighting effects in the furniture. Many people sitting for brief periods of time result in more active lighting, whereas one or two people sitting for a longer period trigger mellow lighting. Courtesy Philips Lighting.

Figure 4-1. Glowing Places combines plastic seating with LED strips and pressure sensors to measure the presence of people over time. Both the number of people sitting and the length of time they stay create lighting effects in the furniture. Many people sitting for brief periods of time result in more active lighting, whereas one or two people sitting for a longer period trigger mellow lighting. Courtesy Philips Lighting.

Swinxs is an outdoor game console for children. Several of the games, such as the racing and hide-and-seek games, involve getting close to the base station with the (included) wristband. For example, in the case of a racing game, Swinxs would use the wearer's proximity to the base station to determine the winner. Courtesy Swinxs BV.

Figure 4-2. Swinxs is an outdoor game console for children. Several of the games, such as the racing and hide-and-seek games, involve getting close to the base station with the (included) wristband. For example, in the case of a racing game, Swinxs would use the wearer's proximity to the base station to determine the winner. Courtesy Swinxs BV.

The Wooden Mirror, which comprises a camera and dozens of individual slats of wood on motors, "reflects" the image of the person standing in front of it. Courtesy Danny Rozen.

Figure 4-3. The Wooden Mirror, which comprises a camera and dozens of individual slats of wood on motors, "reflects" the image of the person standing in front of it. Courtesy Danny Rozen.

MOVE BODY TO ACTIVATE

WHAT

The physical movement of the body or a part of the body (not just its presence as in Proximity Activates/Deactivates and not a specific gesture) triggers an action.

USE WHEN

Use this when the action you are enabling requires that the person be moving, not just be in an area (as in Proximity Activates/Deactivates). This pattern is often found in alarms and other environmental systems.

WHY

Instead of Proximity Activates/Deactivates, use Move Body to Activate either because the movement itself is meaningful (e.g., an activity such as dancing or running) and the action triggered is related to that motion, or to prevent false triggering.

HOW

This pattern requires the ability to sense movement—frequently, directional movement. This can be accomplished with cameras or infrared beams, or via accelerometers embedded in wearable devices.

EXAMPLES

The Nike+iPod Sports Kit consists of an accelerometer that is attached to or embedded in a shoe, which communicates to an iPod Nano. Users can track in real time their calories burned, distance, or time while running. Courtesy Apple.

Figure 4-4. The Nike+iPod Sports Kit consists of an accelerometer that is attached to or embedded in a shoe, which communicates to an iPod Nano. Users can track in real time their calories burned, distance, or time while running. Courtesy Apple.

Fukuda's Automatic Door adjusts to the shape of the user as she approaches the door. Courtesy Olga Itenberg.

Figure 4-5. Fukuda's Automatic Door adjusts to the shape of the user as she approaches the door. Courtesy Olga Itenberg.

The SensitiveWindow detects the positions and walking directions of passersby to trigger the appropriate content, such as a full-screen movie created to surprise someone walking left to right. Courtesy iO.

Figure 4-6. The SensitiveWindow detects the positions and walking directions of passersby to trigger the appropriate content, such as a full-screen movie created to surprise someone walking left to right. Courtesy iO.

POINT TO SELECT/ACTIVATE

WHAT

Pointing with a finger or an object either turns on (activates) an object or selects it for further manipulation.

USE WHEN

Use Point to Select/Activate to select objects on-screen or in the environment so that you can interact with them by sending commands to them.

WHY

Pointing is the most natural gesture for selection.

HOW

Point to Select/Activate requires the ability to detect the orientation of the user's hand. This can be accomplished via cameras or other means involving objects held in the hand (e.g., a remote control) or worn on the hand (e.g., a glove) that can indicate the position of the hand in relation to other objects (either in the environment or on-screen). In an environment, each object could have an individual sensor attached to it that detects when a hand is pointing at it (e.g., via a beam of infrared light).

EXAMPLES

The Wiimote works with a stationary sensor bar on top of the TV screen, allowing users to use the device to Point to Select items on-screen. Courtesy Herman Yung.

Figure 4-7. The Wiimote works with a stationary sensor bar on top of the TV screen, allowing users to use the device to Point to Select items on-screen. Courtesy Herman Yung.

Users can Point to Select items in retail store windows using the GestPoint system. Courtesy GestureTek.

Figure 4-8. Users can Point to Select items in retail store windows using the GestPoint system. Courtesy GestureTek.

Johnny Chung Lee demonstrating one of his infamous Wiimote hacks at the 2008 Maker Faire. Using an infrared "pen," users can, by pointing, select different drawing instruments and turn any wall into a virtual whiteboard. Courtesy Sebastian Heycke.

Figure 4-9. Johnny Chung Lee demonstrating one of his infamous Wiimote hacks at the 2008 Maker Faire. Using an infrared "pen," users can, by pointing, select different drawing instruments and turn any wall into a virtual whiteboard. Courtesy Sebastian Heycke.

WAVE TO ACTIVATE

WHAT

Waving a hand (often in front of an indicated sensor) triggers an action.

USE WHEN

Use Wave to Activate for a simple gesture that is not typically done by accident. As such, this pattern is good for use in public spaces.

WHY

Wave to Activate is a simple, common gesture. It is also used by people trying to manually trigger Move Body to Activate or Proximity to Activate.

HOW

A number of sensors can be employed to detect a wave, including a camera and, most frequently, an infrared beam. A wave can also be captured by a very small range on a motion detector or heat sensor.

EXAMPLES

To use enMotion's Paper Towel Dispenser, you wave your hand in front of the sensor, and a sheet of paper towel comes out of the dispenser. Courtesy Georgia-Pacific.

Figure 4-10. To use enMotion's Paper Towel Dispenser, you wave your hand in front of the sensor, and a sheet of paper towel comes out of the dispenser. Courtesy Georgia-Pacific.

You can turn on an Airswitch light by merely waving your hand slowly over the light. Users can make the light dim or brighten by moving their hand upward or downward. Courtesy Mathmos.

Figure 4-11. You can turn on an Airswitch light by merely waving your hand slowly over the light. Users can make the light dim or brighten by moving their hand upward or downward. Courtesy Mathmos.

Users have only to wave their hand near the Waterdrop Weather Station Clock for it to "snooze" the alarm. Courtesy The Sharper Image.

Figure 4-12. Users have only to wave their hand near the Waterdrop Weather Station Clock for it to "snooze" the alarm. Courtesy The Sharper Image.

PLACE HANDS INSIDE TO ACTIVATE

WHAT

Putting hands inside or beneath (nearly inside) an object triggers an action.

USE WHEN

Use Place Hands Inside to Activate when you want something to affect the hands or you want to turn on controls that would be manipulated by the hands.

WHY

Hands are used in so many different ways, it was inevitable that some patterns would focus on them. This pattern is still emerging, however, and is mostly seen today in bathrooms for washing and drying hands.

HOW

Infrared sensors seem to be the most typically used to execute this pattern.

EXAMPLES

The Tripoint Gooseneck Touchless Electronic Faucet releases water when hands are placed inside the sink. Courtesy Kohler.

Figure 4-13. The Tripoint Gooseneck Touchless Electronic Faucet releases water when hands are placed inside the sink. Courtesy Kohler.

The unfortunately named Dyson AirBlade (it doesn't sound like something I'd want to stick my hands into) dries wet hands that are placed into and slowly drawn out of the device. Courtesy Dyson.

Figure 4-14. The unfortunately named Dyson AirBlade (it doesn't sound like something I'd want to stick my hands into) dries wet hands that are placed into and slowly drawn out of the device. Courtesy Dyson.

The Sensor Soap Pump uses an infrared sensor to detect when hands are placed beneath its spout. Courtesy simplehuman.

Figure 4-15. The Sensor Soap Pump uses an infrared sensor to detect when hands are placed beneath its spout. Courtesy simplehuman.

ROTATE TO CHANGE STATE

WHAT

Turning or tilting an object changes either the functionality of the object or the feedback/display of items on that object.

USE WHEN

Use Rotate to Change State on devices that can be used for multiple purposes or to display content that might be better viewed in portrait or landscape view.

WHY

Particularly useful on devices with limited, rectangular screen real estate, Rotate to Change State allows content to be viewed in ways that best display the content, given the screen size.

HOW

Rotate to Change State requires a gyroscope or a multiaxis accelerometer to determine the position of the device. The threshold for the change in state (e.g., 90 degrees, 180 degrees, etc.) needs to be determined.

If the device lays flat (e.g., on a table), accelerometers may not pick up the rotation.

EXAMPLES

Nokia's newer N series phones, such as the N95 (pictured), are equipped with accelerometers and can change views from portrait to landscape based on how they are held. Courtesy Nokia.

Figure 4-16. Nokia's newer N series phones, such as the N95 (pictured), are equipped with accelerometers and can change views from portrait to landscape based on how they are held. Courtesy Nokia.

The "Bar of Soap" changes its functionality (from camera to PDA to phone to gaming device) based on how it is held, switching modes when rotated. Courtesy Brandon Thomas Taylor and MIT Media Lab.

Figure 4-17. The "Bar of Soap" changes its functionality (from camera to PDA to phone to gaming device) based on how it is held, switching modes when rotated. Courtesy Brandon Thomas Taylor and MIT Media Lab.

Canon's PowerShot SD1000 changes its display (to portrait or landscape) based on how the user is holding it. Courtesy Canon.

Figure 4-18. Canon's PowerShot SD1000 changes its display (to portrait or landscape) based on how the user is holding it. Courtesy Canon.

STEP TO ACTIVATE

WHAT

A user steps (either deliberately or unknowingly) onto a designated spot that triggers an action.

USE WHEN

Use Step to Activate when other proximity detectors (sound, infrared cameras, etc.) would be impractical, or for actions that would naturally be delivered by the feet (stepping, dancing, etc.).

WHY

This pattern is useful for alternative controls when the hands are occupied or for controls in low places where it would be difficult to reach with the hands.

HOW

Step to Activate typically requires a pressure sensor placed inside or beneath a flat object, sometimes to protect the sensor from hard stomps or presses.

EXAMPLES

One of the many incarnations of Dance Dance Revolution, where players follow on-screen cues to step in various directions quickly to play the game (and thus to dance). Courtesy Warren Rohner.

Figure 4-19. One of the many incarnations of Dance Dance Revolution, where players follow on-screen cues to step in various directions quickly to play the game (and thus to dance). Courtesy Warren Rohner.

The X-Keys Foot Pedal can be programmed to perform common computer functions with three different locations that can be used for Step to Activate. Courtesy PI Engineering.

Figure 4-20. The X-Keys Foot Pedal can be programmed to perform common computer functions with three different locations that can be used for Step to Activate. Courtesy PI Engineering.

The Rectangular Sensor Can is an example of Step to Activate that uses an infrared sensor instead of a pressure sensor. When a user slips his foot into the recession at the base of the trash can, the lid opens. Courtesy simplehuman.

Figure 4-21. The Rectangular Sensor Can is an example of Step to Activate that uses an infrared sensor instead of a pressure sensor. When a user slips his foot into the recession at the base of the trash can, the lid opens. Courtesy simplehuman.

SHAKE TO CHANGE

WHAT

Rapidly moving an object back and forth changes its state or triggers an action.

USE WHEN

Use Shake to Change for very simple actions that have little nuance, usually with objects that can be held in the hand easily.

WHY

This is a simple action that most people can perform. Also, it has an element of fun and delight to it.

HOW

Shake to Change requires an accelerometer to detect motion. When enough vibration is detected, the action triggers.

EXAMPLES

You turn on an Imageo Candle Light by shaking it. Courtesy Philips Lighting.

Figure 4-22. You turn on an Imageo Candle Light by shaking it. Courtesy Philips Lighting.

The Sony Ericsson W910i features a three-axis accelerometer, which is used in its shake control feature to change song tracks. Users just shake the phone to change the track being played. Courtesy Sony Ericsson.

Figure 4-23. The Sony Ericsson W910i features a three-axis accelerometer, which is used in its shake control feature to change song tracks. Users just shake the phone to change the track being played. Courtesy Sony Ericsson.

The Sansa Shaker lets users shake the device to jump to the next song. Courtesy SanDisk.

Figure 4-24. The Sansa Shaker lets users shake the device to jump to the next song. Courtesy SanDisk.

TILT TO MOVE

WHAT

Moving the body, limb, or hand in a particular direction moves an avatar, an on-screen object, or a physical object in that direction.

USE WHEN

Use Tilt to Move as a means of controlling direction in a 3D or physical space. Note that body tilting may be difficult for young children or the elderly.

WHY

Tilting allows for a lot of nuanced control of movement, yet it also allows the body to remain mostly stationary while doing so.

HOW

Tilt to Move generally requires some sort of gyroscope or multiaxis accelerometer inside a device to measure angle, or it can be faked with pressure pads on an object that is being stood on, detecting changes in pressure/weight as the user tilts a limb or the body in a specific direction.

EXAMPLES

Designed for navigating 3D worlds, the SpaceNavigator has a cap that users can tilt (as well as twist and move from side to side and up and down) to move their avatar in any direction in a 3D environment or to manipulate 3D objects. Courtesy 3Dconnexion.

Figure 4-25. Designed for navigating 3D worlds, the SpaceNavigator has a cap that users can tilt (as well as twist and move from side to side and up and down) to move their avatar in any direction in a 3D environment or to manipulate 3D objects. Courtesy 3Dconnexion.

To operate a Segway, drivers tilt in the direction they want to go. Courtesy Segway, Inc.

Figure 4-26. To operate a Segway, drivers tilt in the direction they want to go. Courtesy Segway, Inc.

The Wii Balance Board uses a series of pressure sensors to detect a user's balance to play games such as We Ski. Courtesy Nintendo.

Figure 4-27. The Wii Balance Board uses a series of pressure sensors to detect a user's balance to play games such as We Ski. Courtesy Nintendo.

The next chapter looks at documenting gestural systems—systems that may contain the patterns in this chapter and the preceding chapter.



[27] See, for instance, "Billboards That Look Back," by Stephanie Clifford, in the New York Times, May 31, 2008 (http://www.nytimes.com/2008/05/31/business/media/31billboard.html?ex=1370145600&en=c4fb20d8fbcba416&ei=5124&partner=permalink&exprod=permalink).

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required