The scientist and philosopher Alfred Korzybski once remarked, “The map is not the territory,” and it’s in that spirit that this book was written. The map may not be the territory, but it is helpful for getting around the territory and for finding where you are and where you want to go. This book covers a vast range of topics from programming to electronics to interaction design to art, but it doesn’t cover any one of them in great depth. It covers all of these topics because they are part of an emerging territory that is often called interaction design, and that territory encompasses art, design, psychology, engineering, and programming. It’s also a territory that is becoming more and more accessible thanks to excellent projects like the ones that we’ll be exploring in the book—tools that have been created to make code and coding easier to do.
You should use this book like a map to see what technologies exist and the areas in interaction design that you might want to explore. This isn’t a cookbook or an in-depth technical manual, but it will point you in the direction of other books, researchers, designers, projects, and artists as you go along. This book will also give you the technical understanding to know how to find information on almost any kind of project that you want to explore and what to do with that information once you find it.
This book was created under the premise that technology and code are not tools solely for computer scientists or engineers to create applications and that no one be intimidated by or shy away from working with and exploring electronics, hardware, and code. Artists and designers can be interested in enabling interaction between users and between applications in ways that can be accentuated by the addition of custom computer applications or that can be realized only through the use of custom computer applications. You can focus on creating applications that emphasize their technological nature or on creating applications that feel very high-tech or use familiar metaphors like a keyboard and mouse or touchscreen. You can also choose to accentuate other aspects of the interaction or hide the technology behind a more organic interface. This book is specifically about the interactions that users or viewers can have with computers, electronics, tools, and the platforms that artists and designers can use to create applications and electronics that users can interact with. You’ll be learning about three tools: Processing, openFrameworks, and Arduino.
These frameworks are designed specifically for artists and designers and as such are perfect for discussing how we can begin to create interactive designs and artworks. Each of them has a different background and uses different kinds of technology, but all of them are created with the goal of helping you explore and create applications more painlessly and quickly. In addition to showing you specifics of those three tools, this book focuses on three slightly more abstract concepts: code, interaction design, and ideas. Creating code is a similar activity whether you’re writing something in C++ for openFrameworks or you’re creating some logic in a circuit with Arduino. In both cases, you’re creating a process that will run many times, perhaps even thousands of times, and that will generate the outcome you want.
This book also makes a few assumptions about you, the reader. I assume that you don’t have a deep, or even any, programming or technical background. I also assume that you’re a designer, artist, or other creative thinker interested in learning about code to create interactive applications in some way or shape. You might be a designer wanting to begin playing with interactive elements in your designs, wanting to create physically reactive applications to explore some interaction design concept, or wanting to prototype an idea for a product. You might be an artist wanting to begin working with interactive installations or with interactive computer graphics. You might be an architect wanting to get a basic understanding of programming and hardware to explore reactive architecture. You might be none of these at all, which is fine, too, as long as you’re interested in exploring these themes while you learn about the three frameworks this book describes.
You’ll explore the nature of interaction through common tools and techniques as well as through some discussions with designers, engineers, and artists working with interaction. In all likelihood, this book will not radically alter your perception of what interaction is, nor will it introduce you to radically new modes of interaction. This book will introduce to you to methods of creating common interactive elements that you can then use to explore further techniques of facilitating interactions between users or creating interactive elements that a user or viewer can experience.
This book is called Programming Interactivity because it’s focused primarily on programming for interaction design, that is, programming to create an application with which users interact directly. There are many styles of programming, and some techniques and ways of thinking about code are better suited to programming servers or databases than interaction. In this book, we’re going to concentrate explicitly on things you can use to tell users something or to have users tell your application something. One of the great challenges in interaction design is actually creating real interactions between what you’re designing and the user who will be using it.
So then, what exactly is interaction? Interaction could be defined as the exchange of information between two or more active participants. The writer and video game designer Chris Crawford describes interaction as “an iterative process of listening, thinking, and speaking between two or more actors.” Generally, when we’re talking about interaction and programming it’s because one element in the interaction is a computer system of some sort or some control element that a person is trying to get to do something. The person for whom the computer or mechanical system is being designed is called the user, and what the user is using is called the system. There are many different terms floating around today, such as human computer interaction, computer human interaction, or experience design. All mean more or less the same thing: designing a system of some sort that a person can interact with in a way that is meaningful to them. As an interaction designer, you’re trying to understand what the user wants to do and how the system that you’re creating should respond. That system can be almost anything: a game, a menu, a series of connected sensors and lights, a complicated physically interactive application, or even a group of other people.
There is another key concept in interaction design that you should understand: the feedback loop. The feedback loop is a process of an entity communicating with itself while checking with either an internal or external regulatory system. That sounds a little more complex than it actually is. You’re actually already quite familiar with biological regulatory systems; sweating keeps your body cool, breathing keeps oxygen flowing through your body, and blinking keeps your eyes from drying out. When you need more oxygen, your body breathes harder. This isn’t something you have to tell your body to do; it simply does it. To maintain a constant level of oxygen, it sends out signals to breathe more and more deeply or frequently until it reaches the correct level. It feeds back on itself, sending signals to itself to breathe more again and again until it doesn’t need to send those signals anymore. You can also think of the feedback that you give yourself while staying upright on a bicycle. You’re constantly adjusting your balance minutely, with your brain feeding data to your body and your body feeding data back in a constant loop that helps you stay balanced. These loops are important in the notion of a system that does something constantly. Without feedback, systems can’t regulate themselves because they won’t know what they’re doing.
Let’s start at messaging and work our way up to interaction. While one participant certainly may be more active than the other, the “interaction” doesn’t really apply when we use it to describe a transmission, that is, a message sent to someone with no way of handling a response. Think of a television commercial or a radio broadcast: it’s simply a signal that you can listen to if you’re in the right place at the right time and you have the right equipment. These broadcasts flow on regardless of whether you or anyone else is listening, and they occur on their own time, in their own tempo.
When you give a user a way of rewinding or controlling the tempo of information, an extra layer of user control is added. You can’t really interact with a book or a static web page, or even the vast majority of dynamic web pages, but you can control the speed at which you read them, and you can rewind information that you’re not sure about. These are really guided transmissions in that they give you a chunk of information that is more or less established and ask you which part of it you want to view. Scrolling, linking, fast-forwarding, and rewinding are all the techniques of guided transmissions.
When you give a user a way to accomplish a task or input data into the system that changes it in a substantial way and you create a means for that system to respond to what the user is doing, then you’re creating interaction. Reactive interaction is really the beginning of interaction because it gets you started thinking about what the user will do and how your system or object will react. For everything that user does, the system or object needs to have a response, even if that response is “I didn’t understand” or another kind of error message. This can also be built into a single system. Many kinds of applications monitor their own performance, checking the state of a property in the system or the number of boxes available in a warehouse, for instance. If you imagine this as being an interaction between two people, then you might imagine a parent giving a child an order.
A somewhat more complex model of interaction is one where the system is constantly doing a task and the users’ input regulates that task. Many industrial monitoring systems function this way, as do the underlying parts of game engines, and many interactive installations. The difficulty of creating this kind of interaction is ensuring that users always know what the system is doing at any given time, understand how they can modify it, and understand exactly how their modifications to one aspect of the system might affect another. If you imagine this between two people, then you might imagine a parent helping a child walk, ensuring that she doesn’t fall over as she goes. You can also imagine how a regulatory system might function, where the system regulates the user as they’re executing a task. This isn’t really two entities fully communicating because the regulated system doesn’t respond—it simply changes its behavior—but it does involve continuous systems. Systems can perform this task on their own as well, monitoring a process and providing regulation of an ongoing process.
This last mode of interaction blends into another. It is a very similar but slightly more complex model of creating interaction that might be described as the didactic, or learning, mode of interaction. Here, the system is still running continuously, and the user can see into the system, but instead of regulating the behavior, the user is learning from the output data. A lot of monitoring applications function this way, providing a view into relevant data and data points that the user can use to learn about a process. Again, the system isn’t actively conversing with a user; it’s just running and reporting information to the user. The user also has his process driven by the reporting from the system but not really modified by it, which is why it’s a learning model. Both systems and people are more than capable of learning from themselves, albeit in quite different ways.
A more complex mode of interaction is a management type model where the user communicates something to a system and the system communicates something back that allows the user to carry on with a secondary task. This is where you begin to see the real complexities of communication between users and systems. The user is communicating with a system and asks the system to perform some task. The system responds in a way that allows a user to continue with a secondary task. The system continues to run, and the user continues to run even while she has her own internal feedback loop occurring. One can find this in many real-time monitoring applications in fields from finance to medicine.
Finally, we have the most complex mode of interaction: a full-fledged conversation. This is something that humans have mastered doing amongst one another, but it’s another matter altogether to create this between a human and a machine because of how complex the notion of a conversation really is. When you think about how much data is communicated in a conversation through words, tone of voice, facial expressions, body posture, subtext, and context, you realize it’s a substantial amount of information being exchanged and processed at extremely high rates. Most user-system conversations are a great deal less complex.
A simple but good example of this is navigating using a mobile device: the device is constantly updating its position and displaying that back to the user and providing directions, while the user is actively traveling and querying the device for information. Enabling this conversational mode of interaction between users and systems is one of the most pressing challenges in interaction design and engineering. These modes of interaction all present different challenges and help users do different kinds of things. You’ll find that the appropriate mode depends on the users, the task, and the context in which the interaction is taking place.
Interaction happens via messages sent from systems to users, and vice versa. These messages can be text, speech, colors, visual feedback, or mechanical and physical input or feedback. Depending on the kind of application, winking can be just as clear and important a message as pushing a button. One thing that interaction designers talk about a great deal is how to construct and receive messages in a way that is simple and unambiguous for users and for the system.
One of the most difficult tasks in creating interactive applications is to understand how the system sees messages from users and how the user sees messages from the system. With applications that have a great degree of interactivity, allow more tasks for the user and the system, and allow for more sophisticated messages, it is easy for a conversation to become unclear to one party. When a message isn’t understood, it’s quite important to help the other party understand not just what wasn’t understood but also how it can be fixed. If I don’t understand something that someone says to me, I ask that person to repeat it. If I ask for a web page that doesn’t exist, the server responds with an error page that tells me the page doesn’t exist. The more freedom each party has, the greater the possibility of erroneous, unintended messages, and the greater the need for educating one party about what the other party understands and how that understanding is being constructed.
Think for a moment about a conversation between two adults. Communicating like this requires years of what could be described as user training: learning a language, learning appropriate and inappropriate behavior, learning a value system, and so on. It is because of this that the interaction between two humans can be as rich as it is. This idea of training the user to understand what messages the system understands and what a message from the system means is a tricky process. Creating a program with a datagrid where a user can select items is quite simple for the user to begin to understand because most computer literate users are familiar with the notion of a datagrid. We see datagrids quite frequently, and we generally have an understanding of what they can do, what they can’t do, a rough understanding of what error messages coming from datagrids might mean, and how to use them. If you’re using a new kind of control or interface, you’ll have to make sure that you provide ways for users to learn what your system is, how it works, and what they can do with it.
There is a correlation between the richness of interactive systems and the difficulty of creating it: the richer the interaction, the more that can go wrong. This is part of why designers spend so much time and energy attempting to create anticipatable experiences: interactive experiences where a user or viewer can leverage other realms of knowledge or other experiences interacting. Popular slogans in design, like “principle of least surprise,” express the notion that the familiar interaction is the preferable interaction because the learning curve for the user is much more shallow than a truly novel interaction. Users must learn how feedback is returned to them and how to modify their behavior based on the feedback, both of which can be a lengthy process.
One part of the feedback from a system is actual messages sent back and forth—text prompts, for example—but the interface is another important part of the communication of an interaction. An interface sits between two actors and facilitates their communication. This can be a screen, a control panel, an interactive wall, or simply a microphone and a pair of speakers. The interface is whatever shared materials the user and the system use to send and receive messages. Interface design is a very large topic unto itself, but it gets a little more manageable if you consider it in terms of what it means for designing an interaction.
The interface is the medium of the communication between the user and the system. It drives a lot of what is possible and what is not possible, what is efficient and what isn’t, and what the tone of the interaction is. If you think about how you talk to someone on the phone versus how you talk to them in person, you’re probably using more hand gestures, facial expressions, and other forms of nonverbal communication in person and being more direct and using your tone of voice more when you are on the phone. What we use to do something affects a lot of how we do that thing. Having a functional, expressive, and attractive interface is very important in creating the means for an interaction to occur. The attractiveness of an interface is an important part of making an interaction pleasant to a user; the colors, text, symmetry, sounds, and graphics are important and are communicative elements that shape a great deal about what a user thinks about your system. This shouldn’t come as a great surprise to anyone, but users prefer good-looking interfaces. What makes those interfaces attractive is largely a matter of context, both for your users and for the task that they’re trying to accomplish with your system. While users prefer attractive interfaces, they need functional interfaces. The functionality of an interface is part of what makes a system good for a task and what makes a user able to use your system. Even if what that system does is rather opaque, the user still needs a functional interface that shows him what his input does and gives him feedback.
It’s important to remember that interaction is more than the use of an interface. When we consider the most common interactions between a user and a machine—for example, a cell phone call—they’re quite simple in terms of the interaction between the user and the object. For a cell phone, you simply dial numbers to find someone else in a system; it alerts you if you’re being sought, and it sends and receives sound. This relatively simple interaction is important for reasons other than the interaction between the person and the object; it’s important because of the context of that interaction: you can make a cell phone call from almost anywhere. Before cell phones, you needed a phone line available to you, but now, with a cell phone, you simply need a phone and an account. You can reach people while both of you are away from home, and you can be reached when you are away from your home or office. When the cell phone first emerged, cell phone users already understood how to make and receive telephone calls, and the general pattern of the user interface was already established. True innovations in user interfaces are very difficult to realize because they often require very substantial engineering efforts and serious thinking by the interaction designer to ensure that the interface will function properly. Also, they require a lot of user training and retraining. There aren’t a great deal of true revolutions in user interfaces: the creation of the keyboard, Doug Engelbart’s mouse (the prototype of the mouse we know today), Ivan Sutherland’s sketchpad, the desktop GUI, and now the capacitive touchscreen. These were technological changes and impressive feats of engineering, and they were also shifts in the way people used computers. Revolutionary interfaces shape more than just the way that a tool appears; they redefine the possibilities of how a tool can be used.
All interactions have a certain vocabulary that they use. If you think of how you delete something from the desktop with a mouse, you might say, “I select the file and drag it to the trash.” The actual actions that you’re performing when you do this are a little different from what the system understands you to be doing, but that’s not really what’s important. What’s important is that you understand what the actions you can perform are and you know that the system understands those actions in the same way and will perform them in the same way that you expect. Having a meaningful, efficient, and productive interaction, just like creating a language or a code, requires that both parties agree on the meaning of the symbol and the meaning of the order in which actions occur. Those particular understandings are going to be quite different depending on the interface and type of interaction that the user undertakes.
In this book, we’ll examine some of the many different kinds of interactions, but don’t take this next section as a list of categories. Considering the pervasiveness of computing and interactions that exist with computing, there are so very many kinds of interaction between humans and computers that it is difficult to even reference some of the most common modes of interaction without some overlap among categories.
- Physical manipulation
These are the first interfaces that were created for electronics and some of the first designed multifunction man/machine interactions. Typically, before the advent of the car and radio, which were the first two common machines with multiple interface elements, a machine had a single switch or use. The user’s attention was focused on a single task at a time. Radios and automobiles presented novel challenges because both required multiple actions by nonspecialists; in the case of the automobile, this included speed and direction at all times and other tasks at irregular times. The interface might be a control that represents either a state that can be activated by flipping a switch or pushing a button or a range that can be set by turning a knob or pushing a slider. The interface lets users not only control the values that they are setting but also check values via labeling of sliders, knobs, dials, and switches. Dials, oscilloscopes, and other feedback interface elements let users verify information more quickly without referring to the actual interface element that they were manipulating. This requires that the user monitor multiple sources of information at a given time while manipulating controls. Physical manipulation of a control is one of the most important and interesting ways of creating interaction with a system.
- Input using code
At the dawn of the age of computing, the classic user interaction model was a terminal where a user input code commands that were then run and the results were reported to the screen in the form of text. The driving interactive concept was to command the machine via a system of commands that the computer had been preprogrammed to recognize. The user had to be knowledgeable or at the very least comfortable with requesting help from a very bare interface. This is certainly not the end of keyboard-based interactive behaviors, though. Consider the notion of the hot key, for instance Ctrl+Z for undo, beloved by so many programmers and ubiquitous in all applications from word-and image-processing applications to browsers. The hot key is no different from the command line but accentuates the user interface by allowing the user to automate repetitive tasks or perform a task quickly without diverting their attention from another task.
- Mouse manipulation
This is the most common method of interacting with a computer at this moment and the interface for which almost all commonly used applications have been designed. Consider the language of working with the mouse, the techniques that have been implemented by designers and learned by users: drag-and-drop, double-click, and click-and-hold. These movements and the meanings behind them in different applications are not completely standard, nor are they entirely fixed. One application may use a given gesture in many different ways in a single application and rely on the user understanding the feedback given to them by the application to know which meaning of the gesture will be used in the current context.
- Presence, location, and image
The use of the presence and absence of the participant or user is an extremely simple but profoundly intuitive way of interacting. This can be detected by weight, motion, light, heat, or, in certain cases, sound. The reaction to simple presence or absence acts as a switch, begins a process, or ends a process. The presence of the body, though simple, is a powerful basis of interaction; it engages users and asks users to engage with their presence, their position, and their image. This can be as simple as an automatic door sliding open as we approach, or as complex as Theo Watson’s Audio Space, where visitors don a headset equipped with earphones and a microphone and record messages that are then placed in the spot where they were recorded. As another user enters the location where a message was left, the message is played back along with any recorded by previous visitors. Each message sounds as if it is coming from the spot where it was recorded. We can imagine the body as a switch, or we can imagine the body as the image of the body and analyze this using photos or videos in any great number of ways. This theme of embodiment drives a great deal of fascinating interactions using what is called computer vision, that is, the analysis of images input using a camera, turned into pixels, and then analyzed. Later in this book, we’ll examine using computer vision to detect movement in an image and even to detect the location of a human face within an image.
- Haptic interfaces and multitouch
At the time of the writing of this book, Apple iPhone, Microsoft Surface, and a great number of new tools for multiple touch-based interfaces have already been introduced. Given the excitement around these technologies, the speed of change and innovation will likely outstrip any attempts by myself or my editors to keep this text abreast of the most cutting edge products or technologies. Nevertheless, the fundamentals of designing and structuring interactions using these gesture-based interfaces will likely not change. These essentials are based on what will be familiar gestures to anyone who has used any of these products: using two fingers to expand or contract, turning two fingers to rotate, tapping to select. These are not used simply for software applications, either. Consider how often the waving gesture is used in an airport bathroom with sinks to turn on the water, paper towel dispensers, and hand driers. The language of these gestures becomes a language that we can use to enable interaction much as a common natural language, an icon, or a pattern of buttons pressed on a video game controller.
- Gesture
The gesture is a fascinating interactive model because it so readily associates itself with signs, writing, and physicality. This notion of the interaction that is not driven by a keyboard or a mouse is particularly powerful because mouse and key interaction is often nonintuitive for certain kinds of tasks. Gestures are often implemented with touchscreen interfaces or mouse movements or pens and are very often used for drawing applications, simple navigation elements, adaptive technologies, or applications for children. There are many different cutting edge interactive approaches that are being explored, from writing recognition systems and novel key input strategies like Swype to hand motion recognition systems via video.
- Voice and speech recognition
Voice recognition is the programming of a computer to recognize certain words or phrases and perform certain tasks based on those commands. Commands can be as simple as voice activation, that is, having the voice act as a switch to turn something on, and as complex as recognizing different words as commands. For a computer, words or commands are recognized as patterns of sounds that are then strung together and compared with a dictionary of patterns to determine what the command could be. Speech recognition is a much more advanced topic, using roughly the same approach as a simple command recognition engine, but with a far larger dictionary and more powerful tools to determine the input. Beyond speech, the voice itself can be used to provide input, volume, tone, and duration, and can be used to drive the interaction between users and applications.
This is just a short list of some of the most prevalent themes in interaction design. In this book, there won’t be space to cover all of these approaches to interactivity, but you will learn some of the basics behind each of them and get information about further resources that you can use for your own design work.
The great industrial designer Henry Dreyfuss called design “the measure of man.” By this, he meant that the design of things is an excellent way to understand and analyze the activities of human beings. Defining the word design is a task better left to others, so I’ll leave my contribution at this: interaction design is the creation of tools for how we do specific things. The more specific the thing, the more finely the tool can be honed for it, and the more specific the interaction design can be. Interaction is sometimes confused with “doing something with a tool,” and although that’s important, it’s a little less specific than “how we do things with a tool.” Thinking about tools in terms of how, rather than just what, when, or why, isolates those things about the interaction that define the experience of doing that task. A lot depends on the task as well. A singular task with a singular action does not foster much dissonance; therefore, it can bear a lot more dissonance before it becomes meaningless. A task of multiple actions creates much greater dissonance and can lose meaningfulness much more quickly.
The design of an interaction is a complex process that involves a lot of modeling of how a system will work, how a user will approach the goal she’s trying to accomplish, and how the interface needs to be configured to allow for all of these different operations. All of these taken together create the context of the interaction that you’re making. The context is very important to what choices you should make for the design of an interaction. You might want to make the interaction very cut and dry so that everything that the user expects is given to her as quickly as possible and in an unambiguous manner. Most business applications or very task-based applications function this way; users know what they can do in unambiguous terms, and the interaction doesn’t deviate much from that initial information. There is a real pleasure in knowing what to expect and getting it so that you can make the interaction—and by extension the application or object—attractive. Or, you might want to make something much more playful, where the reward is in discovering the interaction and seeing it change throughout the use of it. Either way, a good understanding of the context of the user will help you create a better system and a better experience.
One of the tricks of interaction design is that, fundamentally, what users are trying to do when they’re interacting with a system is to correlate it to something else that they’re more familiar with. Anyone who has ever said or heard anyone else say “the computer is thinking” has seen a little bit of anthropomorphic thought applied to a computer. As human beings, we are very good at a few different things, and when it comes to interaction design, one of the more important things is using our understanding of the inner processes of other people. Interaction with a system doesn’t really involve understanding what someone else is thinking, but it does use some of the same cognitive processes. To that end, as an interaction designer, you want to give good cues that will help users understand what’s going on. They may not need to know exactly what the process of your system is, and probably shouldn’t, but they do need to know more or less what your system is doing with the information that they give it.
Interactivity in art has been a hotly discussed and debated topic for at least 20 years now, and the kinds of interactivity that you see in art pieces are constantly changing to expand the definitions of art and interaction. There are many computer games that can be considered art, many art pieces that can be considered industrial design, and a vast and ever-increasing number of projects that can fit comfortably into art galleries and design shows.
For the purposes of this book, there isn’t much point in differentiating between the fields of interactive art, industrial design, interaction design, and traditional software engineering. Although these different fields might seem quite different from one another, they actually all share common goals. They all attempt to create objects and experiences for users, they use similar tools and processes, and they all share a common workflow that goes from sketch to prototype to final product to showing. You can think of a continuum, where at one end there are predictable and well-defined things that may be more suited for completing a task, and at the other end are more unpredictable and dissonant works that challenge and provoke us but may not be useful in our everyday lives. There is a curious dance between art and design in interactive art that plays on the relationship between simplicity and complexity, usefulness and uselessness, and goals and open interpretations. Deciding which end of that spectrum is more interesting to you has a lot of bearing on how you think about the interaction, but it doesn’t change the code that you write or the way that you design your hardware.
Making interactive art is quite different from making noninteractive art because the real object of interactive art is the situation. In painting, the object is the painting itself; in sculpture, it is the object and the space around it; in a video piece, the object is the video projection. In an interactive artwork, the object of the art is really the interaction between the viewer and the system that the artist has created. That system can be very technologically complex, it can have a single simple technical element, or it can have none at all. This book discusses artists and artworks that make heavy use of technology, what are often called new media artists, because they are artists who use or develop the tools that you’ll be learning how to use. I distinguish new media art from interactive art because projects that use programming (but that aren’t interactive) don’t have that key characteristic of being created in the situation where the viewer encounters them. There are many technologically sophisticated projects that use the tools covered in this book but that are not actually interactive. For the most part, this book covers artists who work with interactivity and projects that generate feedback in response to the actions of a user.
One of the interesting challenges of working with interactive art is that the art can be truly useful and functional in many ways while still being art. You also have a great deal of control over the context of what that art is; an artwork can be viewed or experienced in any location that the user chooses, altered to become something unrecognizable, or used in a way that it was not intended to be used when it was first created. Many designers are exploring what they call critical design, designed objects that not only function but exist to be thought-provoking as well, making users think in critical ways that are usually associated with art rather than with design. This overlap between the design of an object and the creation of an art experience is part of what makes interactivity such a rich topic for artists to explore because you can open the realm of what a user can experience and explore in deeply expressive ways.
The task or goal that an interaction facilitates is as important as the way in which an interaction is carried out between a user and a system. Again, the types listed here aren’t being included to make a list of types of interactive work, but to show some of the common themes that run through interactive art and design and to help you get an idea of what you’ll be exploring in this book:
- Supporting data visualization
Data visualization is an increasingly relevant theme given the amount of data that we as members of an increasingly information-centric society must process. A well-formed data visualization is a powerful tool because it lets a user not only comprehend individual data points but also understand the relationship between what are called data points, detect patterns in the data, and even reconfigure and recontextualize information. Data visualization accelerates the ability of the user to process and synthesize new information by not simply learning a fact but by locating the fact within a discourse quickly.
As the designer and writer Frank van Ham notes, “They should be massively collaborative...not focus on analysis but on communication...it should be visual and end user driven.” The goal of data visualization is to generate, for the user, a view into data. This can be a view that will help the user understand the data better, as in the work of Ben Fry, where he creates beautiful diagrams that let a viewer more quickly and more fully understand the relationships between the objects in the data. His approach is informed by aesthetic considerations and by careful cognitive and psychological research. I Want You to Want Me by Jonathan Harris and Sep Kamvar retrieves data from online data sites and uses that information to generate interactive visualizations. While still data visualization, this piece is a far more dynamic and whimsical approach than the usual graph-and-chart approach used in standard visualizations, and yet it performs deep and meaningful data parsing and analysis.
The interaction of the user can be a process of refining, exploring juxtaposition, mining new data, or storytelling. When designing data and the interaction with it, we must consider not only what data is presented, but also how users will interpret that data, what they might want to do with it, and how they would want to interact with it. Interaction is far easier when the view into data and the visual representation of that view are clearly related. For instance, the visual representation of the data and the visual representation of filtering should be clear so that the user easily understands what is being filtered and how to change it.
- Organizing tasks
Some interactions are interesting because of what they allow us to accomplish. The organization of tasks or actions or of discrete and discontinuous objects is the driving force behind much of the thinking in interface design. When you look back at the history of interfaces for machines and computers, you can see an evolution of organizing tasks, applications, information, and acts. The now ubiquitous desktop model allowed the user to organize tasks in a way that leveraged both natural cognitive abilities, like the ability to organize things spatially, and a familiar motif for any office worker, namely, the desktop.
One challenge for interaction design is to conceive of ways to effectively put working spaces and organizational tools in places other than the traditional desktop environment. Computing is everywhere, and users want access to programs and data at more places. How to enable interaction on very small screens or with no screen at all is an increasingly relevant challenge given the environments in which users are interacting with environments.
Some of the themes of exploration are how to develop novel desktop environments using new tools like multitouch screens, how to create tools for users to create their own interfaces to fit their needs at a particular time, and how to create interfaces for data visualization and tasks around data visualization.
These types of interactive applications tend much more to practical and functional concerns, enabling users to complete tasks, organize information, and save information. This certainly does not mean that they need to attempt to replicate an operating system in functionality, but rather that they draw on that vocabulary of interaction. This can be used in somewhat more subversive ways as well, as with Adrian Ward’s Auto-Illustrator.
- Creating experiences
Not all interactive designs need to rely on the traditional application model. In fact, one of the most common and powerful modes of interaction is what might be called the experiential model of interaction. These are often computer games, reactive drawings, or eye-catching graphic displays that engage and entertain without a set purpose. They often use novel connections between audio and visual stimulation and create either spectacles that entertain users or have entertaining interactions. The experiential interaction is very evident in the design of many kinds of computer games, where the user can play a character in the game and see the world through that character’s eye. Many times the interaction in these kinds of games is goal oriented, whether that goal be moving to the next level, killing enemies, or scoring points in some way or another. Many interactive installations use a similar model of interaction, where the interaction is playful but often lacks the goal-driven nature of gaming and instead focuses on enabling the viewing of a spectacle or playing with some engine that creates sounds or graphics. The goal of this kind of interaction is often simply to entertain or engage.
Both games and interactive installations often allow for fast switching between multiple views, perspectives, or models within the flow of the applications. This can be useful not just in gaming but also in an architectural fly-through, to show what a building will be like to walk through, or in data visualization. Gaming-style interfaces are also quite common, with first person views onto a 3-D world or with a 2-D view onto a world with the user controlling the view onto that world. These also often involve creating an environment that is reactive to user actions and independently active when the user is inactive to create the illusion of a world. Interactive installations or more playful and less task-oriented pieces will sometimes also involve inverted mediums, where one draws a sound, a sound creates an image, a physical object becomes a virtual one, or vice versa.
- Enabling collaboration between users
The interactiveness of an art piece most certainly does not need to be driven by data or a virtual world; it can be driven by multiple participants in concert with one another. We can conceive of this in a very straightforward way, such as in a whiteboard collaborative application, or in an unpredictable and experimental way, where the input of one user is added to the input of the others in a way that can’t easily be anticipated. As with many of these topics, a range exists of predictable and consonant works to unpredictable and dissonant works. Locative gaming where the game play is driven by the users’ locations is another kind of interactive application that uses collaboration between users to drive the interaction. Many network-based applications also use the model of collaboration between users to drive the interaction. The system in these kinds of interactions tends to facilitate communication and ensure that messages from one user are received by another instead of generating messages and feedback for a user like a single-player game.
These applications can use chat-based metaphors or presence-based metaphors like some of the large multiplayer games that have become popular lately, or they can create a physical manifestation of each user. As long as the user has some indication of how many other users are interacting and how their actions are affecting the environment or those other users, the interaction can be very compelling.
- Controlling mechanics
One of my favorite teachers, writers, and engineers, Tom Igoe, wrote, “Computers should take whatever physical form suits our needs for computing.” It is very limiting to think of computing strictly in terms of the computer itself and the traditional interface to the computer, that is, the screen. In fact, interactive designs and artworks can be far more. With the Arduino, we can easily create computers that control machines out in the physical world. These machines can perform tasks as simple as turning lights on and off or as complex as the control of robotics. The machine can be controlled manually by a user or by many users, or it can be reactive, controlled by a program that dictates its responses in reaction to stimulus from users or viewers or from a physical environment.
The control of mechanics can be very task-oriented with rigidly defined effects for each user action, or it can be very playful as in a collaborative application. In the latter case, the controls that the user has can be very loosely defined; that is, the user may have to play with the installation to discover the action that the control performs. In the case of task-oriented controls, the labeling and structure of the controls should be very clearly delineated.
- Using tools for performance and as performance
An application can be used as a way of creating an aspect of performance, aiding a performance, or accentuating a performance. You can think of examples as simple as the modification of an electric guitar to projects as complex as completely new interfaces for musical development. An interactive application or tool is a means to a performance or interaction, driven by a performer or driven by the audience. Some of the most interesting uses of this mode of interaction involve sharing a control between the performer and the audience, though this does require some user training to ensure that the audience understands what they are supposed to be doing. These tools don’t have to be as conceptually simple as creating a new tool for a performer to use on a stage. They can allow users to create new elements in a performance, control objects from remote locations, or create performers out of the audience.
- Creating environments
Many architects are beginning to explore what is called reactive architecture, the marriage of architectural practices with computing to create houses and environments that react to users, environmental factors, and external commands. The model of a feedback system is quite important to this model of interaction. The environment needs to monitor itself using sensors or timers to know when to change its state and when to maintain its state. At the simplest, a timer can be used to tell a house to turn the lights on and off, a temperature sensor can be used to maintain a certain temperature, a motion sensor can detect presence, or a humidity sensor can be used to control a dehumidifier. However, by using more complex sensors and systems, you can track movement in a space by using tags, cameras, microphones, or wireless radios, and have a system use that data to make changes in the environment to make it more comfortable, to make it louder, or to configure it correctly. Many architects and technologists are designing spaces and buildings that can be configured with a command by a user to change the way a space is used or to make the space itself more interactive. These sorts of spaces are often called smart rooms or enabled architecture, and they are an important area of research for both architects and engineers. Computing doesn’t have to be limited to indoor spaces, though; outdoor spaces like parks, walking trails, squares, or streets can also be sites for interesting technological interventions that can be playful, helpful, or thought-provoking. It is important, though, to always consider the appropriateness of an application for the space in which it exists and how the user engages that interaction. In a public space, this becomes especially important since a user should have the decision of whether to interact with it.
- Telling a narrative or story
One of the more interesting themes beginning to emerge in interactive design is the notion of using interaction to tell a story or narrative. These sorts of works typically rely on the interface to allow the user to control the flow or direction of the narrative using techniques cribbed from data visualization or gaming. Despite using concepts familiar from gaming and visualization, narratives offer a different set of challenges more familiar to filmmakers and authors than to engineering concerns.
The actual process of creating interactive work generally follows a combination of any of the following processes:
- Conception
Conception can consist of sketches in a notebook, a daydream, a product or process of research or something developed in consultation with a client who wants the application for commercial purposes, or any combination of the three. You should try to map out in as much detail as possible what you would like the application to do, how the interaction should feel to the user, and the goals of the application. All projects will require research and planning of some kind. You shouldn’t let this suffocate you or stifle your creativity, but you should include it. Starting to sketch without a clear plan of what you’re doing can often lead to great and new ideas. For most people, I dare say that starting to write code without a clear plan of what they’re doing usually doesn’t produce something usable.
- Research
When you’ve decided what you would like your application to look like and how you would like it to function, you’ll need to do the research on what components you might need and what libraries or existing code might be available that can help you create your project. If you need hardware, you should determine your budget for your application and determine which components you’ll need and how they fit into your budget. It’s important to ask questions either on forums or of colleagues to ensure that the components or approaches you’re considering will work. Most projects will require different technologies, but almost all the requisite pieces will be available, or you will be able to leverage existing projects that have been created by you or by another artist or developer. Twenty years ago, this may not have been true, but it is now.
- Design
The design phase is one of the more amorphous because it blends so easily into the research, conception, and actual building of your project. Sometimes you may not have a design phase, and sometimes all you will have is a design phase, depending on the requirements and nature of what you are building. At the very least, you should define all the parts of your application clearly, have a clear vision of how they will appear to a user and how they will respond to a user, and understand exactly how a user’s action should correlate to an action by the system. It may help to create diagrams that show the flow of actions and responses from the user to the system and back. It may also help to create diagrams of different parts of the system and show how those will appear to the user, what the user might want to do with them, and how they relate to the interaction overall. You might not want to plan everything in your application, but the more that you can plan, the easier the actual building of your application will be.
- Build
This is the process of actually putting together hardware and writing code. This is where you’ll write, debug, probably research some more, and ultimately assemble your application.
- Test
Once you’re finished building your application, it’s important to test it. Testing can be as complex as creating situations for users to use your application and observing how they use it, or it can be as simple as using the application or hardware yourself and ensuring that everything works.
After testing, you might be ready to present your project by installing it or making it available for download. If your project is a piece of hardware, then you may be ready to prepare it for manufacture, if that’s your aim, or to place it in the location and environment that you mean it to be used. You’ll want to make sure that everything works as intended before sending a project off, so having a good testing phase is important.
Now that you’ve learned a little bit about what this book is about and the different ideas that you’ll be exploring, you can start learning how to code.
Get Programming Interactivity, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
