Designing for the Internet of Things

Chapter 3. Design for the Networked World: A Practice for the Twenty-First Century

MATT NISH-LAPIDUS

The Future of Design

Bruce Sterling wrote in Shaping Things (MIT Press) that the world is becoming increasingly connected, and the devices by which we are connecting are becoming smarter and more self-aware. When every object in our environment contains data collection, communication, and interactive technology, how do we as human beings learn how to navigate all of this new information? We need new tools as both designers, and humans, to work with all of this information and the new devices that create, consume, and store it.

Today, there’s a good chance that your car can park itself. Your phone likely knows where you are. You can walk through the interiors of famous buildings on the Web. Everything around us is constantly collecting data, running algorithms, calculating outcomes, and accumulating more raw data than we can handle.

We all carry minicomputers in our pockets, often more than one; public and private infrastructure collects terabytes of data every minute; and personal analytics has become so commonplace that it’s more conspicuous to not collect data about yourself than to record every waking moment. In many ways we’ve moved beyond Malcolm McCullough’s ideas of ubiquitous computing put forth in Digital Ground (MIT Press) and into a world in which computing isn’t only ubiquitous and invisible, but pervasive, constant, and deeply embedded in our everyday lives.

Augmented reality (AR) is here, already deeply engrained in our understanding of the world. The screen-based AR espoused by apps such as Layar is primitive compared to the augmentations that we all use on a daily basis. Google Maps, Twitter, Facebook, Nike FuelBand, and more are prime examples of how we are already augmenting our reality in fundamental ways that are less obvious and intrusive than digital overlays (which will see their day eventually, I’m sure). We have been augmenting our reality since the invention of clothing allowed us to live in harsher climates, and now we are augmenting it with networked technology giving us not just a sixth sense, but a seventh, eighth, and ninth, as well.

As augmentation and networks change our understanding of reality, we begin to understand old technology through our lens of new media. A chair is no longer solely a physical object that exists in our environment, it is now an interactive object by which specific behavior and person-to-person relationships can emerge from its use (Buchanan, 2011). A building is no longer only a collection of materials that defines a place, it is also understood through its interactions with people, the interactions it facilitates, and how it interacts or interferes with our networked augmentations. We are McLuhan-esque cyborgs, with media devices that extend our body and mind from the outside. Objects that exist as part of this network become more than their discrete pieces; we internalize their behavior and it changes the way we understand our world and ourselves.

We can see shifts in common language that allude to these changes. We talk about “downloading” knowledge from one person to another and “interfacing” with organizations. Words like “interface,” “download,” and “stream,” once not commonly used outside of technological circles, are now part of our daily lexicon, used in reference to their technological meaning as well as applied to much older concepts in the physical world.

A 2007 study on mobile phone usage conducted by Nokia concluded that the mobile phone is now one of the most essential items for daily use around the world, putting it in the same social category as wallets and keys.¹ They identified that it wasn’t only the object itself that is important to people, it is the social identity it provides that people value. The phone is more than an object—it is a lifeline, a gateway through which people connect with their family, friends, livelihood, and community. This is even truer now with the prevalence of smartphones with always-on Internet access. The smartphone has become one of the current embodiments of the networked world; more than its function, more than its form, it is a social safety net that allows people to travel or live further away from their home and still feel connected.

The smartphone is still a tangible object, one that we can understand through our hands and eyes, and it has connections to the network that we can see and feel. A greater shift is occurring now through objects that connect in less visible ways—objects that act on our behalf, or against us, without our explicit knowledge. The ethical implications and choices made by algorithms that determine the flow of traffic, our food supply chain, market pricing, and how you measure your fitness are present in our lives but are largely below the surface. As connected systems spring up around the world, often bypassing the more outdated infrastructure we are dealing with here in North America, we need to begin considering the biases and implications of our choices when designing these systems, objects, and networks. For example, the current sensors used to trigger traffic lights often rely on induction pads embedded in the road. These sensors only detect cars and other large vehicles, and are unable to sense bicycles and pedestrians. There’s an implicit decision made about the relative importance of different modes of transportation. A traffic system built on an inductive sensor network will always prioritize car and truck traffic over cyclists, for example, making the city a less hospitable place to ride a bike. This can in turn impact population density, pollution, congestion, parking, employment, injury rates, and more.

As we move even further into a networked world, we as designers of these new devices and services need to understand all aspects of our new environment. The complexity of design and architecture will only continue to grow and require a new definition of design foundations, practice, and theory.

This might seem daunting, but no more so than the nature of mass manufacturing and new materials seemed to the early industrial designers and architects of the twentieth century. We must look to new media art practice, design history, and new research in order to apply our craft to our current context. Designers make things that reflect their environment, but also shape that same environment through the objects that they create, laying the foundation for the future.

We have strong foundations stretching back over a century of art, architecture, and industrial design. We don’t need to begin again, but we do need to continue to evolve our practice to incorporate new techniques, tools, and capabilities that help us understand the potential of today’s technology.

What are the aesthetics of feedback, immersion, and communication? How can we apply foundations of interaction, such as time and metaphor, to the exchange of data between machines that facilitates an athlete learning how to perform better? What is a beautiful network and how do we recognize and critique it? These are the questions we now face, ones that we will continue to explore through our work and try to answer with objects, systems, places, and conversations.

New Environment, New Materials

[W]e have witnessed a paradigm shift from cyberspace to pervasive computing. Instead of pulling us through the looking glass into some sterile, luminous world, digital technology now pours out beyond the screen, into our messy places, under our laws of physics; it is built into our rooms, embedded in our props and devices—everywhere.

MALCOLM MCCOLLOUGH, DIGITAL GROUND (MIT PRESS), P 9

Over the past couple of decades, our environment has changed significantly.

Screens are everywhere all the time. This means that the complex interactions afforded by screens are even more important to understand and design properly.

Physical objects are now imbued with “smart” features using sensors, networks, and physical interactions that are often invisible, having no screen whatsoever. This makes physical object design more and more important for designing modern products, shifting focus back toward industrial design and architecture and away from the myopic attention to screens that interaction design has had recently.

Machine to machine communication is at the heart of many interactions and systems that we can’t live without. This means that designers need to think about not just the human actors in a system, but also the objects, networks, and algorithms that run our environments.

This puts the modern designer in a bit of a sticky situation. As an example, a project on which we recently embarked at Normative includes a mobile app that communicates with a physical box of electronics affixed to the back of a ski that is laced with embedded sensors, as shown in Figure 3-1. That box also needs to be aesthetically pleasing and fit the skier’s understanding of how a ski accessory should look and feel. The skier needs to enjoy working with the companion mobile app in a way that enhances the skiing experience. The box of electronics that reads the data from the sensors embedded in the ski needs to communicate that data to the mobile device, and has to communicate that it is doing something to the person on the skis through a simple display of LEDs and recessed buttons. All of this needs to happen in a way that makes sense to the skier, doesn’t detract from skiing, and withstands the environment of the slopes.

Figure 3-1. An early ski prototype²

In this example there are many types of design at work—industrial design for the skis and the electronics box; graphic design for the labels, ski graphics, packaging, and mobile app interface; interaction design for the mobile app; system integration; and coordinated communication between the app and the box. This is in addition to all the engineering involved in the hardware and software to make this all work.

What we witness in projects such as this one is a shift in the way we’re working from the industrial model of design → build → sell to a post-industrial model wherein all those things happen simultaneously in an integrated and iterative way within a small team. The initial prototype of the circuit was created by an interaction designer using an Arduino, and then an engineer and the designer worked together to refine the circuit through iteration. An integrated team of designers from different practices, creative technologists, engineers, and fabricators is required to design, build, and iterate on a system this complex.

At the heart of this team is a design practice that coordinates all the moving pieces, keeps the overall system in mind, and is the arbiter of the aesthetic and functional coherence of the final product. The lead designer needs to have a refined sense of aesthetics as it relates to the appearance of the physical product, the software, and the system that makes them work together. Figure 3-2 demonstrates this team effort at work as the prototype begins to transition toward a more polished product.

The overall aesthetics and quality of the interactive system, product, and associated software is the purview of this new breed of designer, including the impact and implications of the product. The modern designer needs to have a foundation in traditional design disciplines and interaction foundations, which acts as a framework for thinking about the form of objects and interfaces, as well as good understanding of systems theory, cybernetics (the study of feedback, relationships, and communication within a system), and culture, including a basic grasp of ethnography and anthropology in order to understand different contexts and cultures.

Figure 3-2. A higher fidelity prototype of the electronics and enclosure for the skis³

Happenings, Conversations, and Exploration

In late 1968 Jack Burnham, a writer and art history professor, wrote the following in his paper System Esthetics:

The specific function of modern didactic art has been to show that art does not reside in material entities, but in relations between people and between people and the components of their environments.

He was looking at the emergence of large-scale interactive artworks and art events in the 1960s. Artists began to see their work as more than the object itself; they began to think about how the object interacts with the audience and environment to create a conversation.

Artist David Rokeby explored the emotion and aesthetics of environmental feedback systems in his early works Reflexions, Body Language, and Very Nervous System in the 1980s. Rokeby created one of the earliest examples of gestural interface by building his own 8 x 8 pixel digital camera and programming his own software to read the video input and create feedback in the form of sound and video.⁴ To fully understand the aspects of movement and feedback systems he was interested in, he had to learn new technologies, create innovative solutions to unknown problems, and build his own sensors and output devices. If this sounds familiar, it’s because these are exactly the same types of activities and problems facing designers and artists today. Figure 3-3 presents a series of images illustrating the results of people interacting with the system.

Figure 3-3. Various people interacting with David Rokeby’s Very Nervous System (1986 – 2004) at aceartinc., Winnipeg, Canada⁵

To explore new concepts, behaviors, and environments, artists and designers need to develop a new set of tools and skills. Architects and interior designers use physical models, known as a maquette, to experiment with form, materials, lighting, orientation, and other properties of their designs. Similarly, designers working with emerging technologies need tools to experiment, mold, and model the elements of networked devices, software, and complex systems.

The success of new design tools to help work with somewhat intangible materials has to be measured based on how well it helps the designer understand the parameters of her design, and make choices based on experiencing aspects of the design in context. These tools should allow for different levels of generative and synthetic activities, varying fidelity, working with high-level abstract notions all the way down to the small functional and aesthetic details of the final product.

The current generation of digital design tools (CAD, Adobe Creative Suite) created new ways of working on traditional types of outputs. They gave us the ability to create many more variations of layouts, the safety of undo and file versions, and access to previously impossible or difficult processes for creating effects and working with new source material. However, they did not fundamentally change the component pieces of the designer’s process, toolbox, or output.

These tools are coming up short as designers are beginning to work with complex communications between people and machines, interactions and movement that happens over long periods of time and many individual devices, and large data sets that can’t easily be visualized using manual methods.

To add to this complexity, the entire notion of finality has changed. Designers traditionally create outputs that remain static, or have a small set of variations, once produced. Modality in traditional products was more a result of context, use, customization, or modification. In new types of products there is no “final version,” rather the product itself is a system, reacting to its environment and interactions, continually changing and evolving with use.

Twenty-First Century Foundation

Designers in the twentieth century needed to internalize and deeply comprehend things like 2D and 3D form, physical environments, and typography (to name a few areas of practice). The twenty-first century designer needs to build on these foundations with a number of new elements. The traditional elements of design were well established by Rowena Reed-Kostellow and her colleagues in the 1930s: line, plane, color, volume, value, and texture. She used these as the basis for her groundbreaking design foundations pedagogy at Carnegie Tech.⁶ Dave Malouf presented an initial set of interaction design foundations in an article for Boxes and Arrows in 2007,⁷ and then expanded upon it in a presentation at Interaction’09. He includes elements of time, abstraction, metaphor, negativity, and motion in his set of expanded foundations.

The things we design now are beyond screens and objects and we are challenged to think of the next set of foundations for designing these systems. We can begin to draw inspiration and knowledge from cybernetics, soft systems theory, and urbanism along with more commonly referenced practices such as architecture and anthropology.

When working with invisible technology and systems that cannot be observed easily, visualizations become even more important. Often, the only way that a system and all of its interactions and decisions can be understood is through illustrations and narratives that look at the impact as well as the cause of each part of the interaction.

As we examine these systems we should pay special attention to the qualities, aesthetics, of the elements of the system. A set of aesthetics qualities of a system includes new foundational elements that build upon traditional design foundations and Malouf’s interaction foundations.

Texture

What is the connectivity of the system? How do the pieces interact with one another, both human and nonhuman? The texture of the network is what we think about when we look at how easy it is to interface with its different parts. If the connections are obvious and accessible, we might describe the interface as smooth; if the connection points are difficult or confusing, that could be described as rough.

The notion of texture can be applied to graphical interfaces, gestural or spatial interfaces, hardware controls, and APIs alike, among other things. How might one describe the qualities of their bank’s system? This could include their ATMs, customer service, transfer between institutions, and more. Often a designer (or critic) will only be concerned with a subset of a network system, but it’s always good to pay attention to how that piece interacts with the whole and how the system responds to those inputs.

Agency

What is the component’s capacity to act on the other parts of the network or the system as a whole? Can a person interfacing with the product influence the rules of the system? Or, are his potential actions constrained by other aspects of the system? How much freedom does each network component have within the system?

The agency of each actor within the system depends on its role. From a human perspective, agency can describe how much power a user can exert on other parts of the network, versus being limited to specific actions in specific contexts. Different actors will have different amounts of agency at different times.

Opacity

How clear is the network from the perspective of a participant or observer? Are the connections easily visible or are they hidden? The opacity of a network can influence how much agency each actor has and help to create the desired texture.

In our traffic-light example, we see a very opaque system, one where the means of interacting are often completely hidden. It would be easy to interact with the system and still not even know that it exists. In this example, the opacity has a direct impact on a person’s agency, but if the system behaves properly, the texture might still be smooth. Roughness will become apparent if the system misbehaves and nobody can see what is happening.

Reflexivity

How do you know what is happening in the network? How does it inform the different actors, both human and nonhuman, what state it is in and if there are any problems? Feedback and communication is a vital piece of any system.

Reflexivity is the way in which the particular system provides feedback based on states, actions, and behaviors. This is an indication that the rules of the system are enforced. By providing feedback when a component attempts an action the system can let all of its parts know what is happening, if the action was completed, and what the new state looks like. The quality of this feedback is important to crafting the aesthetic of the system. Is it friendly? Verbose? Human readable? All of these things will change the overall feel of the products and services that are part of the network.

These are some possible aesthetic elements we can begin to use to discuss the qualities of a network system. None are inherently good or bad; they are the basis for a common language that lets us discuss the aspects of a network that affect its quality. An opaque network with little agency creates a certain type of interaction, one largely dictated by its owner. A low-opacity network with a lot of agency allows for more flexibility and potential wrangling by the person interfacing with the system.

The types of systems and products described by the above aesthetic language can be understood in two important ways (among others):

1. As a hard system: a system model that is concrete and constructed to achieve an objective. These types of systems are easy to analyze and model because they are generally made up of discrete pieces that each plays a set part, most often actual things that exist in the physical world.

2. As a soft system: a system model that is fuzzy and focuses on the understanding of the system from many perspectives. In this type of model each piece of the system is based on a subjective understanding of the whole, rather than specific objects that exist in the world.

For the type of design discussed in this chapter we are more concerned with soft systems, although both soft and hard must exist in order to fully understand and build a product or service in our networked world.

Soft systems methodology (SSM), a framework for thinking about epistemological systems, gives us tools to help understand an unstructured complex problem through modeling actions and subjective understanding of the situation. Unlike hard systems, soft systems models aren’t about classification; instead the practice seeks to explain different relationships by describing them as they are seen, understood, and acted upon. A single set of objects and relationships could be described in many different ways, each one equally valid from a different perspective. Soft systems have always had a close tie to the way designers work. Peter Checkland, one of the SSM pioneers, said the following in his book Systems Thinking, Systems Practice:

Its rationale lies in the fact that the complexity of human affairs is always a complexity of multiple interacting relationships; and pictures are a better medium than linear prose for expressing relationships. Pictures can be taken in as a whole and help to encourage holistic rather than reductionist thinking about a situation

Design’s tradition of visualization and sketching fit very well with SSM’s tendency toward visualization from the perspective of an actor within the system. In the networked world the designer’s ability to understand, explore, and explain complex interactions between people and machines, and machines to machines, becomes even more important. SSM gives us a starting point to understand how to reframe complex situations through a process that begins by embedding oneself into the situation, expressing what you observe and understand that situation to be, and then creating diagrams that express that understanding. Once the system is visualized it can be compared to observed reality to understand which definition fits best in the given context and what actions one should take to affect the system, described in SSM as feasible and desirable changes. The use of visual tools helps the designers and stakeholders build the same mental model, rather than the ambiguity of individual conceptions.

Tools like this one become a primary piece of the twenty-first century designer’s kit. Making sense of and expressing complex systems of relationships, communication, and feedback lay the foundation for good design decisions when dealing with complex networks, invisible interfaces, and nuanced interactions.

New Tools for a New Craft

Although much of the core design process is fundamentally the same as it was 30 years ago—beginning with exploratory methods including research and sketching, moving through models and prototypes of different fidelities toward a final product—the types of problems we’re trying to solve and the tools we need to explore those solutions continue to change and evolve. New types of products require new types of models and prototypes. Animation, electronics, 3D printing, and interactive programming are all necessary parts of the designer’s repertoire when working with emerging technologies and twenty-first century products.

Tools traditionally thought of as the domain of engineers, data scientists, and hackers are now entering the designer’s toolbox. For example, a designer working with emerging technologies such as sensor networks, data collection, and microcontrollers benefits greatly by learning some basic electronics. Being able to put together a quick prototype by using a platform such as Arduino means that the designer can experiment with the possibilities available to him based on the types of sensors and data at his disposal. Even if the final product will use a different engineering solution, this basic toolset gives designers the capability to model the interactions, data, and physical aspects of a new product at a high level, and with practice, at a detailed level.

Working with large and complex data sets is becoming the norm for designers working on new products. This data can come from custom collectors, such as sensors embedded in products, or from the tangle of information available through web services. When working with large data sets, there is no substitute for working with the data itself. Tools such as Processing or JavaScript and the browser canvas object provide an easy way to start creating rich interactive visualizations from any data.

Rapid fabrication starts to shift industrial design away from being industrial and back to a more artisanal craft. Designers can now imagine a new physical form, model it with traditional tools such as clay, do a digital CAD drawing, and have it fabricated in plastic or metal within a few hours. This facilitates a kind of rapid iteration and prototyping for complex objects that would have been difficult 10 years ago. It also allows for small run production; whereas purely artisan craftspeople could produce only a few objects, and industrial production could only produce high volumes of objects, these new methods make it possible for designers to produce dozens of objects, each the same or slightly different.

These methods can be thought of as a similar process to industrial designers making clay or paper models, or architects using foam-core to make scale models of a new building. None of these things is analogous to the final form, but they are hands-on ways of exploring integral aspects of the design in a fast, cheap, and easy way. Including this in the design process helps illuminate new possibilities and filter out ideas that don’t translate. These are ways of sketching with interactivity, responsiveness, and movement, iterating to a model of the product or pieces of the product.

Along with new tools come new collaborations. The Maker community and local hack-labs, both groups of people who deeply experiment with new technology for creative purposes, are now home to many technologists and designers working together to make interesting and future focused things. These collaborations result in products such as Berg’s Little Printer, the plug-and-play robotics kit, Moti, and DIY home automation tools like Twine. Bio-hack labs are also beginning to pop up, pushing into biology and chemistry, and experimenting with bioengineering in an accessible way. One such group in Toronto, DIYBio Toronto, hosts regular workshops. Companies such as Synbiota, an open source repository for bio-hacking, are forming to support the community.

These are just the beginning, as startups and large companies move into this new space. One of the most successful examples on the market today is the Nest thermostat, which combines innovative physical controls with small screens, microprocessors, and software to add a level of smart automation to the home. A product that started out as a better thermostat is poised to be the hub of a much larger home control system.

How do we begin to work with these new technologies, networks, and systems? There are a few ways to dive in that will help to understand the potential, constraints, and complexities involved.

Experiment

Arduino and similar platforms are easy to find at local stores or online, and they are cheap. Pick one up, find a tutorial, and dive in. Have an idea for a project you’d like to try? Just try it, don’t worry if it seems complicated. Start with the simplest piece. These systems give you all the pieces you need to build network-connected objects.

Learn new skills

If you’ve never programmed before, pick up a JavaScript, Processing, or Ruby tutorial. If you’ve never designed a physical object, get some modeling clay and sculpting tools and try to make some interesting shapes. If you’ve never designed software before, try to map out a flow or design an interface; start with pencil and paper.

Be critical

When you’ve made your first new thing, take some time to think about its qualities using some of the frameworks discussed earlier in this chapter. Use what you learn from this reflection in your next experiments.

Always think about how your new device, software, or system fits into the larger connected world. What possibilities does it create? What potential does it remove? What does it give to people, and what does it take away?

You won’t be satisfied with your first attempt, but design is all about iteration. These types of new skills open many possibilities for your practice as a designer, allowing you to incorporate new technology, processes, and techniques into your work.

Making the Future in Which We Want to Live

The active ingredient of the work is its interface. The interface is unusual because it is invisible and very diffuse, occupying a large volume of space, whereas most interfaces are focussed [sic] and definite. Though diffuse, the interface is vital and strongly textured through time and space. The interface becomes a zone of experience, of multi-dimensional encounter. The language of encounter is initially unclear, but evolves as one explores and experiences.

DAVID ROKEBY ON VERY NERVOUS SYSTEM⁸

David Rokeby used the preceding statement to describe the nature of his Very Nervous System interactive installation. These same words now describe our relationship to an ever-increasing amount of invisible architecture acting around us. The metaphorical handles and buttons that we design into these largely invisible systems will determine people’s ability to comprehend, manage, and benefit from the things we design. Returning to our traffic sensor example, when a hidden sensor at a busy traffic intersection is designed to trigger the lights based on certain physical aspects of a vehicle, the designer of that system needs to decide what types of vehicles are allowed to trigger the lights. Will it work for cars, bicycles, or humans? That choice is a decision that will impact the shape of the urban environment in a way that most people using the intersection will never fully see. How do you indicate the system’s texture, agency, opacity, and reflexivity? Do you add symbols to the road to indicate the existence of a sensor and what will activate it? Do you opt for a different solution entirely because of the needs of the city? These are design problems at a systems scale and are becoming more and more common in the work we do every day. We need to make sure we are arming designers with the tools they need to make these types of decisions intentionally.

Design is a special craft, one that allows us to imagine the future as we would like to see it, and then make the things that will help get us there. Pre-industrial products were the output of a single craftsperson, and expressed their understanding and view of the world. Industrial products represented a move to mass production and consumption, where a designer could envision a product and millions of people could receive an identical object. This was the expression of the collective—the design of objects shaped our environment and culture on a large scale.

As we move deeper into a post-industrial era new products are the expression of the network. Small groups can now cocreate and produce objects at industrial scales, or can create complex objects at minute scales for their own needs. Where pre-industrial objects represented a one-to-one relationship between creator and consumer and industrial objects were one-to-many, post-industrial moves into a many-to-many world. Everybody is enabled to create and consume. With this comes a great freedom, but also a great dilemma. Do all these new objects help us create a better future? Do they represent the world we want to live in? Each new creation warrants a consideration of these questions as we continue to redefine our environment using new technology, and to see the world through our new, networked lens.

This era of post-industrial design brings with it new opportunities and more complex challenges, and we should dive in headfirst.

¹ Cui, Yanqing, Jan Chipchase, and Fumiko Ichikawa. 2007. “A Cross Culture Study on Phone Carrying and Physical Personalization.” Nokia Research, https://research.nokia.com/files/45590483.pdf.

² Copyright Normative, 2013

³ Copyright Normative, 2013

⁴ Rokeby, David. 1982–1984. “Reflexions,” http://www.davidrokeby.com/reflex.html.

⁵ Photos by William Eakin, Liz Garlicki and Risa Horowitz. Image arrray design Mike Carroll. 2003.

⁶ Hannah, Gail Greet. 2002. Elements of Design: Rowena Reed Kostellow and The Structure of Visual Relationships, Princeton Architectural Press.

⁷ Malouf, Dave. 2007. Foundations of Interaction Design. Boxes and Arrows, http://boxesandarrows.com/foundations-of-interaction-design/.

⁸ Rokeby, David. 2010. “Very Nervous System,” http://www.davidrokeby.com/vns.html.