Integrating Agile and UX design

Of all the modern methodologies for software development, perhaps none is so important, impactful, challenging, and rewarding as Agile. Much ink and many pixels have been devoted to documenting its various permutations—from Scrum to extreme programming (XP), feature-driven development (FDD) to Kanban, and everything in between. Suffice it to say, this article will not cover such a scope. It will, however, attempt to highlight Agile’s most basic tenets and discuss how it fits—or does not fit—with UX design.

What is Agile?

Agile is an umbrella term encompassing a variety of development methodologies that have at their core a commitment to highly collaborative, multi-functional, self-organizing teams. Agile’s minimal overhead and focus upon working software as a measure of progress contrasts with the heavy requirements and upfront documentation of waterfall development. Waterfall gets its name from the sequential tasks that drive the process: discovery and requirements gathering lead to design and detailed documentation, which lead to development and testing, and so on, each one cascading down into the other. However, it is rare that the fluidity and adaptability suggested by the waterfall metaphor is maintained. Those traits are actually more appropriately associated with the philosophy that underpins Agile.

In 2001, a group of 17 software engineers wrote the now famous Agile Manifesto, which describes, in four simple statements, the overall approach:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work, we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

What does Agile development look like in practice?

In Scrum, one of the most popular Agile product management frameworks, the team breaks down a digital product’s features and functions to create a backlog of user stories. User stories describe, from a user’s perspective, a particular function of the software and the benefit of that function. For instance, a story for a Dropbox user might be, “As a power user, I can use keyboard shortcuts to quickly navigate my files.” A select group of these stories is then built in a short development cycle, usually in a two-week increment, also known as a sprint. The Scrum team, which consists of a product owner, Scrum master, and development team—ideally 6-9 engineers—uses daily stand-up meetings to stay coordinated and focused. At the conclusion of a sprint, the team produces working software that’s demonstrated to stakeholders in a review meeting.

Importantly, the team also holds a retrospective to go over what went well, and what did not, so they can continually improve. Then, as needed, the process starts again with the next set of stories and the next sprint. To be successful, Scrum requires a high level of transparency, trust, and commitment between team members. Agile methods, like Scrum, work well for developing digital products because they provide adaptive, iterative methods for handling the complexity that comes with the infinitely malleable material that is software. But, at the same time, Agile is not necessarily all encompassing when it comes to product creation. Because it eschews upfront planning for learning as you go, it can create software that lacks cohesiveness from a user perspective.

• To learn more about Agile methodologies, read Learning Agile, by Jennifer Greene and Andrew Stellman.

UX and Agile

Of course, on the other side of the digital product creation equation is UX design—that intuitive, empathetic, and human-centered practice with its focus on the user experience as the driver. There’s no question that design-driven innovation has produced great value for companies in recent years. From Apple to IBM, Kaiser Permanente to Nike, market leaders have used design to innovate their products and services. And design-led companies like these have outperformed the overall market by a significant margin—beating the S&P 500 by 211% over a ten-year period, according to the Design Management Institute’s Design Value Index Study.

There can be a tension between design and Agile delivery. In his foreword to Jeff Patton’s book on User Story Mapping, interaction design pioneer Alan Cooper writes: “In Mary Shelley’s famous science-fiction novel, Frankenstein, the mad Doctor Frankenstein builds a creature from the disparate pieces of dead humans and brings the creature to life with the then-new technology of electricity. Of course, we know this is not actually possible. You cannot create life by sewing together random body parts.”

“Yet this is what software developers attempt to do all the time,” Cooper continues. “They add good features to software, one at a time, and then wonder why few users love their product. The heart of the conundrum is that developers are using their construction method as design tool, but the two are not interchangeable.”

So, should software creation be design-driven or Agile-driven? Does design need to adapt to a world that is increasingly Agile, or does design’s strong value proposition—typified by the outsized valuations of design-centric companies—make it a natural product lead? “The premise that design has to…kneel beneath this juggernaut that is Agile, I’m not sure that’s necessarily the right framing,” says designer, author, and speaker, Cennydd Bowles. It’s probably incorrect to suppose that one of these powerful methods has to conform to the other. For effective, successful software creation, we need to figure out how the two can work side-by-side. But how does this adaptation take place? How do UX design and Agile fit together in an effective way?

• To learn how to integrate Lean UX and Agile, read Lean UX, 2nd edition, by Jeff Gothelf and Josh Seiden.

The miracles of iteration

As you might expect, the relationship between Agile and user experience is continually evolving, as designers and developers find ways to adapt and better work together. Where the two approaches intersect, there are many areas in which they can complement one another. “They’re both desperately iterative,” says Bowles. “They both recognize you’re not going to get it right the first time. Designers are very comfortable with it. We know that. That’s part of the design process. That’s why you have critique. That’s why you have prototyping, and so on.” For designers, making ideas tangible is absolutely crucial, as are the iteration cycles needed for testing and refining those ideas. Design has a bias toward making things, and prototyping is where this all begins. It’s important that designers see how something is working and how users are reacting to it. Through the act of prototyping, designers can view the evidence and see where they got it right and where they got it wrong. Then, based on that learning, they can refine and hone the design. This same iterative philosophy is hardwired into the Agile approach. We learn by testing usage in the real world by real people, and through iteration based on that learning, we progress toward a solution.

But while an iterative approach to software creation is a shared value, transparency throughout that process is much less so. Agile developers are very comfortable externalizing their work-in-progress and revealing solutions early in the process. They prefer to ship code as soon as possible, in order to test assumptions, get feedback from real users, and learn from that feedback. Designers, in contrast, are usually less inclined to reveal work-in-progress. In this respect, design is changing and adapting. “Designers have to become a bit more comfortable with releasing things,” says Bowles. “Releasing, I mean now in the delivery-of-products sense, but also just conceptually, releasing control of things they may not be 100% happy with. That’s tough for designers. That’s kind of an ego check for a lot of designers. But that’s probably an OK thing.”

Designers have to be mindful and willing to take feedback as part of the process. How does this design actually perform in the real world? Does this actually meet the user need the team anticipated? Designers need to have the flexibility to learn. “So, it’s a relaxation of control on both sides,” says Bowles, “but then also being willing to absorb what you learn from that in the design process. I think designers sometimes have been not great at either of those. I think Agile is a nice balance, a nice way of sort of forcing that humility, to an extent.” While designers can learn much from Agile development’s focus on transparency, there are elements of software creation that designers can champion in an Agile environment: one of these is research.

• To learn how to create and iterate on designing interfaces in Agile environments, watch Effective UX Design with Patterns: Build and Leverage a Pattern Library for Rapid Iteration and Prototyping, from Dani Nordin.

Research’s last stand?

Is user research something we do before, during, or after creating a digital product? The Agile process does not favor much in the way of upfront research or discovery, preferring learning through continuous experimentation. Design, in contrast, has a preference for research at the very start—establishing a basic understanding of users and the target market—for informing product strategy and ongoing decision-making. Of course, there’s no reason a product team can’t insert design research upfront, ahead of embarking on the product development. This can allow for some ethnographic study, interviews, and other pre-development activities that can help inform digital product strategy and ideation. But this can be firmly at odds with an Agile approach. The trick is doing enough research, talking to users, etc., to inform the decision-making, but not so much that it wastes valuable time.

Going Lean

Another, complementary approach involves integrating user research into the iterative production process, which is the tactic taken by the increasingly popular Lean UX. This Agile-friendly design approach takes key ingredients from the Lean Startup method and is driven by core principles focused on maximizing value and minimizing waste. Ongoing research gives designers the opportunity to check hypothesis quickly, and test a Minimum Viable Product (MVP) with customers to validate, or invalidate it, as the case may be. Since Lean UX research is collaborative and distributed across the entire team, everyone can have exposure to it and develop a shared understanding. Ideally, the team will talk with customers at regularly scheduled intervals, perhaps once a week, and receive feedback. The formal research process and method is considered scope that can and should be altered as needed. In this way, Lean UX is pragmatic and utilitarian when it comes to soliciting customer and user feedback. However, in order for such an approach to succeed, team collaboration and shared understanding is critical.

Collaboration and co-design

Integrating UX design and Agile requires that user experience practice not be limited just to the designers, but rather that it become a responsibility of the team as a whole. Successfully making this happen requires a collaborative balance: a designer should remain the point person for the output, but many team members help shape and implement. “It’s important that designers retain control over the design process … [You need] someone accountable and someone actually trained and literate to make those decisions,” says Bowles. “Ultimately, the principle is that any problem is a team problem. If code review is proving to be an enormous backlog and choke point for the process, it’s everyone’s problem to fix that, including design [and] product management. It’s not just an engineering problem. Likewise, if we’re not able to get enough [design] research to validate that this is actually the right thing to be building, that’s for the team to solve. That’s not just on the design manager to figure it out.”

“It’s that sharing of responsibility that then, in turn, creates a feeling of shared accountability for the quality of the product. No team should be happy if they’re pumping out 70 points of Agile delivery every week on complete BS that doesn’t matter to the user. … No engineering team should be happy about that. They should be concerned. I think having that kind of shared accountability and shared responsibility is probably the closest thing I’ve got to a golden shining key for how to design in those situations,” says Bowles.

Increasing design literacy

Even on a collaborative, co-located team, designers won’t be involved in all UX decisions. Ultimately, engineers will make design-related choices every day as they construct the product. So, a great way to improve the quality of the product is to increase the design literacy of the engineering team. Designers need to expose developers to and create a shared understanding of design principles. This goal can be achieved via any number of informal methods—from giving lunchtime tech talks to actively discussing design decisions to creating a design resource library. A tech talk at lunch can be accompanied by a lightweight presentation or a group discussion about a targeted design topic. While it’s great to have a design reference library, it’s not necessary to have a dedicated reading area. You might just buy some books to leave in the team’s shared space.

Welcoming input

The design process should be inclusive of everyone on the team. Designers should welcome input, even if (especially if) it’s input with which we disagree. “Probably the best question I [could] ever ask as a designer is ‘what do you think’?” says Bowles. “That draws your developers and your [product managers] into the conversation. Through that, they have buy-in to the solution, if they like it. But also, they’re exposed to the design process a little bit more.” The question is an excellent entry point from which you can draw the team into design discussions. “The minute you shut down, you say, ‘Well, I’m a designer. You’re an engineer. Go back to your work and leave this to me.’ Then you shut off those avenues for a potential benefit,” says Bowles. “So, it’s about retaining that openness and curiosity to accept ideas from wherever they may be.” This method of welcoming design input follows Agile’s primary axioms, valuing individual relationships over process, and conversation over documentation. We can take a similar approach in our attitude toward design deliverables, and emphasize communication rather than artifacts, which, unsurprisingly, is core to Lean UX.

• To learn how to collaborate and design better together, read Chapter 2 of Collaborative Product Design, by Austin Govella.

Waste not, want not?

Design processes tend to favor the creation of a discrete set of deliverables—whether it’s research, information architecture, interaction design, user interface mockups, or even prototyping—prior to engaging engineering in a substantial way. But maintaining this approach can be difficult, if not impossible, when collaborating in an Agile environment. “How do you take all those [design] tools and apply the most relevant ones at the most appropriate time at the appropriate level of effort?” says Jeff Gothelf, an organizational designer and co-author of Lean UX: Designing Great Products with Agile Teams. “You’re eliminating the waste in your design process, and you’re moving away from being in the deliverables business.” By focusing on the communication of design ideas rather than specific deliverables, Lean UX streamlines the process to better match the flexible Agile environment. “I think, probably, the most important [tenet of Lean UX] is the minimum viable conversation,” says Gothelf. “What I mean by that is, as a designer, what’s the most important thing you need to communicate next? Who do you need to communicate it to? Then, what’s the least amount of work that you need to do to make that communication take place? I think if there were a Lean UX commandment, that’s really it.”

Of course, this doesn’t mean that a Lean UX process is devoid of deliverables. It means you only create the design artifacts that are required to move the conversation forward—this might be a quick sketch, a whiteboarding session, or even a prototype, depending on whom you’re talking to. While favoring conversations over deliverables for the sake of themselves is a great example of design adapting to an Agile world, it does raise the question: just how lean do we need to get?

In praise of inefficiency

The flip side of the coin is that sometimes it can be difficult to identify what’s wasteful in the design process. Sometimes inefficiency can be a good thing. As valuable as communication of work-in-progress can be, there’s also value in creating artifacts and products that set the right conditions for design insight and inspiration. Just as deep upfront research and ethnographic techniques of study and observation can reveal previously unknown aspects of user behavior, so, too, can seemingly superfluous exploratory design activities set the stage for an “aha” moment. It’s a balance, of course, and in our quest to be Agile compatible, designers might be careful not to remove the elements of practice that inspire and invoke our intuition—providing the opportunity to play with ideas irrespective of the march to production.

Bring the vision

Because the Agile development process favors incremental building, engineering teams can focus overly so on short-term output. It’s all too common for a holistic product vision to disappear in the rhythm of sprints, in which the team creates discrete components, moving the product forward in releasable chunks. What can designers do to help ensure the value of an overall product vision is not lost? Having a strong design vision can bring additional clarity to product teams. “You absolutely need that in conjunction with the Agile process. … Without that vision, that North Star, you do get that soulless kind of product. It doesn’t hang together. It doesn’t have that coherence,” says Bowles.

Developing and preserving a design vision is critical to the creation of excellent software. But product managers can easily get drawn into managing backlogs and Agile throughputs. And a singular focus on delivery can result in punting potential UX problems down the road—putting these decisions off because delivery is the primary concern—and finding no sensible resolution when you reach the end. Without a coherent vision for the end state of the product, we risk creating the isolated pieces of a final product that does not work in its totality. It’s tricky to balance between vision, figuring out those elements upfront, and incorporating what you’re learning through the delivery process. You may find that your initial assumptions are incorrect. And, in that case, it makes sense to revisit your strategy process. Expect that teams will have to take some time out on a regular basis to check the product vision and strategy.

Agile / UX in action: TweetDeck

In May 2011, Twitter acquired TweetDeck for more than $40 million in cash and stock in a defensive acquisition that kept the aforementioned application—popular with Twitter power users—out of the hands of third-party app developer UberMedia, and brought the product under Twitter’s control. But it was not immediately clear what Twitter would do with the team. Cennydd Bowles was hired as a senior designer to work on TweetDeck. “My job was to try and bring design into that scenario and help kick-start that project again,” says Bowles. “I actually think we did a great job. … That was easily the best Agile team I’ve been in. What really mattered there, I think, was that division of responsibilities, but also, again, that shared responsibility between engineering, product, and design.”

Collaboration and a team responsibility for design enabled the Tweetdeck team to operate closely together. “One of the developers who was inputting the most into the design process was actually a back-end engineer. … He was very interested in UI, and he had a feel for it. So, he was always chipping in things. They may not [all] have been great suggestions, but even if five of his ideas would not work, one of them [might] actually be really promising.”

In his blog post “Designing the new TweetDeck,” Bowles describes how the team’s iterative design process worked hand-in-hand with Twitter’s Agile development, using prototyping to test the software quickly. “After an extensive design review of the app, we sketched dozens of new interface ideas, iterating quickly toward stronger solutions. We then created Photoshop mockups to fill in the detail, before moving into our final design platform, a high-fidelity prototype. This prototype (built from scratch using HTML, Sass, JavaScript, Mustache, JSON, and our internal prototyping tool Protobird) allowed us to tweak the final important details in the browser. With this new prototyping platform in place, we could quickly test out ways to improve the app and ship future improvements faster.”

TweetDeck’s improvements were well received by users, and the application’s reviews improved measurably on the Mac App Store. The combination of user experience and Agile delivery enabled them to create a great product. “That worked really well, to be honest, and I look back at that as sort of a high point in my career in terms of delivering real product turnaround in an Agile environment,” says Bowles.

What’s next?

In the near future, we’ll continue to see a greater level of Agile / UX design integration maturity. This will, no doubt, vary based on company size, type of product, etc. Areas of particular challenge will include scaling these methods for the enterprise, where the sheer size and complexity of software can be difficult to manage.

Bowles sees push back against the empiricism of Lean Startup and Lean UX. “There’s going to be some kind of backlash against Lean Startup. I mean that’s already begun, to be honest,” says Bowles. “Lean Startup has become this dominant ideology, and every company now thinks they’re unique in operating under Lean Startup principles. There’s going to be some kind of reaction against that. Probably a move to slightly more upfront planning, and designers are going to be central to doing that. … There’s going to be something there around repositioning things, like prediction and intuition and deep research into that process, which I think is good news for designers.”

Agile and UX Design are not just ways of creating software, but also, cultural mores—the customs and conventions of work life. And, as these methods converge and mature together, they are establishing new ways of working.

Additional resources

Lean UX: Designing Great Products with Agile Teams, by Jeff Gothelf and Josh Seiden

Learning Agile: Understanding Scrum, XP, Lean, and Kanban, by Andrew Stellman and Jennifer Greene

User Story Mapping, by Jeff Patton

“Agile and the Horizon Effect,” by Cennydd Bowles