DevOps and Business

Chapter 4. Defining Responsibilities

Once you have established strategic Key Performance Indicators (KPIs) and defined measurable organizational success criteria, it’s time to coordinate individual responsibilities to support the effort. For that, you need to make a decision on how to position technology groups to best enable business outcomes. Every organization has its own optimal path to delivery of its goods and services; your challenge is to understand it and inject or adopt technology to support it.

To determine the best approach for positioning technology groups, you need to truly understand your organization’s mission statement and strategic objectives. In most cases, however, technology focus is not included in a mission statement because most companies are not technology companies (even though they rely on technology to support and grow the business). In this chapter, we discuss assumed responsibilities for technical and nontechnical groups within an organization and the separation between local and strategic decision making.

Technology’s Place Within the Organization

As an example of how technology fits into an organization’s strategy, let’s consider a modern media company whose goal is to obtain the industry leadership in production and distribution of quality digital content. Although the business would not exist without technology, the company’s business objective is not software development; it is the production and delivery of its media content. In this scenario, technology is an enabler of business, and the organizational structure should reflect that.

There are a number of approaches in which technology can be integrated into the overall organizational structure, each with its own set of strengths and weaknesses. Depending on business mission and operational practices, an organization’s team structure and interaction methods will differ from company to company. Different organizational structures reflect different strategies and business models. There is not a “silver bullet” approach, but there are certain operational models that you can consider.

Technology is an enabler; organizational structure should reflect that.

A classic role for technology groups is to be treated as standalone units that report to the CTO and are responsible for their own P&Ls within the organization. This approach is generally taken in organizations whose value delivery processes don’t directly rely on technology. This is the most traditional model, though it often incentivizes the wrong goals. With the technology unit being responsible for generating revenue from internal groups and adhering to its own budget, its KPIs are not to deliver the value to business units, but to optimize its own internal margins. Focusing on group outputs instead of business outcomes will misalign objectives and misrepresent the value of your group.

The first sign of a broken model would be a perception of technology group as a pure cost center. Usually, in compensation, a department’s focus is on its own cost reduction as a measure of value, instead of maximizing the business value that technology enablement brings. Refocusing the group purpose from being a self-sustained department to a support model for different nontechnical groups would make technology a stakeholder in the success of business initiatives, breaking down the isolation and realizing objectives.

Another approach comes from applying services thinking to your delivery process. It has been a common practice, especially in enterprise organizations with large and distributed technical groups, to have a shared services or platform group that’s responsible for building and maintaining infrastructure available for use and consumption to other groups. In this model, the delivery method shifts from pipes to platform,¹ adapting a model that’s been prevalent with the growing availability of web services. Instead of operating in a model in which the value is produced by a provider and consumed by users, it allows both value creation and consumption by users themselves.

We can apply an approach similar to service delivery models to that for nontechnical groups. Creating a business platform, a version of internal Software-as-a-Service (SaaS) collection offering, would shift the focus of the technology group from building and delivering code for consumer-facing initiatives to building and supporting a platform to enable nontechnical groups to create consumer-facing offerings. As I’ve mentioned in previous chapters, one of the biggest draws of DevOps is the promise of speed, and a platform concept certainly plays to that. Furthermore, the up-and-coming trend of “low-code” takes the concept of enabling nontechnical staff to produce consumer-facing applications and tools through graphical user interfaces (GUIs) on top of a provided platform. It reduces the time from application inception to getting the product into customers’ hands. It supports perhaps the most drastic organizational shift from a traditional perspective: the “marketing technology” concept, in which technology is a function of the business unit that it supports, like marketing. In this structure, the group reports to a head of the marketing unit (CMO) because its main purpose is to support the marketing group’s revenue-generating initiatives. It has been explored more and more in the industry and has allowed business units, like marketing, to dictate the platform needs to support their initiative development effort even further.

Choosing an operating model is critical to business success and requires an understanding of business needs to which technology services or a platform should be tailored. But even with an established model, there needs to be a clearly defined set of responsibilities for each technical and business group. These responsibilities will dictate both internal and cross-departmental processes and communication methods.

Decision-Making Structure

Even with an existing culture of shared responsibility and streamlined communication, any decision, proactive or reactive, that requires escalation to another group introduces a delay. Even though core DevOps principles promote a culture of shared responsibility and cross-functional collaboration, decentralizing decision-making by area of domain knowledge and maximum value contribution is essential for shortening time-to-value. Trusting individual groups and departments to make local decisions in support of an umbrella strategy shortens feedback cycles, reduces the human resources, and keeps the decision in the hands of experts.

However, not all decisions need to be decentralized. Critical decisions, such as incident responses, should not carry a decision-by-committee overhead. Conversely, including all of the involved parties in more global, strategic decisions—changes that affect organizational processes, for example—is a good idea. Asking yourself the following five questions will help to identify the type of responsibility for particular decision areas:

• Who is affected by the decision (department, organization, customer, etc.)?

• Does this decision require domain knowledge outside of the primary (local) group?

• How frequently does this type of decision need to be made?

• How quickly should the decision be made?

• What is the risk and cost of making a mistake?

Decisions that have only local impact, decisions that require a single area of domain knowledge, decisions that need to be made frequently, and decisions that require a quick feedback should be entrusted to local decision makers. Note that these are not disqualifying questions. These are merely a baseline for optimizing the process for solving certain types of problems. These questions also refer to immediate impact point, not chain value. I’ve emphasized that every decision should be made in support of shared goals. Based on that, theoretically, “everyone” should always be the response to the first question. Practically, however, there is a distinction to be made between “who is affected” and “who benefits.”

By way of example, a quality-of-life improvement (such as switching operating system from Windows to macOS for a developer) does not directly affect anyone but that developer. However, potential productivity improvement due to the developer’s familiarity with the tools benefits the organization as a whole. In this case, a small local decision contributes to the strategic goal of improved productivity. But even with a potentially high-profile impact, this tactical decision should be made locally by the software team lead without invoking a chain of command or soliciting feedback from other departments.

Similarly, as I discuss in Chapter 5, decision makers from all groups can benefit from cross-departmental domain knowledge. Having the director of technology involved in a new initiative brainstorming session with marketing might uncover technical limitations with an idea, inflating the originally projected time and budget scope, and forcing the group to rethink (or modify) the idea. Use these questions as a preliminary guide and adjust your strategy to best reflect your organizational model and culture.

To reiterate, not all decisions need to be decentralized, but understanding the scope and impact of decisions can help you to determine which groups need to be involved. As part of organizational transformation, you need to establish the guidelines for decision making and educate and empower the involved groups. Those guidelines should define individual responsibilities and communication channels for each group.

IT Responsibilities

Tactical responsibilities, often discussed as part of DevOps transformation, are local to technology teams. These responsibilities might include process changes to improve Mean Time To Detection (MTTD), new tool stacks to build out the delivery pipeline, or restructuring and augmenting teams with skills to support those initiatives. For that reason, the natural allocation of these responsibilities should fall solely on technology groups—not a major shift from well-run traditional organizations.

But ensuring autonomy for those decisions is key to DevOps adoption. The ability to make quick local changes in support of changing requirements allows quicker response and adoption, thereby helping the technology landscape to change at the same pace as business dictates. But the only way to reap the benefits of this approach is to gain an understanding of organizational goals and ensure that all of the local decisions support those goals.

Process implementation

Agile-like methodologies are a cornerstone of successful DevOps adoptions. However, the framework and methodology that is most optimal for the delivery of services differ based on each group’s mode of operation, personnel, and a number of other factors. Organizationally, companies can make a strategic decision to adopt Agile methodology company-wide, but the specifics of implementation should be delegated to individual groups.

If your group is responsible for scheduled, iterative deliveries of incremental features, a Scrum framework might be a good fit. However, if the groups are much more interrupt-driven, dealing with unforeseen issues and requests, Kanban might be a more suitable framework.

Similarly, a decision as to whether developers should participate in on-call rotation and the capacity in which they augment that team should be left to the discretion of the group that understands the immediate pains and best remediation steps.

Personnel decisions

Personnel decisions to support DevOps initiatives should also be delegated to the implementation teams. “DevOps Engineer,” a common title in recent years, has even less commonly agreed upon meaning than the term “DevOps” itself.

DevOps is neither a team nor a role. It’s an operational approach to solving problems as a single, cross-functional unit. As such, much like with IT groups in the pre-DevOps era, you need to define and understand the process and determine what skills would be needed to support it. There are newly formed roles within the teams adopting new methodologies; roles that are becoming essential in the DevOps organization, but are intended to supplement the team with the needed skill sets.

That having been said, when reorganizing the department to embrace the new processes, oftentimes both the structure of the organization itself and distribution of roles may need to change (both from a skill and headcount perspective). When increasing automatic test coverage, for example, a reduction in the QA group might be appropriate in lieu of test automation engineers. Adopting a Scrum framework might require investing in Scrum Master training for leads and managers. The changes can even affect the resource distribution, such as hiring more developers to work on new features because the new architecture and delivery pipeline has allowed for better parallelization of tasks. Those are just a few examples of personnel changes that might be required to successfully realize DevOps adoption plans. However, much like with process implementation, local understanding of the problem can help provide a better solution for the desired outcome than a generic strategic mandate would.

Technology selection

Although DevOps, despite common misconception, is not purely about tools, selecting the appropriate tools is an important step in building a successful DevOps organization.

There are a number of valid considerations for selecting a product over its competitors, or as an addition to (or replacement of) part of an existing architecture:

• Documented benefits of technology to improve aspects of your business

• Ability to scale better with company growth

• Technology giving a competitive advantage

• Taking advantage of new and cutting edge technology

The challenge with these criteria is that each can also be presented as a list of invalid considerations without proper context and domain knowledge. As an example, an initiative like a cloud migration needs to be backed by performance, growth planning, and cost-benefit analysis as it relates to its own architecture. Perceived benefits need to be validated by subject matter experts and tied back to organizational KPIs in order to determine the validity of the assumption and the value to the organization.

In a DevOps organization, the responsibilities of technology groups do not differ much from those of traditionally structured organizations. The group should be responsible for local process implementation, hiring needs, and technology choices. A major DevOps culture shift lies in trust and enablement that the organization provides to its technology backbone. Business decision makers need to both trust that all the local decisions will be done in support of the umbrella strategy, and they need to provide appropriate context to allow technology teams to make educated decisions. It’s the responsibility of the organization to cultivate both.

Business Groups Responsibilities

Business groups must define the context for an organization’s strategic vision. Context is important. It establishes a common baseline for the communication. Without it, you’re creating disjointed channels for data sharing, each with its own context. When your message is delivered in one context but received in another, it creates a set of false expectations for both parties. Without full understanding of strategic goals and without being kept in a loop when those goals shift or change, IT cannot effectively plan its support tactics and will fail to deliver value.

Strategic direction

With implementation being delegated to individual groups responsible for their segment of the delivery process, one of the primary responsibilities for business units is to define desired outcomes to which individual group goals should align. For IT, choosing the appropriate processes and tools to accomplish the goals is important. However, to achieve that, the IT team must answer a question: “What is the goal?” And the business group needs to be the one providing the answer. After the business group defines and (perhaps more importantly) communicates its success measures, each group can define local KPIs in support of those shared goals.

One of the major breakdowns occurs when high-level decision makers are unable to separate strategic decisions from tactical decisions. Let’s examine real directives that were given by nontechnical executives and leaders to technical teams for execution:

Bad: “We need to buy Ansible Tower and automate everything.”

Bad: “We need to move to the AWS cloud.”

These examples illustrate poor strategy communication. Neither of these statements frame a problem that needs to be solved or an outcome that needs to be reached. Instead, they attempt to define a solution that needs to be implemented. Purchasing a particular tool or switching a platform is not a goal; it’s a means to achieve a goal that has not been communicated. Requirements like these raise more questions than they answer: why a particular product and not another? What is “everything”? And perhaps the most important question, “What is the goal we are trying to accomplish by these changes?” is left unanswered.

Better: “We need to automate our testing.”

Better: “We need to scale our application better.”

These requirements are formulated a little better in the sense that they move one step up from tool selection, but they still leave a lot to be desired. They still provide solutions instead of defining problems. Do we need to scale horizontally or vertically? Why is concentrating on test automation is a priority? Do we really need to scale or is performance a problem we’re looking to solve? And if we do need to scale, what are the metrics for projected growth? Ultimately, the same question (“What is the problem we are trying to solve?”) is not answered. Instead, a solution is requested without defining an actual problem or a goal.

Best: “We need to deliver product 30% faster to consumers.”

Best: “We need to reduce our operational expense by 20% and prepare for ten times growth over the next year.”

The preceding statements are an example of what properly formulated strategic goals looks like. The desired organizational metric that needs to be improved is clearly communicated, and the means to accomplish the goal is left for local decision makers, allowing them to define local KPIs in support of the strategic goal.

One of responsibilities of a DevOps leader who tries to bridge the gap between business and IT is to make sure the information flow is accurate and, most important, meaningful. Making sure that strategic directions are measurable and actionable is the first step in establishing that feedback loop. The next step is ensuring that data flow is context complete and continuous.

Transparency

Because IT KPIs need to support specific objectives or initiatives originating in one of the nontechnical groups, those objectives, along with changes to them, need to be clearly communicated down to supporting groups. And that information needs to be complete and timely.

Information completeness

One of the common landmines in communication is the assumption that a piece of information, often outside of the receiver’s domain, is not important to context. Contractual requirements and business operation compliance are good examples of domains that are often kept away from IT.

Suppose that, as part of improving the delivery pipeline for managed customers, you plan a move from Subversion to Git to simplify your Continuous Integration (CI). In choosing a tool, GitHub seemed like an obvious choice. Not only is your team already familiar with the tool, but the SaaS nature of GitHub offers the lowest cost option. A no brainer.

But, what if you knew that each customer’s contract contained a standard legacy clause stating that no part of their code or data can leave your network? This piece of information, seemingly unrelated to the initiative at hand, would immediately put a restriction on tool choices and eliminate SaaS options at selection process.

When sharing information between groups, err on the side of more information being better. Naturally, there is information that shouldn’t be shared with everyone due to its sensitivity, but shifting communication models to allow for more transparency, within legal and security guidelines, should reduce decisions made based on incomplete data.

Information timeliness

Another often-encountered issue in communication is time sensitivity. DevOps is all about improving velocity of time-to-market initiatives, but with increased velocity comes increased demands for timely information exchange. The model where technology is the primary backbone of businesses is a relatively new concept, and at times, nontechnical groups either don’t have “notify IT” on their checklist or else do so too late in the process. The reason is not malice. It is the lack of understanding of impact a business change has on technology plans.

Technology working in support of business initiatives is a two-way road. Just as IT must be aware of the effects of changes to the technology stack and processes, business units need to be mindful that technology changes might be required to support changes in business operations.

Perhaps the most common example is the marketing group notifying the operations group about a projected surge of traffic to the website. From a PR perspective, being featured on the front page of a major news site is a big success. From a technology perspective, it’s a disaster that pushes the system over the projected capacity, causing a poor user experience and forcing the entire team into firefighting mode in order to accommodate the traffic spike. Such issues can be avoided with adequate advance notice, allowing some time to prepare to scale with the surge.

The biggest challenge with enforcing business-specific responsibilities is getting buy-in from respective business units to change their processes, which is not always easy, even considering the value the change provides. However, the change has to happen in order for IT to provide the most value to the organization.

With increased velocity of change comes increased demand for timely information exchange.

Separating decision making and responsibilities and aligning local outcomes to shared objectives is crucial in the positioning of your technology team as a supporting pillar for the success of your business. To successfully gain (and maintain) an understanding of strategies and priorities, you need to implement a combination of pull and push techniques at different points of your process to obtain complete and timely information on a consistent basis. In Chapter 5, we talk about establishing clear and reliable communication channels and feedback loops at every step of the delivery life cycle.

¹ Sangeet Paul Choudary, “Why Business Models Fail: Pipes vs Platforms,” Wired, October 2013, https://www.wired.com/insights/2013/10/why-business-models-fail-pipes-vs-platforms.