Chapter 4. Stage 1: Understand

The first stage in achieving effective financial governance is to understand your current cloud usage and therefore identify the gaps between your current position and effective financial governance.

To take this step, it is necessary to gather data to answer the following questions:

  • What is being spent and how is that split across different services?

  • Who is responsible for that spending?

  • How does that spending relate to business objectives or value creation?

In a traditional on-premises infrastructure, data processing costs cannot be easily categorized into infrastructure and data processing buckets. However, in the cloud, billing is a mixture of on-demand infrastructure creation and usage-based service charges—and although cloud providers typically present a single bill for all usage at month end, a cloud platform can (if the right systems are used) break these down at a much more granular level to give very specific levels of traceability.

Financial Governance Tools Provided by Cloud Service Providers

All cloud service providers have billing dashboards that you can use to understand the costs across the cloud estate. Typically, these tools will allow infrastructure managers to report on costs associated with running the compute, network, storage, and services that make up their cloud environments.

The default dashboard views provided give you a useful “at a glance” understanding of where costs are being incurred. Usually, the out-of-the-box functionality allows for the following data to be revealed:

  • The historic trend in monthly costs. This allows for broad-brush visibility of increases or decreases in costs and can be used to map changes in the environments to changing costs.

  • Component costs as a share of total cost (compute, database, storage, etc.). This allows for visibility of high-spend areas or changes to specific infrastructure costs over time.

  • A forecast for the current monthly bill.

Cloud service providers also maintain cost calculators that allow infrastructure managers to estimate the cost of ownership of services prior to deployment. Anticipated usage of cloud infrastructure for new or growing requirements can be mapped into the calculator to give an estimated service cost.

Managed service providers (MSP) or service providers who utilize multiple cloud accounts can combine billing data into a single dataset.

These tools are an invaluable aid to infrastructure managers looking to understand the detailed costs of their cloud environments at a technical level. This understanding, however, is initially limited to oversight of infrastructure. Out-of-the-box billing tools are focused on the usage of cloud components and provide a picture of what is being spent without the context of data specific to the services running on the infrastructure.

The Importance of Tagging

To understand, control, and optimize cloud infrastructure, standard tagging that persists across the cloud estate is required as the means of defining business and technical usage of components. Furthermore, tagging enables the use of automated tools to improve financial, operational, and security governance.

Note

Tagging in Cloud Platforms

Cloud platforms allow you to create infrastructure and use services in an ad hoc, on-demand fashion, in many situations in an entirely automated manner. This means that traditional approaches of asset tracking in a manual register are no longer viable. Cloud platforms mitigate this gap by allowing metadata known as “tags” to be associated with elements as (or after) they are created. Multiple tags can be used to understand the purpose of the item.

Cloud service provider, third-party, and custom reporting all require the user to manage tags in a standard way to allow for clarity on usage and cost. This is the cornerstone of good practice. Regardless of the size of the estate, tagging should be standardized to give the appropriate visibility to all stakeholders. A standard set of cloud tags might contain the following:

Environment

Identify production versus UAT/Dev environments

Service

Identify which service this component is part of (should be multitiered in complex applications)

Function

Identify what this component does

Technical/service/business owner

Identifies the person or department that manages each aspect of the component or service

Operational tags

Used to automate shutdown or other desirable technical functions relating to the automation of the service

Financial Governance Tools Provided by Cloud Management Platforms

The limitations in the reporting offered by cloud service providers has led to many companies creating their own systems to interpret and display the data in a more business-accessible manner. As we discussed earlier, many third-party cloud management platforms have been created to fill this gap and to provide management and configurable financial reports based on cloud service provider data.

Use of these services is generally regarded as essential for any company looking to effectively manage billing for anything beyond the most basic cloud platform.

Using a standard tagging structure (as just described) embedded in the provisioning process allows third-party reporting services such as Cloudability to reveal technical and business insights into spending across the estate.

Having a business-centric view of billing, clustered by applications, customers, or lines of service, will feed into strategic thinking and decision making. Reporting targeted at the business or service owners as well as the technical management moves costs out of the technical sphere of influence and into the business lines that manage the services. Being able to show the value, or lack thereof, in any hosted application or service allows for accurate decision making in strategic planning for success.

Previously complex multiaccount setups or multicloud platforms in which services or customers are spread across multiple domains or service providers can be consolidated by using standard tagging and third-party tooling, again allowing for a view of costs by application, customer or service in a single-pane dashboard, such as that shown in Figure 4-1.

Cost breakdown of a typical application stack
Figure 4-1. Cost breakdown of a typical application stack

These tools drive a business-centric view of the traceability of costs, as illustrated in Figure 4-2.

Cost data shown against business function
Figure 4-2. Cost data shown against business function

They also encourage users to begin considering predictability of costs, both in the short and long term and these tools provide insight and reporting at both levels:

Short term

What do we expect this month’s bill to be, allowing cash flow to be planned and action to be taken if higher than anticipated?

Long term

How have costs varied over time and can those trends be used to predict future costs?

Tools in this category will form the basis of a good financial governance process for a general cloud platform; however, this does not come out of the box. There will be work to do to ensure that your cloud systems are effectively tagged and the tools configured to understand that tagging strategy.

Financial Governance Tools Provided by Cloud-Native Data Platforms

Generic cloud management tools will do a good job of extracting business intelligence from the raw data supplied by the cloud platforms. However, these are aimed at general business use cases, rather than those specifically related to a big data business.

Cloud-native data platforms will sit on top of your cloud platform to provide big data-specific intelligence and management. In real terms, this means that a layer of control above that provided by the cloud platform is added. As a simple example, rather than having to create two separate clusters to run two queries so that usage could be easily traceable, a cloud-native data platform will create a single shared cluster and track the partial usage by each query.

These data platforms will also provide an enhanced level of reporting, allowing you to understand the costs at a much more detailed level. Typically, they move the traceability to thinking in terms of people (who is executing the query) and workload (what that query is being executed for), as demonstrated in Figure 4-3, rather than infrastructure or services.

Examples of reporting of costs per user business activity on Qubole platform
Figure 4-3. Examples of reporting of costs per user/business activity on Qubole platform

This means that you can achieve a very accurate view of the cost of all pieces of data analysis and execution. This offers some real advantages when looking at financial governance:

  • Costs can be related directly to the business value being, or expected to be, achieved.

  • Costs can be tracked and related to budgets or recharged to departments.

  • Future costs can be predicted at a much more granular level, allowing for up front assessment on the cost/value decision on activities.

Get Financial Governance for Data Processing in the Cloud now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.