Go for DevOps

Go for DevOps

by John Doak, David Justice
Released July 2022
Publisher(s): Packt Publishing
ISBN: 9781801818896

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Achieve reliable release automation and get zero troublesome notifications on your release day.

Purchase of the print or Kindle book includes a free eBook in the PDF format.

Key Features

  • Develop the skills to build command-line tools to control thousands of nodes
  • Use Go to create Terraform providers and GitHub actions and extend Kubernetes
  • Gain the knowledge to build DevOps workflows that are understandable, scalable, and safe

Book Description

Go is the go-to language for DevOps libraries and services, and without it, achieving fast and safe automation is a challenge. With the help of Go for DevOps, you'll learn how to deliver services with ease and safety, becoming a better DevOps engineer in the process.

Some of the key things this book will teach you are how to write Go software to automate configuration management, update remote machines, author custom automation in GitHub Actions, and interact with Kubernetes. As you advance through the chapters, you'll explore how to automate the cloud using software development kits (SDKs), extend HashiCorp's Terraform and Packer using Go, develop your own DevOps services with gRPC and REST, design system agents, and build robust workflow systems.

By the end of this Go for DevOps book, you'll understand how to apply development principles to automate operations and provide operational insights using Go, which will allow you to react quickly to resolve system failures before your customers realize something has gone wrong.

What you will learn

  • Understand the basic structure of the Go language to begin your DevOps journey
  • Interact with filesystems to read or stream data
  • Communicate with remote services via REST and gRPC
  • Explore writing tools that can be used in the DevOps environment
  • Develop command-line operational software in Go
  • Work with popular frameworks to deploy production software
  • Create GitHub actions that streamline your CI/CD process
  • Write a ChatOps application with Slack to simplify production visibility

Who this book is for

This book is for Ops and DevOps engineers who would like to use Go to develop their own DevOps tooling or integrate custom features with DevOps tools such as Kubernetes, GitHub Actions, HashiCorp Packer, and Terraform. Experience with some type of programming language, but not necessarily Go, is necessary to get started with this book.

Publisher resources

Download Example Code

Table of contents

  1. Go for DevOps
  2. Contributors
  3. About the authors
  4. About the reviewers
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Share your thoughts
  6. Section 1: Getting Up and Running with Go
  7. Chapter 1: Go Language Basics
    1. Technical requirements
    2. Using the Go Playground
    3. Utilizing Go packages
      1. Declaring a package
      2. Importing a package
      3. Using a package
      4. Package name conflicts
      5. Packages must be used
      6. A Go Hello World
    4. Using Go's variable types
      1. Go's types
      2. Declaring variables
      3. Variable scopes and shadowing
      4. Function/statement variable must be used
    5. Looping in Go
      1. C style
      2. Removing the init statement
      3. Remove the post statement too and you have a while loop
      4. Creating an infinite loop
    6. Using conditionals
      1. if statements
      2. else
    7. Learning about functions
      1. Returning multiple values and named results
      2. Variadic arguments
      3. Anonymous functions
    8. Defining public and private
    9. Using arrays and slices
      1. Arrays
      2. Slices
      3. Extracting all values
      4. Understanding maps
      5. Declaring a map
      6. Accessing values
      7. Adding new values
      8. Extracting all values
    10. Understanding Go pointers
      1. Memory addresses
      2. Function arguments are copies
      3. Pointers to the rescue
    11. Getting to know about structs
      1. Declaring a struct
      2. Declaring a custom type
      3. Custom struct types
      4. Adding methods to a type
      5. Changing a field's value
      6. Changing a field's value in a method
      7. Constructors
    12. Comprehending Go interfaces
      1. Defining an interface type
      2. Important things about interfaces
      3. The blank interface – Go's universal value
      4. Type assertion
    13. Summary
  8. Chapter 2: Go Language Essentials
    1. Handling errors in Go
      1. Creating an error
      2. Using an error
      3. Creating named errors
      4. Custom errors
      5. Wrapping errors
    2. Utilizing Go constants
      1. Declaring a constant
      2. Enumeration via constants
      3. Printing enumerators
    3. Using defer, panic, and recover
      1. defer
      2. panic
      3. recover
    4. Utilizing goroutines for concurrency
      1. Starting a goroutine
      2. Synchronization
      3. WaitGroups
      4. Channels
      5. Sending/receiving
      6. select statements
      7. Channels as an event signal
      8. Mutexes
      9. RWMutex
    5. Understanding Go's Context type
      1. Using a Context to signal a timeout
      2. Honoring a context when receiving
      3. Context in the standard library
      4. Context to pass values
      5. Best practices
    6. Utilizing Go's testing framework
      1. Creating a basic test file
      2. Creating a simple test
      3. Table Driven Tests (TDT)
      4. Creating fakes with interfaces
      5. Third-party testing packages
    7. Generics – the new kid on the block
      1. Type parameters
      2. Using type constraints
      3. We could do better with constraints
      4. Current built-in constraints
      5. Type constraints with methods
      6. Adding type parameters to struct types
      7. Specifying the type when calling a generic function
      8. Gotchas to watch for
      9. When to use generics
    8. Summary
  9. Chapter 3: Setting Up Your Environment
    1. Technical requirements
    2. Installing Go on your machine
      1. macOS installation using the package installer
      2. macOS installation via Homebrew
      3. Windows installation using MSI
      4. Linux
      5. Other platforms
      6. A note on Go compiler version compatibility
    3. Building code locally
      1. Creating a module directory and go.mod file
      2. Updating a module when adding dependencies
      3. Adding a hello world
      4. Running our first program
    4. Summary
  10. Chapter 4: Filesystem Interactions
    1. All I/O in Go are files
      1. I/O interfaces
    2. Reading and writing to files
      1. Reading local files
      2. Writing local files
      3. Reading remote files
    3. Streaming file content
      1. Stdin/Stdout/Stderr are just files
      2. Reading data out of a stream
      3. Writing data into a stream
    4. OS-agnostic pathing
      1. What OS/platform am I running?
      2. Using filepath
      3. Relative and absolute pathing
    5. OS-agnostic filesystems
      1. io.fs filesystems
      2. embed
      3. Walking our filesystem
      4. The io/fs future
    6. Summary
  11. Chapter 5: Using Common Data Formats
    1. Technical requirements
    2. CSV files
      1. Basic value separation using the strings package
      2. Using the encoding/csv package
      3. Using excelize when dealing with Excel
    3. Popular encoding formats
      1. The Go field tags
      2. JSON
      3. YAML encoding
    4. Summary
  12. Chapter 6: Interacting with Remote Data Sources
    1. Technical requirements
    2. Accessing SQL databases
      1. Connecting to a Postgres database
      2. Querying a Postgres database
      3. Null values
      4. Writing data to Postgres
      5. Transactions
      6. Postgres-specific types
      7. Other options
      8. Storage abstractions
      9. Case study – data migration of an orchestration system – Google
    3. Developing REST services and clients
      1. REST for RPCs
    4. Developing gRPC services and clients
      1. Protocol buffers
      2. Stating the prerequisites
      3. Generating your packages
      4. Writing a gRPC client
      5. Writing a gRPC server
      6. Creating a server binary
      7. Creating a client binary
      8. Company-standard RPC clients and servers
    5. Summary
  13. Chapter 7: Writing Command-Line Tooling
    1. Technical requirements
    2. Implementing application I/O
      1. The flag package
      2. Custom flags
      3. Basic flag error handling
      4. Shorthand flags
      5. Accessing non-flag arguments
      6. Retrieving input from STDIN
    3. Using Cobra for advanced CLI applications
      1. Code organization
      2. The optional Cobra generator
      3. The command package
    4. Handling OS signals
      1. Capturing an OS signal
      2. Using Context to cancel
    5. Summary
  14. Chapter 8: Automating Command-Line Tasks
    1. Technical requirements
    2. Using os/exec to automate local changes
      1. Determining the availability of essential tools
    3. Using SSH in Go to automate remote changes
      1. Connecting to another system
    4. Designing safe, concurrent change automations
      1. Components of a change
      2. Writing a concurrent job
      3. Case study – Network rollouts
    5. Writing a system agent
      1. Designing a system agent
      2. Implementing Install
      3. Implementing SystemPerf
    6. Summary
  15. Section 2: Instrumenting, Observing, and Responding
  16. Chapter 9: Observability with OpenTelemetry
    1. Technical requirements
    2. An introduction to OpenTelemetry
      1. Reference architecture for OpenTelemetry
      2. OpenTelemetry components
    3. Logging with context
      1. Our first log statement
      2. Structured and leveled logs with Zap
      3. Ingesting, transforming, and exporting logs using OpenTelemetry
    4. Instrumenting for distributed tracing
      1. The life cycle of a distributed trace
      2. Correlating traces and logs
      3. Adding log entries to spans
    5. Instrumenting for metrics
      1. The life cycle of a metric
      2. Client/server metrics with OpenTelemetry
    6. Alerting on metrics abnormalities
      1. Adding and configuring Alertmanager
    7. Summary
  17. Chapter 10: Automating Workflows with GitHub Actions
    1. Technical requirements
    2. Understanding the basics of GitHub Actions
      1. Exploring the components of a GitHub Action
      2. How to build and trigger your first GitHub Action
    3. Building a continuous integration workflow
      1. Introducing the tweeter command-line tool
      2. Goals of the tweeter continuous integration workflow
      3. Continuous integration workflow for tweeter
    4. Building a release workflow
      1. GitHub releases
      2. Release automation for tweeter
    5. Creating a custom GitHub Action using Go
      1. Basics of custom actions
      2. Goals for the tweeter custom GitHub Action
      3. Creating the tweeter action
    6. Publishing a custom Go GitHub Action
      1. The basics of publishing actions
      2. Goals for publishing the tweeter custom action
      3. Managing action semantic versioning
      4. Publishing the tweeter action to the GitHub Marketplace
    7. Summary
  18. Chapter 11: Using ChatOps to Increase Efficiency
    1. Technical requirements
    2. Environment architecture
    3. Using an Ops service
    4. Building a basic chatbot
    5. Creating event handlers
      1. Case Study – Regexes versus Lexer and Parser
    6. Creating our Slack application
      1. Running the applications
    7. Summary
  19. Section 3: Cloud ready Go
  20. Chapter 12: Creating Immutable Infrastructure Using Packer
    1. Technical requirements
    2. Building an Amazon Machine Image
      1. Setting up an AWS source
      2. Defining a build block and adding some provisioners
      3. Executing a Packer build
    3. Validating images with Goss
      1. Creating a spec file
      2. Adding a Packer provisioner
    4. Customizing Packer with plugins
      1. Writing your own plugin
      2. Releasing a plugin
      3. Using our plugin in a build
      4. Debugging a Packer plugin
    5. Summary
  21. Chapter 13: Infrastructure as Code with Terraform
    1. Technical requirements
    2. An introduction to IaC
    3. Understanding the basics of Terraform
      1. Initializing and applying infrastructure specs using Terraform
    4. Understanding the basics of Terraform providers
      1. Defining and provisioning cloud resources
    5. Building a pet store Terraform provider
      1. Resources for building custom providers
      2. The pet store provider
      3. Publishing custom providers
    6. Summary
  22. Chapter 14: Deploying and Building Applications in Kubernetes
    1. Technical requirements
    2. Interacting with the Kubernetes API
      1. Creating a KinD cluster
      2. Using kubectl to interact with the API
    3. Deploying a load-balanced HTTP application using Go
      1. It all starts with main
      2. Creating a ClientSet
      3. Creating a namespace
      4. Deploying the application into the namespace
      5. Creating the NGINX deployment
      6. Waiting for ready replicas to match desired replicas
      7. Creating a Service to load-balance
      8. Creating an ingress to expose our application on a local host port
      9. Streaming pod logs for the NGINX application
    4. Extending Kubernetes with custom resources and operators
      1. Custom Resource Definitions
      2. Controllers
      3. Standing on the shoulders of giants
    5. Building a pet store operator
      1. Initializing the new operator
    6. Summary
  23. Chapter 15: Programming the Cloud
    1. Technical requirements
    2. What is the cloud?
    3. Learning the basics of the Azure APIs
      1. A background on cloud APIs and SDKs
      2. Microsoft Azure identity, RBAC, and resource hierarchy
      3. Creating an Azure account and accessing the API
    4. Building infrastructure using Azure Resource Manager
      1. Azure SDK for Go
      2. Setting up your local environment
      3. Building an Azure virtual machine
    5. Using provisioned Azure infrastructure
      1. Building an Azure Storage account
    6. Summary
  24. Chapter 16: Designing for Chaos
    1. Technical requirements
    2. Using overload prevention mechanisms
      1. Case study – AWS client requests overwhelm the network
      2. Using circuit breakers
      3. Using backoff implementations
      4. Combining circuit breakers with backoff
    3. Using rate limiters to prevent runaway workflows
      1. Case study – Google satellite disk erase
      2. Channel-based rate limiter
      3. Token-bucket rate limiter
    4. Building workflows that are repeatable and never lost
      1. Building idempotent workflows
      2. Using three-way handshakes to prevent workflow loss
    5. Using policies to restrict tools
      1. Defining a gRPC workflow service
      2. Creating a policy engine
      3. Writing a policy
      4. Cautions on policy engines
    6. Building systems with an emergency stop
      1. Understanding emergency stops
      2. Building an emergency-stop package
      3. Using the emergency-stop package
      4. Case study – Google's network backbone emergency stop
    7. Summary
    8. Why subscribe?
  25. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Share your thoughts

Product information

  • Title: Go for DevOps
  • Author(s): John Doak, David Justice
  • Release date: July 2022
  • Publisher(s): Packt Publishing
  • ISBN: 9781801818896