Bootstrapping Microservices with Docker, Kubernetes, and Terraform

Bootstrapping Microservices with Docker, Kubernetes, and Terraform

by Ashley Davis
Released February 2021
Publisher(s): Manning Publications
ISBN: 9781617297212

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to live online training experiences, plus books, videos, and digital content from O’Reilly and nearly 200 trusted publishing partners.

Start your free trial

Book description

In Bootstrapping Microservices with Docker, Kubernetes, and Terraform, author Ashley Davis lays out a comprehensive approach to building microservices. You’ll start with a simple design and work layer-by-layer until you’ve created your own video streaming application. As you go, you’ll learn to configure cloud infrastructure with Terraform, package microservices using Docker, and deploy your finished project to a Kubernetes cluster.

Table of contents

  1. Bootstrapping Microservices with Docker, Kubernetes, and Terraform
  2. Copyright
  3. brief contents
  4. contents
  5. front matter
    1. preface
    2. acknowledgments
    3. about this book
    4. Who should read this book
    5. How this book is organized: A roadmap
    6. About the code
    7. liveBook discussion forum
    8. about the author
    9. about the cover illustration
  6. 1 Why microservices?
    1. 1.1 This book is practical
    2. 1.2 What will I learn?
    3. 1.3 What do I need to know?
    4. 1.4 Managing complexity
    5. 1.5 What is a microservice?
    6. 1.6 What is a microservices application?
    7. 1.7 What’s wrong with the monolith?
    8. 1.8 Why are microservices popular now?
    9. 1.9 Benefits of microservices
    10. 1.10 Drawbacks of microservices
    11. 1.11 Modern tooling for microservices
    12. 1.12 Designing a microservices application
    13. 1.13 An example application
    14. Summary
  7. 2 Creating your first microservice
    1. 2.1 New tools
    2. 2.2 Getting the code
    3. 2.3 Why Node.js?
    4. 2.4 Our philosophy of development
    5. 2.5 Establishing our single-service development environment
      1. 2.5.1 Installing Git
      2. 2.5.2 Cloning the code repo
      3. 2.5.3 Getting Visual Studio (VS) Code
      4. 2.5.4 Installing Node.js
    6. 2.6 Building an HTTP server for video streaming
      1. 2.6.1 Creating a Node.js project
      2. 2.6.2 Installing Express
      3. 2.6.3 Creating the Express boilerplate
      4. 2.6.4 Running our simple web server
      5. 2.6.5 Adding streaming video
      6. 2.6.6 Configuring our microservice
      7. 2.6.7 Setting up for production
      8. 2.6.8 Live reloading for fast iteration
      9. 2.6.9 Running the finished code from this chapter
    7. 2.7 Node.js review
    8. 2.8 Continue your learning
    9. Summary
  8. 3 Publishing your first microservice
    1. 3.1 New tools
    2. 3.2 Getting the code
    3. 3.3 What is a container?
    4. 3.4 What is an image?
    5. 3.5 Why Docker?
    6. 3.6 What are we doing with Docker?
    7. 3.7 Extending our development environment with Docker
      1. 3.7.1 Installing Docker
      2. 3.7.2 Checking your Docker installation
    8. 3.8 Packaging our microservice
      1. 3.8.1 Creating a Dockerfile
      2. 3.8.2 Packaging and checking our Docker image
      3. 3.8.3 Booting our microservice in a container
    9. 3.9 Publishing our microservice
      1. 3.9.1 Creating a private container registry
      2. 3.9.2 Pushing our microservice to the registry
      3. 3.9.3 Booting our microservice from the registry
    10. 3.10 Docker review
    11. 3.11 Continue your learning
    12. Summary
  9. 4 Data management for microservices
    1. 4.1 New tools
    2. 4.2 Getting the code
    3. 4.3 Developing microservices with Docker Compose
      1. 4.3.1 Why Docker Compose?
      2. 4.3.2 Installing Docker Compose
      3. 4.3.3 Creating our Docker Compose file
      4. 4.3.4 Booting our microservices application
      5. 4.3.5 Working with the application
      6. 4.3.6 Shutting down the application
      7. 4.3.7 Can we use Docker Compose for production?
    4. 4.4 Adding file storage to our application
      1. 4.4.1 Using Azure Storage
      2. 4.4.2 Updating the video-streaming microservice
      3. 4.4.3 Adding our new microservice to the Docker Compose file
      4. 4.4.4 Testing the updated application
      5. 4.4.5 Cloud storage vs. cluster storage
      6. 4.4.6 What did we achieve?
    5. 4.5 Adding a database to our application
      1. 4.5.1 Why MongoDB?
      2. 4.5.2 Adding a database server in development
      3. 4.5.3 Adding a database server in production
      4. 4.5.4 Database-per-microservice or database-per-application?
      5. 4.5.5 What did we achieve?
    6. 4.6 Docker Compose review
    7. 4.7 Continue your learning
    8. Summary
  10. 5 Communication between microservices
    1. 5.1 New and familiar tools
    2. 5.2 Getting the code
    3. 5.3 Getting our microservices talking
    4. 5.4 Introducing the history microservice
    5. 5.5 Live reload for fast iterations
      1. 5.5.1 Creating a stub for the history microservice
      2. 5.5.2 Augmenting the microservice for live reload
      3. 5.5.3 Splitting our Dockerfile for development and production
      4. 5.5.4 Updating the Docker Compose file for live reload
      5. 5.5.5 Trying out live reload
      6. 5.5.6 Testing production mode in development
      7. 5.5.7 What have we achieved?
    6. 5.6 Methods of communication for microservices
      1. 5.6.1 Direct messaging
      2. 5.6.2 Indirect messaging
    7. 5.7 Direct messaging with HTTP
      1. 5.7.1 Why HTTP?
      2. 5.7.2 Directly targeting messages at particular microservices
      3. 5.7.3 Sending a message with HTTP POST
      4. 5.7.4 Receiving a message with HTTP POST
      5. 5.7.5 Testing the updated application
      6. 5.7.6 Sequencing direct messages
      7. 5.7.7 What have we achieved?
    8. 5.8 Indirect messaging with RabbitMQ
      1. 5.8.1 Why RabbitMQ?
      2. 5.8.2 Indirectly targeting messages to microservices
      3. 5.8.3 Creating a RabbitMQ server
      4. 5.8.4 Investigating the RabbitMQ dashboard
      5. 5.8.5 Connecting our microservice to the message queue
      6. 5.8.6 Single-recipient indirect messaging
      7. 5.8.7 Multiple-recipient messages
      8. 5.8.8 Sequencing indirect messages
      9. 5.8.9 What have we achieved?
    9. 5.9 Microservices communication review
    10. 5.10 Continue your learning
    11. Summary
  11. 6 Creating your production environment
    1. 6.1 New tools
    2. 6.2 Getting the code
    3. 6.3 Getting to production
    4. 6.4 Infrastructure as code
    5. 6.5 Hosting microservices on Kubernetes
      1. 6.5.1 Why Kubernetes?
      2. 6.5.2 How does Kubernetes work?
    6. 6.6 Working with the Azure CLI
      1. 6.6.1 Installing the Azure CLI
      2. 6.6.2 Authenticating with Azure
      3. 6.6.3 Which version of Kubernetes?
      4. 6.6.4 What have we achieved?
    7. 6.7 Creating infrastructure with Terraform
      1. 6.7.1 Why Terraform?
      2. 6.7.2 Installing Terraform
      3. 6.7.3 Terraform project setup
    8. 6.8 Creating an Azure resource group for your application
      1. 6.8.1 Evolutionary architecture with Terraform
      2. 6.8.2 Scripting infrastructure creation
      3. 6.8.3 Initializing Terraform
      4. 6.8.4 By-products of Terraform initialization
      5. 6.8.5 Fixing provider version numbers
      6. 6.8.6 Building your infrastructure
      7. 6.8.7 Understanding Terraform state
      8. 6.8.8 Destroying and recreating your infrastructure
      9. 6.8.9 What have we achieved?
    9. 6.9 Creating your container registry
      1. 6.9.1 Continuing the evolution of our infrastructure
      2. 6.9.2 Creating the container registry
      3. 6.9.3 Terraform outputs
      4. 6.9.4 What have we achieved?
    10. 6.10 Refactoring to share configuration data
      1. 6.10.1 Continuing the evolution of our infrastructure
      2. 6.10.2 Introducing Terraform variables
    11. 6.11 Creating our Kubernetes cluster
      1. 6.11.1 Scripting creation of your cluster
      2. 6.11.2 Cluster authentication with Azure
      3. 6.11.3 Building your cluster
      4. 6.11.4 What have we achieved?
    12. 6.12 Interacting with Kubernetes
      1. 6.12.1 Kubernetes authentication
      2. 6.12.2 The Kubernetes CLI
      3. 6.12.3 The Kubernetes dashboard
      4. 6.12.4 What have we achieved?
    13. 6.13 Terraform review
    14. 6.14 Continue your learning
    15. Summary
  12. 7 Getting to continuous delivery
    1. 7.1 New and familiar tools
    2. 7.2 Getting the code
    3. 7.3 Continuing to evolve our infrastructure
    4. 7.4 Continuous delivery (CD)
    5. 7.5 Deploying containers with Terraform
      1. 7.5.1 Configuring the Kubernetes provider
      2. 7.5.2 Deploying our database
      3. 7.5.3 Preparing for continuous delivery
      4. 7.5.4 Testing the new database server
      5. 7.5.5 Deploying and testing RabbitMQ
      6. 7.5.6 Tightening our security
      7. 7.5.7 What have we achieved?
    6. 7.6 Deploying our first microservice with Terraform
      1. 7.6.1 Using local variables to compute configuration
      2. 7.6.2 Building and publishing the Docker image
      3. 7.6.3 Authentication with the container registry
      4. 7.6.4 Deploying the video-streaming microservice
      5. 7.6.5 Testing your microservice
      6. 7.6.6 What have we achieved?
    7. 7.7 Continuous delivery with Bitbucket Pipelines
      1. 7.7.1 Why Bitbucket Pipelines?
      2. 7.7.2 Importing the example code to Bitbucket
      3. 7.7.3 Creating a deployment shell script
      4. 7.7.4 Managing Terraform state
      5. 7.7.5 The Bitbucket Pipelines script
      6. 7.7.6 Configuring environment variables
      7. 7.7.7 Testing your deployment pipeline
      8. 7.7.8 Debugging your deployment pipeline
      9. 7.7.9 What have we achieved?
    8. 7.8 Continue your learning
    9. Summary
  13. 8 Automated testing for microservices
    1. 8.1 New tools
    2. 8.2 Getting the code
    3. 8.3 Testing for microservices
    4. 8.4 Automated testing
    5. 8.5 Testing with Jest
      1. 8.5.1 Why Jest?
      2. 8.5.2 Setting up Jest
      3. 8.5.3 The math library to test
      4. 8.5.4 Your first Jest test
      5. 8.5.5 Running your first test
      6. 8.5.6 Live reload with Jest
      7. 8.5.7 Interpreting test failures
      8. 8.5.8 Invoking Jest with npm
      9. 8.5.9 Populating your test suite
      10. 8.5.10 Mocking with Jest
      11. 8.5.11 What have we achieved?
    6. 8.6 Unit testing
      1. 8.6.1 The metadata microservice
      2. 8.6.2 Creating unit tests with Jest
      3. 8.6.3 Running the tests
      4. 8.6.4 What have we achieved?
    7. 8.7 Integration testing
      1. 8.7.1 The code to test
      2. 8.7.2 Running a MongoDB database
      3. 8.7.3 Loading database fixtures
      4. 8.7.4 Creating an integration test with Jest
      5. 8.7.5 Running the test
      6. 8.7.6 What have we achieved?
    8. 8.8 End-to-end testing
      1. 8.8.1 Why Cypress?
      2. 8.8.2 Installing Cypress
      3. 8.8.3 Starting the Cypress UI
      4. 8.8.4 Setting up database fixtures
      5. 8.8.5 Booting your application
      6. 8.8.6 Creating an end-to-end test with Cypress
      7. 8.8.7 Invoking Cypress with npm
      8. 8.8.8 What have we achieved?
    9. 8.9 Automated testing in the CD pipeline
    10. 8.10 Review of testing
    11. 8.11 Continue your learning
    12. Summary
  14. 9 Exploring FlixTube
    1. 9.1 No new tools!
    2. 9.2 Getting the code
    3. 9.3 Revisiting essential skills
    4. 9.4 Overview of FlixTube
      1. 9.4.1 FlixTube microservices
      2. 9.4.2 Microservice project structure
      3. 9.4.3 FlixTube project structure
    5. 9.5 Running FlixTube in development
      1. 9.5.1 Booting a microservice
      2. 9.5.2 Booting the application
    6. 9.6 Testing FlixTube in development
      1. 9.6.1 Testing a microservice with Jest
      2. 9.6.2 Testing the application with Cypress
    7. 9.7 FlixTube deep dive
      1. 9.7.1 Database fixtures
      2. 9.7.2 Mocking storage
      3. 9.7.3 The gateway
      4. 9.7.4 The user interface (UI)
      5. 9.7.5 Video streaming
      6. 9.7.6 Video upload
    8. 9.8 Manually deploying FlixTube to production with Terraform
      1. 9.8.1 The Terraform scripts structure
      2. 9.8.2 Prerequisites
      3. 9.8.3 Azure authentication
      4. 9.8.4 Configuring storage
      5. 9.8.5 Deploying the application
      6. 9.8.6 Checking that it works
      7. 9.8.7 Teardown
      8. 9.8.8 Terraform modules
    9. 9.9 Continuous delivery to production
      1. 9.9.1 Prerequisites
      2. 9.9.2 Setting up your code repository
      3. 9.9.3 Preparing the backend
      4. 9.9.4 The deployment shell script
      5. 9.9.5 FlixTube’s CD configuration
      6. 9.9.6 Testing the continuous delivery (CD) pipeline
      7. 9.9.7 Adding automated testing
    10. 9.10 Review
    11. 9.11 FlixTube in the future
    12. 9.12 Continue your learning
    13. Summary
  15. 10 Healthy microservices
    1. 10.1 Maintaining healthy microservices
    2. 10.2 Monitoring your microservices
      1. 10.2.1 Logging in development
      2. 10.2.2 Error handling
      3. 10.2.3 Logging with Docker Compose
      4. 10.2.4 Basic logging with Kubernetes
      5. 10.2.5 Roll your own log aggregation for Kubernetes
      6. 10.2.6 Enterprise logging, monitoring and alerts
      7. 10.2.7 Automatic restarts with Kubernetes health checks
      8. 10.2.8 Tracing across microservices
    3. 10.3 Debugging microservices
      1. 10.3.1 The debugging process
      2. 10.3.2 Debugging production microservices
    4. 10.4 Reliability and recovery
      1. 10.4.1 Practice defensive programming
      2. 10.4.2 Practice defensive testing
      3. 10.4.3 Protect your data
      4. 10.4.4 Replication and redundancy
      5. 10.4.5 Fault isolation and graceful degradation
      6. 10.4.6 Simple techniques for fault tolerance
      7. 10.4.7 Advanced techniques for fault tolerance
    5. 10.5 Continue your learning
    6. Summary
  16. 11 Pathways to scalability
    1. 11.1 Our future is scalable
    2. 11.2 Scaling the development process
      1. 11.2.1 Multiple teams
      2. 11.2.2 Independent microservices
      3. 11.2.3 Splitting the code repository
      4. 11.2.4 Splitting the continuous delivery (CD) pipeline
      5. 11.2.5 The meta-repo
      6. 11.2.6 Creating multiple environments
      7. 11.2.7 Production workflow
    3. 11.3 Scaling performance
      1. 11.3.1 Vertically scaling the cluster
      2. 11.3.2 Horizontally scaling the cluster
      3. 11.3.3 Horizontally scaling an individual microservice
      4. 11.3.4 Elastic scaling for the cluster
      5. 11.3.5 Elastic scaling for an individual microservice
      6. 11.3.6 Scaling the database
      7. 11.3.7 Managing changes to infrastructure
    4. 11.4 Security
      1. 11.4.1 Trust models
      2. 11.4.2 Sensitive configuration
    5. 11.5 Refactoring to microservices
    6. 11.6 Microservices on a budget
    7. 11.7 From simple beginnings . . .
    8. 11.8 Continue your learning
    9. Summary
  17. appendix A. Creating a development environment with Vagrant
    1. A.1 Installing VirtualBox
    2. A.2 Installing Vagrant
    3. A.3 Creating your virtual machine (VM)
    4. A.4 Connecting to your VM
    5. A.5 Installing software in the VM
    6. A.6 Using the example setup
    7. A.7 Turning your VM off
  18. appendix B. Bootstrapping Microservices cheat sheet
    1. B.1 Node.js commands
    2. B.2 Docker commands
    3. B.4 Docker Compose commands
    4. Terraform commands
    5. B.5 Testing commands
  19. index

Product information

  • Title: Bootstrapping Microservices with Docker, Kubernetes, and Terraform
  • Author(s): Ashley Davis
  • Release date: February 2021
  • Publisher(s): Manning Publications
  • ISBN: 9781617297212