O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Puppet: Mastering Infrastructure Automation

Book Description

Start pulling the strings of your IT infrastructure effectively with Puppet – learn how to configure, customize and manage your systems more intelligently in an enterprise deployment.

About This Book

  • Step-by-step instructions to jump start your Puppet deployment and automate your IT configurations
  • Design, implement, and deploy Puppet in your infrastructure and tackle everyday IT infrastructure challenges
  • Gain expert understanding of Puppet's latest and most advanced features

Who This Book Is For

If you're an experienced IT professional and a new Puppet user, this course will provide you with all you need to know to go from installation to advanced automation.

What You Will Learn

  • Write and employ individual Puppet manifests
  • Understand how Puppet implements system abstraction
  • Use Puppet to manage network, cloud, and virtualization devices
  • Manage and test the Puppet code workflow
  • Tweak, hack, and adapt the Puppet extension points
  • Use exported resources to orchestrate change between machines
  • Debug a puppetserver using Java techniques

In Detail

Puppet is a configuration management tool that allows you to automate all your IT configurations, giving you control. It was written for and by system administrators to manage large numbers of systems efficiently and prevent configuration drifts. Puppet gives you control of what you do to each node, when you do it, and how you do it.This Learning Path will take you from the basics of Puppet to extending it and then mastering it. We will start off with a primer on Puppet, its installation and get a sneak peek under its hood. We will then move on to designing and deploying your Puppet architecture, learning best practices for writing reusable and maintainable code, and executing, testing, and deploying Puppet across your systems. Finally, we will deal with problems of scale and exceptions in your code, automate workflows, and support multiple developers working simultaneously.This course is based on the following books:

  1. Puppet 4 Essentials - Second Edition
  2. Extending Puppet - Second Edition
  3. Mastering Puppet - Second Edition

Style and approach

This learning path follows a practical approach, starting from the basic commands to advanced aspects. Through a series of examples you will learn the most important skills essential to manage your IT infrastructure effectively.

Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

Table of Contents

  1. Puppet: Mastering Infrastructure Automation
    1. Table of Contents
    2. Puppet: Mastering Infrastructure Automation
    3. Puppet: Mastering Infrastructure Automation
    4. Credits
    5. Preface
      1. What this learning path covers
      2. What you need for this learning path
      3. Who this learning path is for
        1. Reader feedback
      4. Customer support
        1. Downloading the example code
        2. Errata
        3. Piracy
        4. Questions
    6. 1. Module 1
      1. 1. Writing Your First Manifests
        1. Getting started
        2. Introducing resources and properties
        3. Interpreting the output of the puppet apply command
          1. Dry-testing your manifest
        4. Adding control structures in manifests
        5. Using variables
          1. Variable types
        6. Controlling the order of evaluation
          1. Declaring dependencies
          2. Error propagation
          3. Avoiding circular dependencies
        7. Implementing resource interaction
        8. Examining the most notable resource types
          1. The user and group types
          2. The exec resource type
          3. The cron resource type
          4. The mount resource type
        9. Summary
      2. 2. The Master and Its Agents
        1. The Puppet Master
          1. Puppet Master and Puppet Server
          2. Setting up the master machine
          3. Creating the master manifest
          4. Inspecting the configuration settings
        2. Setting up the Puppet Agent
          1. The agent's life cycle
          2. Renewing an agent's certificate
          3. Running the agent from cron
        3. Performance considerations
          1. Tuning puppetserver
        4. Using Phusion Passenger with Nginx
          1. Comparing Passenger with puppetserver
        5. Completing the stack with PuppetDB
        6. Troubleshooting SSL issues
        7. Summary
      3. 3. A Peek under the Hood – Facts, Types, and Providers
        1. Summarizing systems with Facter
          1. Accessing and using fact values
          2. Extending Facter with custom facts
            1. Simplifying things using external facts
          3. Goals of Facter
        2. Understanding the type system
          1. The resource type's life cycle on the agent side
        3. Substantiating the model with providers
          1. Resource types with generic providers
          2. Summarizing types and providers
        4. Putting it all together
        5. Summary
      4. 4. Modularizing Manifests with Classes and Defined Types
        1. Introducing classes and defined types
          1. Defining and declaring classes
          2. Creating and using defined types
          3. Understanding and leveraging the differences
        2. Structured design patterns
          1. Writing comprehensive classes
          2. Writing component classes
          3. Using defined types as resource wrappers
          4. Using defined types as resource multiplexers
          5. Using defined types as macros
          6. Exploiting array values using defined types
            1. Using iterator functions
        3. Including classes from defined types
        4. Establishing relationships among containers
          1. Passing events between classes and defined types
          2. Ordering containers
          3. Limitations
          4. The performance implications of container relationships
          5. Mitigating the limitations
            1. The anchor pattern
            2. The contain function
        5. Making classes more flexible through parameters
          1. The caveats of parameterized classes
          2. Preferring the include keyword
        6. Summary
      5. 5. Extending Your Puppet Infrastructure with Modules
        1. An overview of Puppet's modules
          1. Parts of a module
          2. Module structure
          3. Documentation in modules
        2. Maintaining environments
          1. Configuring environment locations
          2. Obtaining and installing modules
          3. Modules' best practices
          4. Putting everything in modules
          5. Avoiding generalization
          6. Testing your modules
            1. Safe testing with environments
        3. Building a specific module
          1. Naming your module
          2. Making your module available to Puppet
          3. Implementing basic module functionality
          4. Creating utilities for derived manifests
            1. Adding configuration items
            2. Allowing customization
            3. Removing unwanted configuration items
            4. Dealing with complexity
          5. Enhancing the agent through plugins
            1. Replacing a defined type with a native type
              1. Naming your type
              2. Creating the resource type interface
              3. Designing sensible parameter hooks
              4. Using resource names
              5. Adding a provider
              6. Declaring management commands
              7. Implementing the basic functionality
              8. Allowing the provider to prefetch existing resources
              9. Making the type robust during provisioning
          6. Enhancing Puppet's system knowledge through facts
          7. Refining the interface of your module through custom functions
          8. Making your module portable across platforms
        4. Finding helpful Forge modules
          1. Identifying modules' characteristics
        5. Summary
      6. 6. Leveraging the Full Toolset of the Language
        1. Templating dynamic configuration files
          1. Learning the template syntax
          2. Using templates in practice
          3. Avoiding performance bottlenecks from templates
        2. Creating virtual resources
          1. Realizing resources more flexibly using collectors
        3. Exporting resources to other agents
          1. Exporting and importing resources
          2. Configuring the master to store exported resources
          3. Exporting SSH host keys
          4. Managing hosts files locally
          5. Automating custom configuration items
          6. Simplifying the configuration of Nagios
          7. Maintaining your central firewall
          8. Removing obsolete exports
        4. Overriding resource parameters
          1. Saving redundancy using resource defaults
        5. Avoiding antipatterns
        6. Summary
      7. 7. New Features from Puppet 4
        1. Upgrading to Puppet 4
          1. Using Puppet 3.8 and environment directories
          2. The new Puppet 4 Master
          3. Updating the Puppet agent
          4. Testing Puppet DSL code
        2. Using the type system
        3. Learning lambdas and functions
        4. Creating Puppet 4 functions
        5. Leveraging the new template engine
        6. Handling multiline with HEREDOC
        7. Breaking old practices
          1. Converting node inheritance
          2. Dealing with bool algebra on Strings
          3. Using strict variable naming
          4. Learning the new reference syntax
          5. Cleaning hyphens in names
          6. No Ruby DSL anymore
          7. Relative class name resolution
          8. Dealing with different data types
        8. Summary
      8. 8. Separating Data from Code Using Hiera
        1. Understanding the need for separate data storage
          1. Consequences of defining data in the manifest
        2. Structuring configuration data in a hierarchy
          1. Configuring Hiera
          2. Storing Hiera data
          3. Choosing your backends
        3. Retrieving and using Hiera values in manifests
          1. Working with simple values
          2. Binding class parameter values automatically
          3. Handling hashes and arrays
        4. Converting resources to data
          1. Choosing between manifest and Hiera designs
          2. A practical example
        5. Debugging Hiera lookups
        6. Implementing the Roles and Profiles pattern
        7. Summary
    7. 2. Module 2
      1. 1. Puppet Essentials
        1. The Puppet ecosystem
          1. Why configuration management matters
        2. Puppet components
        3. Installing and configuring Puppet
        4. Puppet in action
          1. Resources
          2. The resource abstraction layer
          3. Nodes
          4. Classes and defines
          5. Class inheritance
          6. Resource defaults
          7. Resource references
        5. Variables, facts, and scopes
          1. System's facts
          2. External facts
          3. User variables in Puppet DSL
          4. Type-checking
          5. User variables in an ENC
          6. User variables in Hiera
          7. Puppet built-in variables
          8. Variables scope
        6. Meta parameters
        7. Managing order and dependencies
          1. Run stages
        8. Reserved names and allowed characters
        9. Conditionals
        10. Comparison operators
        11. Iteration and lambdas
          1. The in operator
          2. Expressions combinations
        12. Exported resources
          1. Virtual resources
        13. Modules
          1. Paths of a module and auto loading
          2. Templates
        14. Restoring files from a filebucket
        15. Summary
      2. 2. Managing Puppet Data with Hiera
        1. Installing and configuring Hiera
          1. The hiera.yaml configuration file
            1. Global settings
            2. Backend specific settings
            3. Examples
        2. Working with the command line on a YAML backend
        3. Using Hiera in Puppet
          1. Dealing with hashes in Puppet code
          2. Puppet 3 automatic parameter lookup
          3. Evolving usage patterns for class parameters
        4. Additional Hiera backends
          1. Hiera-file
          2. Encryption plugins
            1. Hiera-gpg
          3. Hiera-eyaml
          4. Hiera-http, hiera-mysql
        5. Using Hiera as an ENC
        6. Summary
      3. 3. Introducing PuppetDB
        1. Installation and configuration
          1. Installing PuppetDB
          2. PuppetDB configurations
            1. Init script configuration
            2. Configuration settings
            3. Logging configuration
          3. Configurations on the Puppet Master
        2. Dashboards
          1. PuppetDB performance dashboard
          2. Puppetboard—Querying PuppetDB from the Web
        3. PuppetDB API
          1. Querying PuppetDB (read)
          2. PuppetDB commands (write)
        4. Querying PuppetDB for fun and profit
          1. /facts endpoint
          2. /resources endpoint
          3. /nodes endpoint
          4. /catalogs endpoint
          5. /fact-names endpoint
          6. /metrics endpoint
          7. /reports endpoint
          8. /events endpoint
          9. /event-counts endpoint
          10. /aggregated-event-counts endpoint
          11. /server-time endpoint
          12. /version endpoint
        5. The puppetdbquery module
          1. Query format
            1. Querying from the command line
            2. Querying from Puppet manifests
            3. The PuppetDB Hiera backend
        6. How Puppet code may change in the future
        7. Summary
      4. 4. Designing Puppet Architectures
        1. Components of a Puppet architecture
          1. Definition of the classes to include in each node
          2. Definition of the parameters to use for each node
          3. Definition of the configuration files provided to the nodes
          4. Definition of custom resources and classes
        2. The Foreman
        3. Roles and profiles
        4. The data and the code
        5. Sample architectures
          1. The default approach
          2. Basic ENC, logic on site module, Hiera backend
          3. The Foreman and Hiera
          4. Hiera-based setup
          5. Foreman smart variables
          6. Facts-driven truths
          7. Nodeless site.pp
        6. Summary
      5. 5. Using and Writing Reusable Modules
        1. Modules layout evolution
          1. Class parameters—from zero to data bindings
          2. Params pattern
          3. Data in modules
          4. Files and class names
          5. The anchor pattern
        2. The parameters dilemma
          1. Naming standards
        3. Reusability patterns
          1. Managing files
            1. Managing configuration hash patterns
            2. Managing multiple configuration files
            3. Managing users and dependencies
            4. Managing installation options
            5. Managing extra resources
        4. Summary
      6. 6. Higher Abstraction Modules
        1. The OpenStack example
          1. Component (application) modules
          2. Raising abstraction – the official OpenStack module
          3. Raising abstraction – Puppet Labs OpenStack module
          4. Taking an alternate approach
        2. An approach to reusable stack modules
        3. Tiny Puppet
        4. Summary
      7. 7. Puppet Migration Patterns
        1. Examining potential scenarios and approaches
          1. New infrastructures
          2. Existing manually managed infrastructures
            1. Node migration
            2. Node update
          3. Existing automated infrastructures
        2. Patterns for extending Puppet coverage
          1. Raising the bar, step by step
          2. Knowing the territory
          3. Defining priorities
          4. Evaluating solutions
          5. Coding
          6. Applying changes
        3. Things change
          1. Puppet friendly infrastructure
          2. Puppet-friendly software
        4. Summary
      8. 8. Code Workflow Management
        1. Write Puppet code
          1. Geppetto
          2. Vim
        2. Git workflows
          1. Code management using Git
          2. Git hooks
          3. Environments and branches
            1. Branch based automatic environments
            2. Simplified developer workdir environments
        3. Code review
          1. Gerrit
          2. Online resources for peer review
        4. Testing Puppet code
          1. Using rspec-puppet
          2. Vagrant
          3. Beaker and beaker-rspec
        5. Deploying Puppet code
          1. Using librarian-puppet for deployments
          2. Deploying code with r10k
        6. Propagating Puppet changes
        7. Puppet continuous integration
          1. Travis
          2. Jenkins
        8. Summary
      9. 9. Scaling Puppet Infrastructures
        1. Scaling Puppet
          1. As simple as puppet apply
          2. Default Puppet Master
          3. Puppet Master with Passenger
            1. Installing and configuring Passenger
          4. Puppet Master based on Trapperkeeper
          5. Multi-Master scaling
            1. Managing certificates
            2. Managing SSL termination
            3. Managing code and data
          6. Load balancing alternatives
          7. Masterless Puppet
          8. Configurations and infrastructure optimizations
            1. Traffic compression
            2. Caching
            3. Distributing Puppet run times
            4. File system changes check
            5. Scaling stored configs
        2. Measuring performance
          1. Puppet Metrics
          2. Optimizing code
          3. Testing different Puppet versions
        3. Summary
      10. 10. Extending Puppet
        1. Anatomy of a Puppet run, under the hood
        2. Puppet extension alternatives
          1. ENC and Hiera Extendibility
          2. Modules pluginsync
          3. Puppet indirector and its termini
        3. Custom functions
        4. Custom facts
          1. Ruby facts distributed via pluginsync
          2. External facts in facts.d directory
        5. Custom types and providers
        6. Custom report handlers
        7. Custom faces
        8. Summary
      11. 11. Beyond the System
        1. Puppet on network equipment
          1. Proxy mode with Puppet device application
          2. Native Puppet on network equipment
            1. Cisco onePK
            2. Juniper and the netdev_stdlib
        2. Puppet for cloud and virtualization
          1. VMware
            1. VM provisioning on vCenter and vSphere
            2. vCenter configuration
            3. vSphere virtual machine management with resource types
          2. Amazon Web Services
            1. Cloud provisioning on AWS
          3. AWS provisioning and configuration with resource types
            1. Managing CloudFormation
          4. Cloud provisioning on Google Compute Engine
        3. Puppet on storage devices
        4. Puppet and Docker
        5. Summary
      12. 12. Future Puppet
        1. Changing the serialization format
        2. Direct Puppet
          1. File sync
        3. Other changes
          1. Certificate authority in Clojure
          2. Package management
          3. Changes in faces
        4. Beyond Puppet 4.x
        5. Summary
    8. 3. Module 3
      1. 1. Dealing with Load/Scale
        1. Divide and conquer
          1. Certificate signing
          2. Reporting
          3. Storeconfigs
          4. Catalog compilation
          5. puppetserver
          6. Building a Puppet master
            1. Certificates
            2. systemd
            3. Creating a load balancer
              1. Apache proxy
              2. TLS headers
                1. SELinux
              3. Configuring masters
            4. Keeping the code consistent
              1. rsync
              2. NFS
              3. Clustered filesystem
              4. Git
          7. One more split
          8. One last split or maybe a few more
        2. Conquer by dividing
          1. Creating an rpm
            1. Using Puppet resource to configure cron
          2. Creating the yum repository
        3. Summary
      2. 2. Organizing Your Nodes and Data
        1. Getting started
        2. Organizing the nodes with an ENC
          1. A simple example
            1. Hostname strategy
            2. Modified ENC using hostname strategy
          2. LDAP backend
            1. OpenLDAP configuration
        3. Hiera
          1. Configuring Hiera
          2. Using hiera_include
        4. Summary
      3. 3. Git and Environments
        1. Environments
          1. Environments and Hiera
            1. Multiple hierarchies
            2. Single hierarchy for all environments
          2. Directory environments
        2. Git
          1. Why Git?
          2. A simple Git workflow
          3. Git hooks
            1. Using post-receive to set up environments
            2. Puppet-sync
          4. Using Git hooks to play nice with other developers
          5. Not playing nice with others via Git hooks
        3. Git for everyone
        4. Summary
      4. 4. Public Modules
        1. Getting modules
        2. Using GitHub for public modules
          1. Updating the local repository
        3. Modules from the Forge
        4. Using Librarian
        5. Using r10k
        6. Using Puppet-supported modules
          1. concat
          2. inifile
          3. firewall
          4. Logical volume manager
          5. Standard library
        7. Summary
      5. 5. Custom Facts and Modules
        1. Module manifest files
          1. Module files and templates
          2. Naming a module
          3. Creating modules with a Puppet module
            1. Comments in modules
          4. Multiple definitions
        2. Custom facts
          1. Creating custom facts
          2. Creating a custom fact for use in Hiera
        3. CFacter
        4. Summary
      6. 6. Custom Types
        1. Parameterized classes
          1. Data types
        2. Defined types
        3. Types and providers
          1. Creating a new type
        4. Summary
      7. 7. Reporting and Orchestration
        1. Turning on reporting
        2. Store
        3. Logback
        4. Internet relay chat
        5. Foreman
          1. Installing Foreman
          2. Attaching Foreman to Puppet
          3. Using Foreman
        6. Puppet GUIs
        7. mcollective
          1. Installing ActiveMQ
          2. Configuring nodes to use ActiveMQ
          3. Connecting a client to ActiveMQ
          4. Using mcollective
        8. Ansible
        9. Summary
      8. 8. Exported Resources
        1. Configuring PuppetDB – using the Forge module
        2. Manually installing PuppetDB
          1. Installing Puppet and PuppetDB
          2. Installing and configuring PostgreSQL
          3. Configuring puppetdb to use PostgreSQL
          4. Configuring Puppet to use PuppetDB
        3. Exported resource concepts
          1. Declaring exported resources
          2. Collecting exported resources
          3. Simple example – a host entry
        4. Resource tags
        5. Exported SSH keys
          1. sshkey collection for laptops
        6. Putting it all together
        7. Summary
      9. 9. Roles and Profiles
        1. Design pattern
        2. Creating an example CDN role
          1. Creating a sub-CDN role
        3. Dealing with exceptions
        4. Summary
      10. 10. Troubleshooting
        1. Connectivity issues
        2. Catalog failures
          1. Full trace on a catalog compilation
          2. The classes.txt file
        3. Debugging
          1. Personal and bugfix branches
            1. Echo statements
          2. Scope
          3. Profiling and summarizing
        4. Summary
    9. A. Bibliography
    10. Index