O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Mastering SaltStack

Book Description

Take charge of SaltStack to automate and configure enterprise-grade environments

About This Book

  • Automate tasks effectively, so that your infrastructure can run itself
  • Take advantage of cloud-based services that can expand the capabilities of your own data centers
  • Tackle real-world problems that appear in everyday situations

In Detail

SaltStack is known as a popular configuration management system, but that barely scratches the surface. It is, in fact, a powerful automation suite, which is designed not only to help you manage your servers, but to help them manage themselves. SaltStack is used worldwide by organizations ranging from just a few servers, to tens of thousands of nodes across data centers in multiple continents. This award-winning software is fast becoming the standard for systems management in the cloud world.

This book will take you through the advanced features of SaltStack, bringing forward capabilities that will help you excel in the management of your servers.

You will be taken through the the mind of the modern systems engineer, and discover how they use Salt to manage their infrastructures, and why those design decisions are so important. The inner workings of Salt will be explored, so that as you advance your knowledge of Salt, you will be able to swim with the current, rather than against it.

Various subsystems of Salt are explained in detail, including Salt SSH, Salt Cloud, and external pillars, filesystems, and job caches.

You will be taken through an in-depth discussion of how to effectively scale Salt to manage thousands of machines, and how to troubleshoot issues when things don't go exactly the way you expect them to.

You will also be taken through an overview of RAET, Salt's new transport protocol, and given an insight into how this technology improves Salt, and the possibilities that it brings with it.

What You Will Learn

  • Learn how the pros are managing their infrastructures, and what techniques they use to keep everything running smoothly with Salt
  • Understand what makes Salt tick, and how that affects the way you use it
  • Take a look at familiar features in a new light, so that you have a better handle on how to approach tasks
  • Use Salt SSH to manage servers that don't fit Salt's traditional use cases
  • Besides automating your configuration, you will be able to automate your servers, and make them behave more intelligently
  • Make better use of cloud-based services, including compute clouds such as EC2, Azure and Openstack
  • Learn about the new RAET protocol, and how it changes the way automation works

Who This Book Is For

This book is ideal for professionals who have been managing groups of servers, and want to learn how to add functionality and expand their tool set. This book will also explain some of the more advanced features of Salt, and explore how to use them to bring additional power to the fundamentals that the professionals have already been using.

Style and approach

This book speaks informally, sometimes almost conversationally, to the user. Topics are covered in detail, using examples that should be comfortable to most users.

Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

Table of Contents

  1. Mastering SaltStack
    1. Table of Contents
    2. Mastering SaltStack
    3. Credits
    4. Foreword
    5. About the Author
    6. About the Reviewers
    7. www.PacktPub.com
      1. Support files, eBooks, discount offers, and more
        1. Why subscribe?
        2. Free access for Packt account holders
    8. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Errata
        2. Piracy
        3. Questions
    9. 1. Reviewing a Few Essentials
      1. Executing commands remotely
        1. Master and Minions
        2. Targeting Minions
          1. Glob
          2. Perl Compatible Regular Expression (PCRE)
          3. List
          4. Subnet
          5. Grain
          6. Grain PCRE
          7. Pillar
          8. Compound
          9. Nodegroup
        3. Using module functions
          1. test.ping
          2. test.echo
          3. test.sleep
          4. test.version
          5. pkg.install
          6. pkg.remove
          7. file.replace
          8. Other file functions
          9. Various user and group functions
          10. sys.doc
      2. SLS file trees
        1. SLS files
        2. Tying things together with top files
        3. Organizing the SLS directories
      3. Using States for configuration management
        1. Using include blocks
        2. Ordering with requisites
          1. require
          2. watch
          3. onchanges
          4. onfail
          5. use
        3. prereq
        4. Inverting requisites
        5. Extending SLS files
      4. The basics of Grains, Pillars, and templates
        1. Using Grains for Minion-specific data
        2. Centralizing variables with Pillars
        3. Managing files dynamically with templates
        4. A quick Jinja primer
      5. Summary
    10. 2. Diving into Salt Internals
      1. Understanding the Salt configuration
        1. Following the configuration tree
          1. Looking inside /etc/salt/
          2. Managing Salt keys
          3. Exploring the SLS directories
        2. Examining the Salt cache
          1. The Master job cache
          2. The Master-side Minion cache
          3. The external file server cache
          4. The Minion-side proc/ directory
          5. External modules
      2. The Renderer
        1. Rendering SLS files
        2. Render pipes
        3. Serving templated files
      3. Understanding the Loader
        1. Dynamic modules
        2. Execution modules
        3. Cloud modules
      4. Plunging into the State compiler
        1. Imperative versus declarative
        2. Requisites
        3. High and Low States
        4. High States
        5. Low States
        6. Enforcing statefulness
          1. name
          2. result
          3. changes
          4. comment
      5. Summary
    11. 3. Exploring Salt SSH
      1. Grappling with SSH
        1. Remote shells
      2. Using rosters
        1. The flat roster
          1. host
          2. port
          3. user
          4. passwd
          5. sudo
          6. priv
          7. timeout
          8. thin_dir
        2. Other built-in rosters
          1. scan
          2. cache
          3. cloud
          4. ansible
        3. Building dynamic rosters
        4. Using Salt SSH
          1. Using a Saltfile
        5. Salt versus Salt SSH
          1. Architecture
          2. Performance
      3. Understanding the salt-thin agent
        1. Building the thin package
          1. Including extra modules
        2. Deploying the thin package
        3. Executing the thin package
          1. The Salt SSH shim
          2. Preparing for Salt States
          3. Running Salt
          4. Salt's running data
      4. Using the raw SSH mode
        1. Caching SSH connections
      5. Summary
    12. 4. Managing Tasks Asynchronously
      1. Looking at the event system
        1. Reviewing the basics
        2. The structure of event data
        3. Watching event data
          1. Installing the event listener
          2. Using the event listener
          3. Firing custom data
        4. Namespacing events
          1. Namespacing guidelines
        5. Some common events
          1. salt/auth
          2. salt/key
          3. salt/minion/<minion_id>/start
          4. salt/job/<job_id>/new
          5. salt/job/<job_id>/ret/<minion_id>
          6. salt/presence/present
          7. salt/presence/change
        6. Common cloud events
          1. salt/cloud/<vm_name>/creating
          2. salt/cloud/<vm_name>/requesting
          3. salt/cloud/<vm_name>/querying
          4. salt/cloud/<vm_name>/waiting_for_ssh
          5. salt/cloud/<vm_name>/deploying
          6. salt/cloud/<vm_name>/created
          7. salt/cloud/<vm_name>/destroying
          8. salt/cloud/<vm_name>/destroyed
        7. Salt API events
          1. salt/netapi/<url_path>
      2. Building Reactors
        1. Configuring Reactors
        2. Writing Reactors
          1. Calling execution modules
          2. Calling runner modules
          3. Calling wheel modules
        3. Writing more complex Reactors
          1. Sending out alerts
          2. Using webhooks
          3. Reactors calling Reactors
      3. Using the queue system
        1. Learning how queues work
          1. Adding to the queue
          2. Listing queues
          3. Listing items in a queue
          4. Processing queue items
          5. Deleting items from a queue
        2. Using queues with the Reactor
          1. Spreading out State runs
          2. Dividing tasks among Minions
      4. Summary
    13. 5. Taking Salt Cloud to the Next Level
      1. Examining the Salt Cloud configuration
        1. Global configurations
        2. The provider and profile configuration
          1. Providers
          2. Profiles
        3. Extending configuration blocks
      2. Building custom deploy scripts
        1. Understanding the Salt Bootstrap script
          1. Installing from prebuilt packages
          2. Installing from Git
        2. Looking back at legacy deploy scripts
        3. Writing your own deploy scripts
        4. Passing arguments to scripts
        5. Using file maps
      3. Taking a look at cloud maps
      4. Working with autoscale Reactors
        1. The cloud cache
        2. Using cloud cache events
          1. Setting up a schedule
          2. Catching cloud cache events
      5. Summary
    14. 6. Using Salt with REST
      1. Looking at Salt's HTTP library
        1. Why a Salt-specific library?
        2. Using the http.query function
          1. GET versus POST
          2. Decoding return data
        3. Using the http.query state
          1. Using http.query with Reactors
      2. Understanding Salt API
        1. What is Salt API?
        2. Setting up Salt API
          1. CherryPy
          2. Tornado
          3. WSGI
        3. Creating SSL certificates
        4. Configuring authentication
          1. The external authentication module
        5. Taking your first steps with Salt API
          1. Issuing one-off commands
      3. Working with webhooks
        1. Security considerations
        2. More complex authentication
      4. Summary
    15. 7. Understanding the RAET Protocol
      1. Comparing RAET and ZeroMQ
        1. Starting with HTTP
        2. SSH, the old favorite
        3. Using ZeroMQ
          1. ZeroMQ and security
        4. The need for RAET
      2. Flow-based programming
        1. The pieces of the puzzle
          1. Black boxes
          2. Shared storage
          3. Concurrent scheduling
      3. Driving with RAET
        1. Configuring RAET
        2. The RAET architecture
          1. The basics
          2. The RAET scheduler
          3. Estates and yards
      4. Summary
    16. 8. Strategies for Scaling
      1. All about syndication
        1. Different folks, different strokes
        2. No need for micro-managing
        3. Configuring syndication
      2. High availability with multiple Masters
        1. Built-in high availability configuration
        2. Old-school high availability
          1. The round robin DNS
          2. IP-based load balancing
        3. Synchronizing files
          1. Base configuration files
          2. Synchronizing the nonexternal files
          3. Using rsync
          4. Using the event Reactor
      3. Incorporating external data sources
        1. The external job cache
          1. Using Returners on the Minions
          2. Using the Master job cache
        2. External filesystems
          1. GitFS
            1. base
            2. root
            3. mountpoint
            4. user and password
            5. insecure_auth
            6. pubkey, privkey, and passphrase
          2. Other source control backends
          3. SVNFS
            1. root and mountpoint
            2. trunk
            3. branches
            4. tags
          4. HGFS
          5. S3FS
            1. One environment per bucket
            2. Multiple environments per bucket
          6. AzureFS
        3. External Pillars
          1. cmd_yaml/cmd_json
          2. git
          3. redis
          4. mysql
      4. Using the Master API
        1. The Salt keys
        2. Configuration
        3. The file and Pillar roots
        4. Using the Wheel Reactor
      5. Testing load in the infrastructure
        1. Using the Minion Swarm
          1. Swarm internals
      6. Summary
    17. 9. Monitoring with Salt
      1. Monitoring basics
        1. Establishing a baseline
        2. Reading the system vitals in Salt
          1. status.loadavg
          2. status.cpustats
          3. status.meminfo
          4. status.vmstats
          5. disk.usage, status.diskusage
          6. status.w
          7. status.all_status, status.custom
      2. Monitoring with Returners
        1. Deciding on a Returner
      3. Using monitoring states
        1. Defining a monitoring state
        2. Monitoring with web calls
      4. Working with beacons
        1. Monitoring file changes
        2. Beacon intervals
      5. Setting up alerts
        1. Alerting in State files
        2. Alerting from beacons
          1. Watching file changes
          2. Monitoring bad logins
      6. Summary
    18. 10. Exploring Best Practices
      1. Future-proofing your infrastructure
      2. Setting up your directories
        1. Standard directory locations
        2. <module>.sls versus init.sls
        3. Shallow versus deep
        4. Subdividing further
      3. The SLS efficiency
        1. Includes and extends
          1. Using includes
          2. Using extends
        2. Using templates to simplify SLS files
          1. Working with loops
          2. Decisions, decisions
        3. Using the built-in States
      4. Naming conventions
        1. Generic names
        2. Explicit names
      5. Templates and variables
        1. Nested variables
        2. Referring to variables in templates
      6. Summary
    19. 11. Troubleshooting Problems
      1. What the…?
        1. Addressing the problem source
        2. Where is the trouble?
          1. The Master to Minion communication
          2. The network and CPU congestion
          3. Checking the Minion load
          4. Querying the Salt job data
        3. Using debug and trace modes
          1. info
          2. warn
          3. error
          4. debug/trace
          5. Running services in debug mode
      2. Using salt-call locally
      3. Working with YAML
        1. YAML basics
          1. dict
          2. list
        2. YAML idiosyncrasies
          1. Spacing
          2. Numbers
          3. Booleans
          4. List items
        3. Troubleshooting YAML
      4. Asking the community for help
        1. The salt-users mailing list
          1. Asking questions
        2. The Salt issue tracker
          1. Researching before posting
          2. Formatting your issues
          3. Requesting features
        3. #salt on IRC
        4. Final community thoughts
      5. Summary
    20. Index