book

Chef: Powerful Infrastructure Automation

Name: Chef: Powerful Infrastructure Automation
ISBN: 9781788392976

by John Ewart, Matthias Marschall, Earl Waud

May 2017

Intermediate to advanced

726 pages

15h 38m

English

Packt Publishing

Read now

Unlock full access

Chef: Powerful Infrastructure Automation
Table of Contents
Chef: Powerful Infrastructure Automation
Chef: Powerful Infrastructure Automation
Credits
Preface
What this learning path covers
What you need for this learning path
Who this learning path is for
Reader feedback
Customer support
Downloading the example codeErrataPiracyQuestions

1. Module 1
1. Installing Chef
Terminology
Working with Chef
Installing chef-soloThe Ruby gemManaging gemsVerifying that chef-solo works
Installing a Chef server
Requirements and recent changesInstallation requirementsWhat you will be installingGetting the installerInstallation outlineInstalling on UbuntuDownloading the packageInstalling the packageInstalling on Red Hat Enterprise LinuxDownloading the package
Configuring a Chef server
Understanding how chef-server-ctl worksWhat's happening on my server?Verifying that the services are running
Validating that your service is working
Ensuring that your knife configuration works
Summary
2. Modeling Your Infrastructure
Getting to know Chef
Modeling your infrastructure
RolesDefining rolesA web application service roleAn image-processing roleAn image search roleA PostgreSQL service roleA Solr service roleAn OpenSSH service roleImplementing a roleDetermining which recipes you needInstalling a cookbookApplying recipes to rolesMapping your roles to nodesConverging a nodeEnvironments
Organizing your configuration data
Example attribute dataData bagsKnowing when to use data bags
Large-scale infrastructure
Summary
3. Integrating with the Cloud
Leveraging the cloudAmazon EC2Installing the EC2 knife pluginSetting up EC2 authenticationProvisioning an instanceBootstrapping the instanceTerminating the instanceRemoving the Chef nodeRackspace CloudProvisioning an instanceTerminating an instanceRemoving the Chef node
Summary
4. Working with Cookbooks
AttributesMultiple attribute filesSupporting multiple platformsLoading external attributesUsing attributes
Metadata
Recipes
Resources
Using resourcesOverriding a default behavior
Templates
Why use templates?A quick ERB primerExecuting RubyVariable replacementThe template resourceThe template variablesPassing variables to a templateAccessing computed configurationsSearching for templates
Definitions
Recipes
Developing recipes
Writing recipes
Starting out smallInstalling a simple serviceGetting more advanced
Summary
5. Testing Your Recipes
Testing recipes
RSpec
RSpec and ChefSpec
Testing basicsComparing RSpec with other testing librariesUsing ChefSpec
Getting started with ChefSpec
Installing ChefSpecLocking your dependencies in RubyCreating a simple recipe and a matching ChefSpec testWriting a ChefSpec testBuilding your recipeExecuting testsUnderstanding failuresExpanding your testsMultiple examples in a spec testTesting for multiple platforms
Summary
6. From Development to Deployment
Describing the setup
Deploying software with Chef
Configuring your local environment
Modeling a simple Python application
Managing the cookbooks
Downloading cookbooksLooking at the database recipeLooking at your application deployment cookbookPreparing the directoriesConstructing your Python virtual environmentChecking the source codeInstalling any extra dependenciesManaging dependencies in ChefManaging dependencies elsewhereUsing Python's requirements fileConfiguring your applicationKeeping your application running
Defining roles
Creating the base server roleCreating the database server roleCreating the web server role
Adding users
Provisioning EC2 instances
Configuring the database host
Configuring the web server
Deploying your software
Manually deploying updatesAutomating deployment
Summary
7. Beyond Basic Recipes and Cookbooks
Managing usersEvolution of a shell user recipeStoring data in data bagsCreating a data bag for usersSearching for dataSearching inside recipesEnhancing your user cookbookDistributing SSH keysTemplating the authorized keysAdding deployment keys
Writing custom extensions
Developing a custom definitionOrganizing your codeWriting a definition for using PIPDefining a full application templateBuilding a resourceDefining the resourceImplementing the providerModifying resourcesLoading an existing resourceDeclaring that a resource was updated
Working with data bags
Securing your data bagsSecret keysEncrypting your dataDecrypting your dataStoring keys on nodesSearching your dataSearching your data bags with knifeSearching your data bags from a recipeQuerying your dataManaging multiple machines with search queries
Summary
8. Extras You Need to Know
Vagrant and Chef-soloInstalling VagrantProvisioning a new host with VagrantBooting your Vagrant imageCombining Vagrant with Chef-soloUnderstanding the limitations of Chef-soloConfiguring Chef-soloTelling Chef-solo what to runUsing roles and data bags with Chef-soloInjecting custom JSON dataProviding a custom node name
Getting to know the Chef shell
Using the Chef shellThe standalone modeThe solo modeThe client modeInteracting with the Chef server using the shellInteracting with dataSearching your dataEditing your dataTransforming dataExecuting recipes with Chef shellCreating a recipe in the shellDefining node attributesSetting attributesAccessing attributesUsing configuration blocksInteractively executing recipesDebugging with the Chef shellUsing the breakpoint resource
Integration testing
Using Test KitchenInstalling Test KitchenTesting with Test KitchenBuilding a simple cookbookPreparing your cookbook for the kitchenTesting your new cookbookProvisioning the instanceConverging the newly created instanceWriting a simple testCombining all the steps
Extending Chef
Writing an Ohai pluginA note about writing Ohai pluginsChef with Capistrano
Automation and integration
Automated updates and deployments
Summary
2. Module 2
1. Chef Infrastructure
Introduction
Using version control
Getting readyHow to do it...How it works...There's more...See also
Installing the Chef development kit on your workstation
How to do it...How it works...See also
Using the hosted Chef platform
Getting readyHow to do it...How it works...There's more...See also
Managing virtual machines with Vagrant
Getting readyHow to do it...How it works...There's more...See also
Creating and using cookbooks
Getting readyHow to do it...How it works...See also
Inspecting files on your Chef server with knife
Getting readyHow to do it...How it works...There's more...See also
Defining cookbook dependencies
Getting readyHow to do it...How it works...There's more...See also
Managing cookbook dependencies with Berkshelf
Getting readyHow to do it...How it works...There's more...See also
Downloading and integrating cookbooks as vendor branches into your Git repository
Getting readyHow to do it...How it works...There's more...See also
Using custom knife plugins
Getting readyHow to do it...How it works...There's more...See also
Deleting a node from the Chef server
Getting readyHow to do it...How it works...There's more...See also
Developing recipes with local mode
Getting readyHow to do it...How it works...There's more...Running knife in local modeMoving to hosted Chef or your own Chef serverSee also
Using roles
Getting readyHow to do it...How it works...See also
Using environments
Getting readyHow to do it...How it works...There's more...See also
Freezing cookbooks
Getting readyHow to do it...How it works...There's more...See also
Running Chef client as a daemon
Getting readyHow to do it...How it works...There's more...
Using chef-shell
How to do it...How it works...There's more...See also
2. Evaluating and Troubleshooting Cookbooks and Chef Runs
Introduction
Testing your Chef cookbooks
Getting readyHow to do it...How it works...There's more...See also
Flagging problems in your Chef cookbooks
Getting readyHow to do it...How it works...There's more...See also
Test-driven development for cookbooks using ChefSpec
Getting readyHow to do it...How it works...There's more...See also
Integration testing your Chef cookbooks with Test Kitchen
Getting readyHow to do it...How it works...There's more...See also
Showing affected nodes before uploading cookbooks
Getting readyHow to do it...How it works...See also
Overriding a node's run list to execute a single recipe
Getting readyHow to do it...How it works...See also
Using why-run mode to find out what a recipe might do
Getting readyHow to do it...How it works...See also
Debugging Chef client runs
Getting readyHow to do it...How it works...There's more...See also
Inspecting the results of your last Chef run
Getting readyHow to do it...How it works...See also
Raising and logging exceptions in recipes
Getting readyHow to do it...How it works...There's more...See also
Diff-ing cookbooks with knife
Getting readyHow to do it...How it works...There's more...See also
Using community exception and report handlers
Getting readyHow to do it...How it works...There's more...See also
Creating custom handlers
Getting ready...How to do it...How it works...There's more...See also
3. Chef Language and Style
Introduction
Using community Chef style
Getting readyHow to do it...How it works...There's more...See also
Using attributes to dynamically configure recipes
Getting readyHow to do it...How it works...There's more...Calculating values in the attribute filesSee also
Using templates
Getting readyHow to do it...How it works...There's more...See also
Mixing plain Ruby with Chef DSL
Getting readyHow to do it...How it works...There's more...See also
Installing Ruby gems and using them in recipes
Getting readyHow to do it...How it works...See also
Using libraries
Getting readyHow to do it...How it works...There's more...See also
Using definitions
Getting readyHow to do it...How it works...There's more...See also
Creating your own Lightweight Resource Providers (LWRP)
Getting readyHow to do it...How it works...There's more...See also
Extending community cookbooks by using application wrapper cookbooks
Getting readyHow to do it...How it works...There's more...See also
Creating custom Ohai plugins
Getting readyHow to do it...How it works...There's more...See also
Creating custom knife plugins
Getting readyHow to do it...How it works...There's more...See also
4. Writing Better Cookbooks
Introduction
Setting the environment variables
Getting readyHow to do it...How it works...There's more...See also
Passing arguments to shell commands
Getting readyHow to do it...How it works...There's more...See also
Overriding attributes
Getting readyHow to do it...How it works...There's more...See also
Using search to find nodes
Getting readyHow to do it...How it works...There's more...Using knife to search for nodesSearching for arbitrary node attributesUsing Boolean operators in searchSee also
Using data bags
Getting readyHow to do it...How it works...See also
Using search to find data bag items
Getting readyHow to do it...How it works...There's more...See also
Using encrypted data bag items
Getting readyHow to do it...How it works...There's more...Using a private key fileSee also
Accessing data bag values from external scripts
Getting readyHow to do it...How it works...There's more...See also
Getting information about the environment
Getting readyHow to do it...How it works...There's more...See also
Writing cross-platform cookbooks
Getting readyHow to do it...How it works...There's more...Avoiding case statements to set values based on the platformDeclaring support for specific operating systems in your cookbook's metadataSee also
Finding the complete list of operating systems you can use in cookbooks
How to do it...How it works...There's more...See also
Making recipes idempotent by using conditional execution
Getting readyHow to do it...How it works...There's more...See also
5. Working with Files and Packages
Introduction
Creating configuration files using templates
Getting readyHow to do it...How it works...There's more...See also
Using pure Ruby in templates for conditionals and iterations
Getting readyHow to do it...How it works...There's more...See also
Installing packages from a third-party repository
Getting readyHow to do it...How it works...See also
Installing software from source
Getting readyHow to do it...How it works...There's more...See also
Running a command when a file is updated
Getting readyHow to do it...How it works...There's more...See also
Distributing directory trees
Getting readyHow to do it...How it works...There's more...See also
Cleaning up old files
Getting readyHow to do it...How it works...There's more...See also
Distributing different files based on the target platform
Getting readyHow to do it...How it works...See also
6. Users and Applications
Introduction
Creating users from data bags
Getting readyHow to do it...How it works...There's more...See also
Securing the Secure Shell Daemon (SSHD)
Getting readyHow to do it...How it works...There's more...See also
Enabling passwordless sudo
Getting readyHow to do it...How it works...There's more...See also
Managing NTP
Getting readyHow to do it...How it works...There's more...See also
Managing nginx
Getting readyHow to do it...How it works...There's more...See also
Creating nginx virtual hosts
Getting readyHow to do it...How it works...There's more...See also
Creating MySQL databases and users
Getting readyHow to do it...How it works...There's more...See also
Managing WordPress sites
Getting readyHow to do it...How it works...There's more...See also
Managing Ruby on Rails applications
Getting readyHow to do it...How it works...There's more...See also
Managing Varnish
Getting readyHow to do it...How it works...There's more...See also
Managing your local workstation
Getting readyHow to do it...How it works...There's more...See also
7. Servers and Cloud Infrastructure
Introduction
Creating your infrastructure using Chef Provisioning
Getting readyHow to do it...How it works...There's more...See also
Creating cookbooks from a running system with Blueprint
Getting readyHow to do it...How it works...There's more...See also
Running the same command on many machines at once
How to do it...How it works...There's more...See also
Setting up SNMP for external monitoring services
Getting readyHow to do it...How it works...There's more...See also
Deploying a Nagios monitoring server
Getting readyHow to do it...How it works...There's more...See also
Building high-availability services using heartbeat
Getting readyHow to do it...How it works...There's more...See also
Using HAProxy to load-balance multiple web servers
Getting readyHow to do it...How it works...See also
Using custom bootstrap scripts
Getting readyHow to do it...How it works...There's more...See also
Managing firewalls with iptables
Getting readyHow to do it...How it works...See also
Managing fail2ban to ban malicious IP addresses
Getting readyHow to do it...How it works...There's more...See also
Managing Amazon EC2 instances
Getting readyHow to do it...How it works...There's more...See also
Loading your Chef infrastructure from a file with spiceweasel and knife
Getting readyHow to do it...How it works...There's more...See also
3. Module 3
1. Setting Up a Development Environment on Your Workstation
Re-introducing ChefWhat version of the Chef tools do I use?References
Filling in the gaps in your Chef knowledge base
Major pieces of a Chef puzzleThe Chef clientThe Ohai Chef toolRecipes and cookbooksChef run listsChef rolesChef resourcesMore things to considerData bagsEnvironmentsThe Chef supermarketChef Development KitSetting up an on-premise (private) Chef ServerReferences
Exploring the Chef Development Kit
The Chef KnifeCommunity toolsReferences
Installing Chef Development Kit
Downloading ChefDKInstalling ChefDKVerifying ChefDKReferences
Setting up a standard Chef repo
Using the Chef tool for Chef repoThe repo starter kitFiles for your .chef folder – a safer wayGrowing your own treeSelf-signed certificatesReferences
What else do you need?
Using your favorite editorVersion control systemsVirtualization hypervisorVagrantReferences
What is chef-apply?
Why is that useful?References
Summary
2. Knife Tooling and Plugins
Knife helpReferences
Knife subcommands
Knife BootstrapKnife NodeKnife DiffKnife ExecKnife SSHKnife SearchReferences
Knife to editor integration
Using the "wait" optionDoes my editor require the wait option?References
Knife Windows plugin
Installing the Knife Windows pluginBootstrapping a Windows nodeSearching for Windows node dataReferences
Knife EC2 plugin
Installing the Knife EC2 pluginCreating an EC2 instanceBootstrapping an existing EC2 instanceDeleting EC2 instancesReferences
Creating new Knife plugins
The basic plugin file formatExample – A simpler EC2 Create Server pluginReferences
Summary
3. Leveraging Roles, Environments, and Policies
Using Chef rolesHow to create a roleUsing per-environment run-lists in rolesUsing nested rolesThere is a downside to rolesReferences
Using Chef environments
The _default environmentCreating environmentsSpecifying cookbook versions in environmentsReferences
Using Chef organizations
Creating organizationsAccess controlReferences
Learning the attribute precedence hierarchy
Understanding attribute precedence factorsResulting precedence hierarchy
Exploring policies
What exactly is a policy?Why would I use a policy?How do I use a policy?Another exampleHow to specify the source options for cookbooksWhat are the "Gotchas"?References
Summary
4. Custom Resources
Out with the old (LWRPs), in with the new (Custom Resources)What is the difference between the old way and the new way of defining resources?What is a Custom Resource?References
Creating an example use case
What is the goal?Creating the cookbookCreating the default recipeThe recipe worked, but what is missing?References
Creating and using a Custom Resource
What makes up a Custom Resource?Creating the "website" Custom ResourceUsing our Custom ResourceReferences
Making a Custom Resource more reusable
Making a Custom Resource-only cookbookReuse the code from the "mywebapp" exampleUsing the reusable Custom Resource-only cookbookHandling idempotenceReferences
Subclassing a Custom Resource
Convert our resource into a library moduleSubclassing the library module Custom ResourceReferences
Summary
5. Provisioning in the Traditional Data Center
VMware provisioningInstalling the knife-vsphere pluginProvisioning VMs using the knife-vsphere pluginCleaning up our vSphere and Chef environmentsUsing Chef with VMware and VagrantReferences
OpenStack provisioning
Setting up a DevStack OpenStack all-in-one environmentDeploying the DevStack Host VMDownloading and installing DevStackDownloading the demo project rc file and applying itAdding the security rules to allow ping and ssh trafficInstalling the knife OpenStack pluginSetting up the DevStack public keyAllocating some floating IP addresses to the projectConfiguring the knife-openstack plugin knife settingsAdding a CentOS instance image to provisionProvisioning a new instanceReferences
Network automation
Automating juniper devicesAutomating Cisco systemsAutomating F5 LTMsReferences
Summary
6. Provisioning in the Cloud
Provisioning AWS EC2 instancesUnderstanding EC2 authenticationCreating key pairsCreating security groupsInstalling Chef Server into an EC2 instanceReferences
Provisioning Microsoft Azure instances
Installing the knife-azure pluginConfiguring the pluginVerifying the configurationGenerating public/private key pairsProvisioning virtual machinesProvisioning a new Chef server into AzureReferences
Provisioning in the Google Cloud platform
Installing and configuring the knife-google pluginCreating and registering your key pairsProvisioning virtual machines with the GCEDeploying Chef server into a GCE instanceReferences
Looking at an honorable mention – Linode
References
Provisioning containers in Docker
Setting up a Docker server to host our containersInstalling and using Docker on our Chef WorkstationUsing the knife-docker pluginReferences
Summary
7. Test-Driven Development
Analyzing cookbooks with RuboCopUsing RuboCopExamining the RuboCop outputCustomizing RuboCopReferences
Analyzing cookbooks with Foodcritic
Using FoodcriticExamining the Foodcritic outputCustomizing FoodcriticReferences
Unit testing with ChefSpec
Using ChefSpecExpanding our testsReferences
Integration testing with Kitchen and ServerSpec
Preparing our testsKitchen createKitchen convergeKitchen verifyReferences
Summary
8. Using Chef Provisioning
What is Chef provisioning?It's highly idempotentA hypothetical exampleReferences
How to use Chef Provisioning
Chef Provisioning resourcesA Vagrant exampleA cloud example with AWSA Docker exampleReferences
Summary
A. Bibliography
Index

Content preview from Chef: Powerful Infrastructure Automation

Large-scale infrastructure

One of the many benefits of Chef is the power to apply roles to nodes at scale. This means that once you define a set of roles and some supporting recipes, you can apply them to one host just as easily as any other. There are many organizations that manage very large infrastructure using Chef, including companies such as Facebook, Ancestry, and Riot Games. With Chef, configuring one hundred hosts is as straightforward as configuring one host. Being able to achieve scalability goals while remaining cost-effective is a critical part of running a technology business. To this end, Chef provides tools to automate the creation, provisioning, maintenance and termination of virtual hosts using the provided tools, which can help ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781788392976Purchase Link

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Chef: Powerful Infrastructure Automation

by John Ewart, Matthias Marschall, Earl Waud

Large-scale infrastructure

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

More than 5,000 organizations count on O’Reilly

Julian F.

Addison B.

Amir M.

Mark W.

You might also like

Automation through Chef Opscode

Chef Infrastructure Automation Cookbook Second Edition

Mastering Chef Provisioning

Testing Cloud Services

Publisher Resources