book

Python for DevOps

by Noah Gift, Kennedy Behrman, Alfredo Deza, Grig Gheorghiu

December 2019

Intermediate to advanced

506 pages

11h 55m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

What Does DevOps Mean to the Authors?How to Use This BookConceptual TopicsConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsNoahKennedyAlfredoGrig
Installing and Running PythonThe Python ShellJupyter NotebooksProcedural ProgrammingVariablesBasic MathCommentsBuilt-in FunctionsPrintRangeExecution Controlif/elif/elsefor Loopswhile LoopsHandling ExceptionsBuilt-in ObjectsWhat Is an Object?Object Methods and AttributesSequencesFunctionsAnatomy of a FunctionFunctions as ObjectsAnonymous FunctionsUsing Regular ExpressionsSearchingCharacter SetsCharacter ClassesGroupsNamed GroupsFind AllFind IteratorSubstitutionCompilingLazy EvaluationGeneratorsGenerator ComprehensionsMore IPython FeaturesUsing IPython to Run Unix Shell CommandsExercises
Reading and Writing FilesUsing Regular Expressions to Search TextDealing with Large FilesEncrypting TextHashing with HashlibEncryption with CryptographyThe os ModuleManaging Files and Directories Using os.pathWalking Directory Trees Using os.walkPaths as Objects with Pathlib
Working with the ShellTalking to the Interpreter with the sys ModuleDealing with the Operating System Using the os ModuleSpawn Processes with the subprocess ModuleCreating Command-Line ToolsUsing sys.argvUsing argparseUsing clickfireImplementing Plug-insCase Study: Turbocharging Python with Command-Line ToolsUsing the Numba Just-in-Time (JIT) CompilerUsing the GPU with CUDA PythonRunning True Multicore Multithreaded Python Using NumbaKMeans ClusteringExercises
Disk UtilitiesMeasuring PerformancePartitionsRetrieving Specific Device InformationNetwork UtilitiesSSH TunnelingBenchmarking HTTP with Apache Benchmark (ab)Load Testing with molotovCPU UtilitiesViewing Processes with htopWorking with Bash and ZSHCustomizing the Python ShellRecursive GlobbingSearching and Replacing with Confirmation PromptsRemoving Temporary Python FilesListing and Filtering ProcessesUnix TimestampMixing Python with Bash and ZSHRandom Password GeneratorDoes My Module Exist?Changing Directories to a Module’s PathConverting a CSV File to JSONPython One-LinersDebuggersHow Fast Is this Snippet?straceExercisesCase Study Question
Why Is Packaging Important?When Packaging Might Not Be NeededPackaging GuidelinesDescriptive VersioningThe changelogChoosing a StrategyPackaging SolutionsNative Python PackagingDebian PackagingRPM PackagingManagement with systemdLong-Running ProcessesSetting It UpThe systemd Unit FileInstalling the UnitLog HandlingExercisesCase Study Question
Real-World Case Study: Converting a Poorly Maintained WordPress Site to HugoSetting Up HugoConverting WordPress to Hugo PostsCreating an Algolia Index and Updating ItOrchestrating with a MakefileDeploying with AWS CodePipelineReal-World Case Study: Deploying a Python App Engine Application with Google Cloud BuildReal-World Case Study: NFSOPS
Key Concepts in Building Reliable SystemsImmutable DevOps PrinciplesCentralized LoggingCase Study: Production Database Kills Hard DrivesDid You Build It or Buy It?Fault ToleranceMonitoringGraphiteStatsDPrometheusInstrumentationNaming ConventionsLoggingWhy Is It Hard?The basicconfigDeeper ConfigurationCommon PatternsThe ELK StackLogstashElasticsearch and KibanaExercisesCase Study Question
Testing Superpowers with pytestGetting Started with pytestTesting with pytestDifferences with unittestpytest Featuresconftest.pyThe Amazing assertParametrizationFixturesGetting StartedBuilt-in FixturesInfrastructure TestingWhat Is System Validation?Introduction to TestinfraConnecting to Remote NodesFeatures and Special FixturesExamplesTesting Jupyter Notebooks with pytestExercisesCase Study Question
Cloud Computing FoundationsTypes of Cloud ComputingTypes of Cloud ServicesInfrastructure as a ServiceMetal as a ServicePlatform as a ServiceServerless ComputingSoftware as a ServiceInfrastructure as CodeContinuous DeliveryVirtualization and ContainersHardware VirtualizationSoftware Defined NetworksSoftware Defined StorageContainersChallenges and Opportunities in Distributed ComputingPython Concurrency, Performance, and Process Management in the Cloud EraProcess ManagementManage Processes with SubprocessUsing Multiprocessing to Solve ProblemsForking Processes with Pool()Function as a Service and ServerlessHigh Performance Python with NumbaUsing Numba Just in Time CompilerUsing High-Performance ServersConclusionExercisesCase Study Questions

A Classification of Infrastructure Automation ToolsManual ProvisioningAutomated Infrastructure Provisioning with TerraformProvisioning an S3 BucketProvisioning an SSL Certificate with AWS ACMProvisioning an Amazon CloudFront DistributionProvisioning a Route 53 DNS RecordCopying Static Files to S3Deleting All AWS Resources Provisioned with TerraformAutomated Infrastructure Provisioning with PulumiCreating a New Pulumi Python Project for AWSCreating Configuration Values for the Staging StackProvisioning an ACM SSL CertificateProvisioning a Route 53 Zone and DNS RecordsProvisioning a CloudFront DistributionProvisioning a Route 53 DNS Record for the Site URLCreating and Deploying a New StackExercises
What Is a Docker Container?Creating, Building, Running, and Removing Docker Images and ContainersPublishing Docker Images to a Docker RegistryRunning a Docker Container with the Same Image on a Different HostRunning Multiple Docker Containers with Docker ComposePorting the docker-compose Services to a New Host and Operating SystemExercises
Short Overview of Kubernetes ConceptsUsing Kompose to Create Kubernetes Manifests from docker-compose.yamlDeploying Kubernetes Manifests to a Local Kubernetes Cluster Based on minikubeLaunching a GKE Kubernetes Cluster in GCP with PulumiDeploying the Flask Example Application to GKEInstalling Prometheus and Grafana Helm ChartsDestroying the GKE ClusterExercises
Deploying the Same Python Function to the “Big Three” Cloud ProvidersInstalling Serverless FrameworkDeploying Python Function to AWS LambdaDeploying Python Function to Google Cloud FunctionsDeploying Python Function to AzureDeploying a Python Function to Self-Hosted FaaS PlatformsDeploying Python Function to OpenFaaSProvisioning DynamoDB Table, Lambda Functions, and API Gateway Methods Using the AWS CDKExercises
What Is Machine Learning?Supervised Machine LearningModelingPython Machine learning EcosystemDeep Learning with PyTorchCloud Machine learning PlatformsMachine learning Maturity ModelMachine Learning Key TerminologyLevel 1: Framing, Scope Identification, and Problem DefinitionLevel 2: Continuous Delivery of DataLevel 3: Continuous Delivery of Clean DataLevel 4: Continuous Delivery of Exploratory Data AnalysisLevel 5: Continuous Delivery of Traditional ML and AutoMLLevel 6: ML Operational Feedback LoopSklearn Flask with Kubernetes and DockerSklearn Flask with Kubernetes and DockerEDAModelingTune Scaled GBMFit ModelEvaluateadhoc_predictJSON WorkflowScale Inputadhoc_predict from PickleScale InputExercisesCase Study QuestionLearning Assessments
Small DataDealing with Small Data FilesWrite a FileRead a FileGenerator Pipeline to Read and Process LinesUsing YAMLBig DataBig Data Tools, Components, and PlatformsData SourcesFilesystemsData StorageReal-Time Streaming IngestionCase Study: Building a Homegrown Data PipelineServerless Data EngineeringUsing AWS Lambda with CloudWatch EventsUsing Amazon CloudWatch Logging with AWS LambdaUsing AWS Lambda to Populate Amazon Simple Queue ServiceWiring Up CloudWatch Event TriggerCreating Event-Driven LambdasReading Amazon SQS Events from AWS LambdaConclusionExercisesCase Study Question
Film Studio Can’t Make FilmGame Studio Can’t Ship GamePython Scripts Take 60 Seconds to LaunchPutting Out a Fire with a Cache and Intelligent InstrumentationYou’ll Automate Yourself Out of a Job!DevOps AntipatternsNo Automated Build Server AntipatternFlying BlindDifficulties in Coordination as an Ongoing AccomplishmentNo TeamworkInterviewsGlenn SolomonAndrew NguyenGabriella RomanRigoberto RocheJonathan LaCourVille TuulosJoseph ReisTeijo HolzerMatt HarrisonMichael FoordRecommendationsExercisesChallengesCapstone Project

Overview

Much has changed in technology over the past decade. Data is hot, the cloud is ubiquitous, and many organizations need some form of automation. Throughout these transformations, Python has become one of the most popular languages in the world. This practical resource shows you how to use Python for everyday Linux systems administration tasks with today’s most useful DevOps tools, including Docker, Kubernetes, and Terraform.

Learning how to interact and automate with Linux is essential for millions of professionals. Python makes it much easier. With this book, you’ll learn how to develop software and solve problems using containers, as well as how to monitor, instrument, load-test, and operationalize your software. Looking for effective ways to "get stuff done" in Python? This is your guide.

Python foundations, including a brief introduction to the language
How to automate text, write command-line tools, and automate the filesystem
Linux utilities, package management, build systems, monitoring and instrumentation, and automated testing
Cloud computing, infrastructure as code, Kubernetes, and serverless
Machine learning operations and data engineering from a DevOps perspective
Building, deploying, and operationalizing a machine learning project