book

Python for DevOps

by Noah Gift, Kennedy Behrman, Alfredo Deza, Grig Gheorghiu

December 2019

Intermediate to advanced

506 pages

11h 55m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

What Does DevOps Mean to the Authors?How to Use This BookConceptual TopicsConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsNoahKennedyAlfredoGrig
Installing and Running PythonThe Python ShellJupyter NotebooksProcedural ProgrammingVariablesBasic MathCommentsBuilt-in FunctionsPrintRangeExecution Controlif/elif/elsefor Loopswhile LoopsHandling ExceptionsBuilt-in ObjectsWhat Is an Object?Object Methods and AttributesSequencesFunctionsAnatomy of a FunctionFunctions as ObjectsAnonymous FunctionsUsing Regular ExpressionsSearchingCharacter SetsCharacter ClassesGroupsNamed GroupsFind AllFind IteratorSubstitutionCompilingLazy EvaluationGeneratorsGenerator ComprehensionsMore IPython FeaturesUsing IPython to Run Unix Shell CommandsExercises
Reading and Writing FilesUsing Regular Expressions to Search TextDealing with Large FilesEncrypting TextHashing with HashlibEncryption with CryptographyThe os ModuleManaging Files and Directories Using os.pathWalking Directory Trees Using os.walkPaths as Objects with Pathlib
Working with the ShellTalking to the Interpreter with the sys ModuleDealing with the Operating System Using the os ModuleSpawn Processes with the subprocess ModuleCreating Command-Line ToolsUsing sys.argvUsing argparseUsing clickfireImplementing Plug-insCase Study: Turbocharging Python with Command-Line ToolsUsing the Numba Just-in-Time (JIT) CompilerUsing the GPU with CUDA PythonRunning True Multicore Multithreaded Python Using NumbaKMeans ClusteringExercises
Disk UtilitiesMeasuring PerformancePartitionsRetrieving Specific Device InformationNetwork UtilitiesSSH TunnelingBenchmarking HTTP with Apache Benchmark (ab)Load Testing with molotovCPU UtilitiesViewing Processes with htopWorking with Bash and ZSHCustomizing the Python ShellRecursive GlobbingSearching and Replacing with Confirmation PromptsRemoving Temporary Python FilesListing and Filtering ProcessesUnix TimestampMixing Python with Bash and ZSHRandom Password GeneratorDoes My Module Exist?Changing Directories to a Module’s PathConverting a CSV File to JSONPython One-LinersDebuggersHow Fast Is this Snippet?straceExercisesCase Study Question
Why Is Packaging Important?When Packaging Might Not Be NeededPackaging GuidelinesDescriptive VersioningThe changelogChoosing a StrategyPackaging SolutionsNative Python PackagingDebian PackagingRPM PackagingManagement with systemdLong-Running ProcessesSetting It UpThe systemd Unit FileInstalling the UnitLog HandlingExercisesCase Study Question
Real-World Case Study: Converting a Poorly Maintained WordPress Site to HugoSetting Up HugoConverting WordPress to Hugo PostsCreating an Algolia Index and Updating ItOrchestrating with a MakefileDeploying with AWS CodePipelineReal-World Case Study: Deploying a Python App Engine Application with Google Cloud BuildReal-World Case Study: NFSOPS
Key Concepts in Building Reliable SystemsImmutable DevOps PrinciplesCentralized LoggingCase Study: Production Database Kills Hard DrivesDid You Build It or Buy It?Fault ToleranceMonitoringGraphiteStatsDPrometheusInstrumentationNaming ConventionsLoggingWhy Is It Hard?The basicconfigDeeper ConfigurationCommon PatternsThe ELK StackLogstashElasticsearch and KibanaExercisesCase Study Question
Testing Superpowers with pytestGetting Started with pytestTesting with pytestDifferences with unittestpytest Featuresconftest.pyThe Amazing assertParametrizationFixturesGetting StartedBuilt-in FixturesInfrastructure TestingWhat Is System Validation?Introduction to TestinfraConnecting to Remote NodesFeatures and Special FixturesExamplesTesting Jupyter Notebooks with pytestExercisesCase Study Question
Cloud Computing FoundationsTypes of Cloud ComputingTypes of Cloud ServicesInfrastructure as a ServiceMetal as a ServicePlatform as a ServiceServerless ComputingSoftware as a ServiceInfrastructure as CodeContinuous DeliveryVirtualization and ContainersHardware VirtualizationSoftware Defined NetworksSoftware Defined StorageContainersChallenges and Opportunities in Distributed ComputingPython Concurrency, Performance, and Process Management in the Cloud EraProcess ManagementManage Processes with SubprocessUsing Multiprocessing to Solve ProblemsForking Processes with Pool()Function as a Service and ServerlessHigh Performance Python with NumbaUsing Numba Just in Time CompilerUsing High-Performance ServersConclusionExercisesCase Study Questions

A Classification of Infrastructure Automation ToolsManual ProvisioningAutomated Infrastructure Provisioning with TerraformProvisioning an S3 BucketProvisioning an SSL Certificate with AWS ACMProvisioning an Amazon CloudFront DistributionProvisioning a Route 53 DNS RecordCopying Static Files to S3Deleting All AWS Resources Provisioned with TerraformAutomated Infrastructure Provisioning with PulumiCreating a New Pulumi Python Project for AWSCreating Configuration Values for the Staging StackProvisioning an ACM SSL CertificateProvisioning a Route 53 Zone and DNS RecordsProvisioning a CloudFront DistributionProvisioning a Route 53 DNS Record for the Site URLCreating and Deploying a New StackExercises
What Is a Docker Container?Creating, Building, Running, and Removing Docker Images and ContainersPublishing Docker Images to a Docker RegistryRunning a Docker Container with the Same Image on a Different HostRunning Multiple Docker Containers with Docker ComposePorting the docker-compose Services to a New Host and Operating SystemExercises
Short Overview of Kubernetes ConceptsUsing Kompose to Create Kubernetes Manifests from docker-compose.yamlDeploying Kubernetes Manifests to a Local Kubernetes Cluster Based on minikubeLaunching a GKE Kubernetes Cluster in GCP with PulumiDeploying the Flask Example Application to GKEInstalling Prometheus and Grafana Helm ChartsDestroying the GKE ClusterExercises
Deploying the Same Python Function to the “Big Three” Cloud ProvidersInstalling Serverless FrameworkDeploying Python Function to AWS LambdaDeploying Python Function to Google Cloud FunctionsDeploying Python Function to AzureDeploying a Python Function to Self-Hosted FaaS PlatformsDeploying Python Function to OpenFaaSProvisioning DynamoDB Table, Lambda Functions, and API Gateway Methods Using the AWS CDKExercises
What Is Machine Learning?Supervised Machine LearningModelingPython Machine learning EcosystemDeep Learning with PyTorchCloud Machine learning PlatformsMachine learning Maturity ModelMachine Learning Key TerminologyLevel 1: Framing, Scope Identification, and Problem DefinitionLevel 2: Continuous Delivery of DataLevel 3: Continuous Delivery of Clean DataLevel 4: Continuous Delivery of Exploratory Data AnalysisLevel 5: Continuous Delivery of Traditional ML and AutoMLLevel 6: ML Operational Feedback LoopSklearn Flask with Kubernetes and DockerSklearn Flask with Kubernetes and DockerEDAModelingTune Scaled GBMFit ModelEvaluateadhoc_predictJSON WorkflowScale Inputadhoc_predict from PickleScale InputExercisesCase Study QuestionLearning Assessments
Small DataDealing with Small Data FilesWrite a FileRead a FileGenerator Pipeline to Read and Process LinesUsing YAMLBig DataBig Data Tools, Components, and PlatformsData SourcesFilesystemsData StorageReal-Time Streaming IngestionCase Study: Building a Homegrown Data PipelineServerless Data EngineeringUsing AWS Lambda with CloudWatch EventsUsing Amazon CloudWatch Logging with AWS LambdaUsing AWS Lambda to Populate Amazon Simple Queue ServiceWiring Up CloudWatch Event TriggerCreating Event-Driven LambdasReading Amazon SQS Events from AWS LambdaConclusionExercisesCase Study Question
Film Studio Can’t Make FilmGame Studio Can’t Ship GamePython Scripts Take 60 Seconds to LaunchPutting Out a Fire with a Cache and Intelligent InstrumentationYou’ll Automate Yourself Out of a Job!DevOps AntipatternsNo Automated Build Server AntipatternFlying BlindDifficulties in Coordination as an Ongoing AccomplishmentNo TeamworkInterviewsGlenn SolomonAndrew NguyenGabriella RomanRigoberto RocheJonathan LaCourVille TuulosJoseph ReisTeijo HolzerMatt HarrisonMichael FoordRecommendationsExercisesChallengesCapstone Project

Content preview from Python for DevOps

Chapter 6. Continuous Integration and Continuous Deployment

Author: Noah

The practices of continuous integration (CI) and continuous deployment (CD) are essential to a modern software development life cycle process. A CI system clones the codebase for the software under consideration from a source control system such as GitHub, builds the software into an artifact that can be a binary, a tar archive, or a Docker image, and, very importantly, also runs unit and/or integration tests for the software. A CD system deploys the artifacts built by the CI system to a target environment. This deployment can be automated for nonproduction environments, but usually includes a manual approval step for production. A more advanced type of such systems is a continuous delivery platform, which automates the deployment step to production and is capable of rolling back the deployment based on metrics obtained from monitoring and logging platforms.

Real-World Case Study: Converting a Poorly Maintained WordPress Site to Hugo

A while back, a friend asked for a favor fixing their company website. The company sold very expensive, used scientific equipment and its inventory was served via a WordPress site that was frequently hacked, performed horribly, or was down for days. Typically I try to avoid getting sucked into projects like this, but since it was a friend, I decided to help. You can reference the code for the conversion project at this Git repository.

Each step of the conversion process is covered ...