Pro Python System Administration, Second Edition

Pro Python System Administration, Second Edition

Released November 2014
Publisher(s): Apress
ISBN: 9781484202173

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Pro Python System Administration, Second Edition explains and shows how to apply Python scripting in practice. It will show you how to approach and resolve real-world issues that most system administrators will come across in their careers. This book has been updated using Python 2.7 and Python 3 where appropriate. It also uses various new and relevant open source projects and tools that should now be used in practice.

In this updated edition, you will find several projects in the categories of network administration, web server administration, and monitoring and database management. In each project, the author will define the problem, design the solution, and go through the more interesting implementation steps. Each project is accompanied by the source code of a fully working prototype, which you’ll be able to use immediately or adapt to your requirements and environment.

This book is primarily aimed at experienced system administrators whose day-to-day tasks involve looking after and managing small-to-medium-sized server estates. It will also be beneficial for system administrators who want to learn more about automation and want to apply their Python knowledge to solve various system administration problems. Python developers will also benefit from reading this book, especially if they are involved in developing automation and management tools.

Table of contents

  1. Cover
  2. Title
  3. Copyright
  4. Dedication
  5. Contents at a Glance
  6. Contents
  7. About the Author
  8. About the Technical Reviewers
  9. Acknowledgments
  10. Introduction
  11. Chapter 1: Reading and Collecting Performance Data Using SNMP
    1. Application Requirements and Design
      1. Specifying the Requirements
      2. High-Level Design Specification
    2. Introduction to SNMP
      1. The System SNMP Variables Node
      2. The Interfaces SNMP Variables Node
      3. Authentication in SNMP
      4. Querying SNMP from the Command Line
    3. Querying SNMP Devices from Python
      1. Configuring the Application
      2. Using the PySNMP Library
      3. Implementing the SNMP Read Functionality
    4. Storing Data with RRDTool
      1. Introduction to RRDTool
      2. Using RRDTool from a Python Program
      3. Creating a Round Robin Database
      4. Writing and Reading Data from the Round Robin Database
      5. Plotting Graphs with RRDTool
      6. Integrating RRDTool with the Monitoring Solution
    5. Creating Web Pages with the Jinja2 Templating System
      1. Loading Template Files with Jinja2
      2. The Jinja2 Template Language
      3. Generating Website Pages
    6. Summary
  12. Chapter 2: Managing Devices Using the SOAP API
    1. What Is the SOAP API?
      1. The Structure of a SOAP Message
      2. Requesting Services with SOAP
      3. Finding Information About Available Services with WSDL
    2. SOAP Support in Python
    3. Converting WSDL Schema to Python Helper Module
    4. Defining Requirements for Our Load Balancer Tool
      1. Basic Requirements
      2. Code Structure
      3. Configuration
    5. Accessing Citrix Netscaler Load Balancer with the SOAP API
      1. Fixing Issues with Citrix Netscaler WSDL
      2. Creating a Connection Object
      3. Logging In: Our First SOAP Call
    6. Gathering Performance Statistics Data
      1. SOAP Methods for Reading Statistical Data and its Return Values
      2. Reading System Health Data
      3. Reading Service Status Data
    7. Automating Some Administration Tasks
      1. Device Configuration SOAP Methods
      2. Setting a Service State
    8. A Word About Logging and Error Handling
      1. Using the Python Logging Module
      2. Handling Exceptions
    9. NetScaler NITRO API
      1. Download
      2. Using the Nitro-Python Module
    10. Summary
  13. Chapter 3: Creating a Web Application for IP Address Accountancy
    1. Designing the Application
      1. Setting out the Requirements
      2. Making Design Decisions
      3. Defining the Database Schema
      4. Creating the Application Workflow
    2. The Basic Concepts of the Django Framework
      1. What Is Django?
      2. The Model/View/Controller Pattern
      3. Installing the Django Framework
      4. The Structure of a Django Application
      5. Using Django with Apache Web Server
    3. Implementing Basic Functionality
      1. Defining the Database Model
      2. URL Configuration
      3. Using the Management Interface
      4. Viewing Records
      5. Using Templates
      6. Deleting Records
      7. Adding New Records
      8. Modifying Existing Records
    4. Summary
  14. Chapter 4: Integrating the IP Address Application with DHCP
    1. Extending the Design and Requirements
      1. Extending the Database Schema
      2. Making Additions to the Workflow
    2. Adding DHCP Network Data
      1. The Data Models
      2. Additional Workflows
      3. The Add Function
      4. The Modify Function
      5. The Delete Function
    3. Extending the DHCP Configuration with Address Pools
      1. The Address Pool Data Model
      2. The DHCP Network Details
      3. The Add and Delete Functions
    4. Reworking the URL Structure
      1. Generation of URLs in the Model Class
      2. Reverse Resolution of URLs
      3. Assignment of Names to URL Patterns
      4. Use of URL References in the Templates
    5. Adding Client Classification
      1. Additions to the Data Model
      2. Template Inheritance
      3. Class Rules Management
    6. Generating the DHCP Configuration File
    7. Other Modifications
      1. Resolving IPs to Hostnames
      2. Checking Whether the Address Is in Use
    8. Dynamic DHCP Lease Management
      1. Employ Python Interface to OMAPI
      2. Set up the ISC DHCP Server
      3. Add a New Host Lease Record
      4. Delete a Host Lease Record
    9. Summary
  15. Chapter 5: Maintaining a List of Virtual Hosts in an Apache Configuration File
    1. Specifying the Design and Requirements for the Application
      1. Functional Requirements
      2. High-Level Design
    2. Setting Up the Environment
      1. Apache Configuration
      2. Creating a Django Project and Application
      3. Configuring the Application
      4. Defining the URL Structure
    3. The Data Model
      1. The Basic Model Structure
    4. Modifying the Administration Interface
      1. Improving the Class and Object Lists
      2. Adding Custom Object Actions
    5. Generating the Configuration File
    6. Summary
  16. Chapter 6: Gathering and Presenting Statistical Data from Apache Log Files
    1. Application Structure and Functionality
      1. Application Requirements
      2. Application Design
    2. Plug-in Framework Implementation in Python
      1. The Mechanics of a Plug-in Framework
      2. Creating the Plug-in Framework
    3. Log-Parsing Application
      1. Format of Apache Log Files
      2. Log File Reader
      3. Calling the Plug-in Methods
    4. Plug-in Modules
      1. Installing the Required Libraries
      2. Writing the Plug-in Code
      3. Visualizing the Data
    5. Summary
  17. Chapter 7: Performing Complex Searches and Reporting on Application Log Files
    1. Defining the Problem
      1. Why We Use Exceptions
      2. Are Exceptions Always a Bad Sign?
      3. Why We Should Analyze Exceptions
    2. Parsing Complex Log Files
      1. What Can We Find in a Typical Log File?
      2. The Structure of an Exception Stack Trace Log
    3. Handling Multiple Files
      1. Handling Multiple Files
      2. Using the Built-in Bzip2 Library
      3. Traversing Large Data Files
      4. What Are Generators, and How Are They Used?
    4. Detecting the Exceptions
      1. Detecting the Potential Candidates
      2. Filtering the Legitimate Exception Traces
    5. Storing Data in Data Structures
      1. The Structure of Exception Stack Trace Data
      2. Generating an Exception Fingerprint for Unknown Exceptions
      3. Detecting Known Exceptions
    6. Producing Reports
      1. Grouping the Exceptions
      2. Producing Differently Formatted Outputs for the Same Dataset
      3. Calculating Group Statistics
    7. Summary
  18. Chapter 8: A Website Availability Check Script for Nagios
    1. Requirements for the Check System
    2. The Nagios Monitoring System
      1. Nagios Plug-In Architecture
    3. The Site Navigation Check
      1. Installing the Beautiful Soup HTML Parsing Library
      2. Retrieving a Web Page
      3. Parsing the HTML Pages with Beautiful Soup
      4. Adding the New Check to the Nagios System
      5. Emulating the User Login Process
      6. Simplifying HTTP Client with Requests Module
    4. Summary
  19. Chapter 9: Management and Monitoring Subsystem
    1. Design
      1. The Components
      2. The Data Objects
    2. The Data Structures
      1. Introduction to Data Normalization
      2. Configuration Data
      3. Performance Data
      4. Scheduling
      5. Site Configuration
    3. Communication Flows
      1. XML-RPC for Information Exchange
      2. CherryPy
    4. The Server Process
      1. Storing Data in a SQLite3 Database
      2. Actions
    5. The Scheduler
      1. Actions
      2. Running Multiple Processes
      3. Running Methods at Equal Intervals
      4. A Cron-Like Scheduler
      5. Ticket Dispatcher
    6. Summary
  20. Chapter 10: Remote Monitoring Agents
    1. Design
      1. The Passive Component
      2. Architecture
      3. Actions
    2. The Security Model
    3. Configuration
      1. The ConfigParser Library
      2. The Configuration Class Wrapper
    4. The Sensor Design
    5. Running External Processes
      1. Using the Subprocess Library
      2. Controlling the Running Processes
      3. Communicating with External Processes
    6. Automatically Updating Sensor Code
      1. Sending and Receiving Binary Data with XML-RPC
      2. Working with Files and Archives (TAR and BZip2)
    7. Summary
  21. Chapter 11: Statistics Gathering and Reporting
    1. Application Requirements and Design
    2. Using the NumPy Library
      1. Installing NumPy
      2. NumPy Examples
    3. Representing Data with matplotlib
      1. Installing matplotlib
      2. Understanding the Library Structure
      3. Plotting Graphs
      4. Saving Plots to a File
    4. Graphing Statistical Data
      1. Collating Data from the Database
      2. Drawing Timescale Graphs
      3. Host Details Page
    5. Summary
  22. Chapter 12: Distributed Message Processing System
    1. Quick Introduction to Message and Task Queues
      1. Task Queuing Systems
      2. Message Queuing Systems
    2. Setting up the Celery Server and Client
      1. Installing and Setting up RabbitMQ
      2. Installing and Setting up Celery
    3. Celery Basics
      1. Layout of a Typical Celery Application
      2. Creating aTasks Module
      3. Routing Tasks
    4. Summary
  23. Chapter 13: Automatic MySQL Database Performance Tuning
    1. Requirements Specification and Design
      1. Basic Application Requirements
      2. System Design
    2. Modifying the Plug-in Framework
      1. Changes to the Host Application
      2. Modifying the Plug-in Manager
    3. Writing the Producer Plug-ins
      1. Accessing the MySQL Database from Python Applications
      2. Querying the Configuration Variables
      3. Querying the Server Status Variables
      4. Collecting the Host Configuration Data
    4. Writing the Consumer Plug-ins
      1. Checking the MySQL Version
      2. Checking the Key Buffer Size Setting
      3. Checking the Slow Queries Counter
    5. Summary
  24. Chapter 14: Using Amazon EC2/S3 as a Data Warehouse Solution
    1. Specifying the Problem and the Solution
      1. The Problem
      2. Our Solution
      3. Design Specifications
    2. The Amazon EC2 and S3 Crash Course
      1. Authentication and Security
      2. The Simple Storage System Concepts
      3. The Elastic Computing Cloud Concepts
      4. User Interfaces
    3. Creating a Custom EC2 Image
      1. Reusing Existing Images
      2. Making Modifications
      3. Bundling the New AMI
    4. Controlling the EC2 Using the Boto Python Module
      1. Setting Up the Configuration Variables
      2. Initializing the EC2 Instance Programmatically
      3. Transferring the Data
      4. Destroying the EC2 Instance Programmatically
    5. Summary
  25. Index

Product information

  • Title: Pro Python System Administration, Second Edition
  • Author(s):
  • Release date: November 2014
  • Publisher(s): Apress
  • ISBN: 9781484202173