The Logstash Book

The Logstash Book

by James Turnbull
Released December 2013
Publisher(s): Turnbull Press
ISBN: None

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Updated for Logstash and ELK v5.0.0.

A book designed for SysAdmins, Operations staff, Developers and DevOps who are interested in deploying a log management solution using the open source Elasticsearch Logstash & Kibana or ELK stack.

In this book, we will walk you through installing, deploying, managing and extending Logstash. We're going to do that by introducing you to Example.com, where you're going to start a new job as one of its SysAdmins. The first project you'll be in charge of is developing its new log management solution.

We'll teach you how to:

* Install and deploy Logstash.
* Ship events from a Logstash Shipper to a central Logstash server.
* Filter incoming events using a variety of techniques.
* Add structured logging to your applications and parse your application logs.
* Output those events to a selection of useful destinations.
* Use Logstash's awesome web interface Kibana.
* Scale out your Logstash implementation as your environment grows.
* Quickly and easily extend Logstash to deliver the additional functionality you might need.

By the end of the book, you should have a functional and effective log management solution that you can deploy into your own environment.

Table of contents

  1. The Logstash Book
    1. Who is this book for?
    2. Credits and Acknowledgments
    3. Technical Reviewers
      1. Jan-Piet Mens
      2. Paul Stack
    4. Technical Illustrator
    5. Author
    6. Conventions in the book
    7. Code and Examples
    8. Colophon
    9. Errata
    10. Trademarks
    11. Version
    12. Copyright
  2. Introduction or Why Should I Bother?
    1. Introducing Logstash and the Elastic Stack
    2. Logstash design and architecture
    3. What's in the book?
    4. Logstash resources
    5. Getting help with Logstash
    6. A mild warning
  3. Getting Started with Logstash
    1. Installing Java
      1. On the Red Hat family
      2. On Debian & Ubuntu
      3. Testing Java is installed
    2. Getting Logstash
    3. Starting Logstash
      1. Our sample configuration file
      2. Running the Logstash agent
      3. Testing the Logstash agent
    4. Summary
  4. Shipping Events
    1. Our Event Lifecycle
    2. Installing Logstash on our central server
      1. Installing Java
      2. Installing Logstash
      3. Elasticsearch for search
      4. Creating a basic central configuration
      5. Running Logstash as a service
    3. An interlude about plugins
    4. The Kibana Console
      1. Installing Kibana
      2. Configuring Kibana
      3. Running Kibana
    5. Installing a Filebeat on our first agent
      1. Installing the Filebeat
      2. Our agent configuration
      3. Installing Filebeat as a service
    6. Sending our first events
      1. Looking at our events in Kibana
    7. Summary
  5. Shipping Events
    1. Using Syslog
      1. A quick introduction to Syslog
      2. Configuring Logstash for Syslog
      3. Configuring Syslog on remote agents
    2. Filebeat
      1. Configure Filebeat on our central server
      2. Installing Filebeat on the remote host
      3. Configuring Filebeat
    3. Other log shippers
      1. Log-Courier
      2. Beaver
      3. Woodchuck
      4. Others
    4. Summary
  6. Filtering Events with Logstash
    1. Apache Logs
      1. Configuring Apache for Custom Logging
      2. Sending Apache events to Logstash
    2. Postfix Logs
      1. Filtering
      2. Collecting Postfix logs
      3. Our first filter
      4. Adding our own filters
      5. Extracting from different events
      6. Setting the timestamp
    3. Filtering Java application logs
      1. Handling blank lines with drop
      2. Handling multi-line log events
      3. Grokking our Java events
    4. Parsing an in-house custom log format
    5. Summary
  7. Structured Application Logging
    1. Application logging primer
      1. Where should I instrument?
      2. Instrument schemas
      3. Time and the observer effect
      4. Logging patterns, or where to put your logging
      5. The utility pattern
      6. The external pattern
    2. Adding our own structured log entries
      1. Adding structured logging to a sample application
      2. Structured logging libraries
    3. Working with your existing logs
    4. Summary
  8. Outputting Events from Logstash
    1. Send email alerts
      1. Updating our multiline filter
      2. Configuring the email output
      3. Email output
    2. Send instant messages
      1. Identifying the event to send
      2. Sending the instant message
    3. Send alerts to Nagios
      1. Nagios check types
      2. Identifying the trigger event
      3. The nagios output
      4. The Nagios external command
      5. The Nagios service
    4. Outputting metrics
      1. Collecting metrics
      2. StatsD
      3. Setting the date correctly
      4. The StatsD output
      5. Sending to a different StatsD server
    5. Summary
  9. Scaling Logstash
    1. Scaling Elasticsearch
      1. Installing additional Elasticsearch hosts
      2. Monitoring our Elasticsearch cluster
      3. Managing Elasticsearch data retention
      4. More Information
    2. Scaling Logstash
      1. Creating a second indexer
      2. Load balancing
    3. Summary
  10. Extending Logstash
    1. Plugin organization
    2. Anatomy of a plugin
    3. Creating our own input plugin
    4. Building our plugin
    5. Adding new plugins
    6. Writing a filter
    7. Writing an output
    8. Summary

Product information

  • Title: The Logstash Book
  • Author(s): James Turnbull
  • Release date: December 2013
  • Publisher(s): Turnbull Press
  • ISBN: None