O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

The Logstash Book

Book Description

Updated for Logstash and ELK v5.0.0.

A book designed for SysAdmins, Operations staff, Developers and DevOps who are interested in deploying a log management solution using the open source Elasticsearch Logstash & Kibana or ELK stack.

In this book, we will walk you through installing, deploying, managing and extending Logstash. We're going to do that by introducing you to Example.com, where you're going to start a new job as one of its SysAdmins. The first project you'll be in charge of is developing its new log management solution.

We'll teach you how to:

* Install and deploy Logstash.
* Ship events from a Logstash Shipper to a central Logstash server.
* Filter incoming events using a variety of techniques.
* Add structured logging to your applications and parse your application logs.
* Output those events to a selection of useful destinations.
* Use Logstash's awesome web interface Kibana.
* Scale out your Logstash implementation as your environment grows.
* Quickly and easily extend Logstash to deliver the additional functionality you might need.

By the end of the book, you should have a functional and effective log management solution that you can deploy into your own environment.

Table of Contents

  1. The Logstash Book
    1. Who is this book for?
    2. Credits and Acknowledgments
    3. Technical Reviewers
      1. Jan-Piet Mens
      2. Paul Stack
    4. Technical Illustrator
    5. Author
    6. Conventions in the book
    7. Code and Examples
    8. Colophon
    9. Errata
    10. Trademarks
    11. Version
    12. Copyright
  2. Introduction or Why Should I Bother?
    1. Introducing Logstash and the Elastic Stack
    2. Logstash design and architecture
    3. What's in the book?
    4. Logstash resources
    5. Getting help with Logstash
    6. A mild warning
  3. Getting Started with Logstash
    1. Installing Java
      1. On the Red Hat family
      2. On Debian & Ubuntu
      3. Testing Java is installed
    2. Getting Logstash
    3. Starting Logstash
      1. Our sample configuration file
      2. Running the Logstash agent
      3. Testing the Logstash agent
    4. Summary
  4. Shipping Events
    1. Our Event Lifecycle
    2. Installing Logstash on our central server
      1. Installing Java
      2. Installing Logstash
      3. Elasticsearch for search
      4. Creating a basic central configuration
      5. Running Logstash as a service
    3. An interlude about plugins
    4. The Kibana Console
      1. Installing Kibana
      2. Configuring Kibana
      3. Running Kibana
    5. Installing a Filebeat on our first agent
      1. Installing the Filebeat
      2. Our agent configuration
      3. Installing Filebeat as a service
    6. Sending our first events
      1. Looking at our events in Kibana
    7. Summary
  5. Shipping Events
    1. Using Syslog
      1. A quick introduction to Syslog
      2. Configuring Logstash for Syslog
      3. Configuring Syslog on remote agents
    2. Filebeat
      1. Configure Filebeat on our central server
      2. Installing Filebeat on the remote host
      3. Configuring Filebeat
    3. Other log shippers
      1. Log-Courier
      2. Beaver
      3. Woodchuck
      4. Others
    4. Summary
  6. Filtering Events with Logstash
    1. Apache Logs
      1. Configuring Apache for Custom Logging
      2. Sending Apache events to Logstash
    2. Postfix Logs
      1. Filtering
      2. Collecting Postfix logs
      3. Our first filter
      4. Adding our own filters
      5. Extracting from different events
      6. Setting the timestamp
    3. Filtering Java application logs
      1. Handling blank lines with drop
      2. Handling multi-line log events
      3. Grokking our Java events
    4. Parsing an in-house custom log format
    5. Summary
  7. Structured Application Logging
    1. Application logging primer
      1. Where should I instrument?
      2. Instrument schemas
      3. Time and the observer effect
      4. Logging patterns, or where to put your logging
      5. The utility pattern
      6. The external pattern
    2. Adding our own structured log entries
      1. Adding structured logging to a sample application
      2. Structured logging libraries
    3. Working with your existing logs
    4. Summary
  8. Outputting Events from Logstash
    1. Send email alerts
      1. Updating our multiline filter
      2. Configuring the email output
      3. Email output
    2. Send instant messages
      1. Identifying the event to send
      2. Sending the instant message
    3. Send alerts to Nagios
      1. Nagios check types
      2. Identifying the trigger event
      3. The nagios output
      4. The Nagios external command
      5. The Nagios service
    4. Outputting metrics
      1. Collecting metrics
      2. StatsD
      3. Setting the date correctly
      4. The StatsD output
      5. Sending to a different StatsD server
    5. Summary
  9. Scaling Logstash
    1. Scaling Elasticsearch
      1. Installing additional Elasticsearch hosts
      2. Monitoring our Elasticsearch cluster
      3. Managing Elasticsearch data retention
      4. More Information
    2. Scaling Logstash
      1. Creating a second indexer
      2. Load balancing
    3. Summary
  10. Extending Logstash
    1. Plugin organization
    2. Anatomy of a plugin
    3. Creating our own input plugin
    4. Building our plugin
    5. Adding new plugins
    6. Writing a filter
    7. Writing an output
    8. Summary