Splunk Best Practices

Splunk Best Practices

by Travis Marlette
Released September 2016
Publisher(s): Packt Publishing
ISBN: 9781785281396

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Design, implement, and publish custom Splunk applications by following best practices

About This Book
  • This is the most up-to-date guide on the market and will help you finish your tasks faster, easier, and more efficiently.
  • Highly practical guide that addresses common and not-so-common pain points in Splunk.
  • Want to explore shortcuts to perform tasks more efficiently with Splunk? This is the book for you!
Who This Book Is For

This book is for administrators, developers, and search ninjas who have been using Splunk for some time. A comprehensive coverage makes this book great for Splunk veterans and newbies alike.

What You Will Learn
  • Use Splunk effectively to gather, analyze, and report on operational data throughout your environment
  • Expedite your reporting, and be empowered to present data in a meaningful way
  • Create robust searches, reports, and charts using Splunk
  • Modularize your programs for better reusability.
  • Build your own Splunk apps and learn why they are important
  • Learn how to integrate with enterprise systems
  • Summarize data for longer term trending, reporting, and analysis
In Detail

This book will give you an edge over others through insights that will help you in day-to-day instances. When you're working with data from various sources in Splunk and performing analysis on this data, it can be a bit tricky. With this book, you will learn the best practices of working with Splunk.

You'll learn about tools and techniques that will ease your life with Splunk, and will ultimately save you time. In some cases, it will adjust your thinking of what Splunk is, and what it can and cannot do.

To start with, you'll get to know the best practices to get data into Splunk, analyze data, and package apps for distribution. Next, you'll discover the best practices in logging, operations, knowledge management, searching, and reporting. To finish off, we will teach you how to troubleshoot Splunk searches, as well as deployment, testing, and development with Splunk.

Style and approach

If you're stuck or want to find a better way to work with Splunk environment, this book will come handy. This easy-to-follow, insightful book contains step-by-step instructions and examples and scenarios that you will connect to.

Publisher resources

Download Example Code

Table of contents

  1. Splunk Best Practices
    1. Splunk Best Practices
    2. Credits
    3. About the Author
    4. About the Reviewer
    5. www.PacktPub.com
      1. eBooks, discount offers, and more
        1. Why subscribe?
    6. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Downloading the color images of this book 
        3. Errata
        4. Piracy
        5. Questions
    7. 1. Application Logging
      1. Loggers
        1. Anatomy of a log
          1. Log4*
          2. Pantheios
          3. Logging - logging facility for Python
          4. Example of a structured log
      2. Data types
        1. Structured data - best practices
          1. Log events
            1. Common Log Format
            2. Automatic Delimited Value Extraction (IIS/Apache) - best practice
            3. Manual Delimited Value Extraction with REGEX
          2. Step 1 - field mapping - best practice
          3. Step 2 - adding the field map to structure the data (props/transforms)
            1. Use correlation IDs - best practice
            2. Correlation IDs and publication transactions - best practice
            3. Correlation IDs and subscription transactions - best practices
            4. Correlation IDs and database calls - best practices
      3. Unstructured data
        1. Event breaking - best practice
          1. Best practices
        2. Configuration transfer - best practice
      4. Summary
    8. 2. Data Inputs
      1. Agents
        1. Splunk Universal Forwarder
        2. Splunk Heavy Forwarder
        3. Search Head Forwarder
      2. Data inputs
        1. API inputs
        2. Database inputs
        3. Monitoring inputs
        4. Scripted inputs
          1. Custom or not
        5. Modular inputs
        6. Windows inputs
          1. Windows event logs / Perfmon
      3. Deployment server
        1. Know your data
        2. Long delay intervals with lots of data
      4. Summary
    9. 3. Data Scrubbing
      1. Heavy Forwarder management
        1. Managing your Heavy Forwarder
          1. Manual administration
          2. Deployment server
        2. Important configuration files
      2. Even data distribution
        1. Common root cause
        2. Knowledge management
        3. Handling single- versus multi-line events
      3. Manipulating raw data (pre-indexing)
        1. Routing events to separate indexes
        2. Black-holing unwanted events (filtering)
        3. Masking sensitive data
          1. Pre-index data masking
          2. Post-index data masking
        4. Setting a hostname per event
      4. Summary
    10. 4. Knowledge Management
      1. Anatomy of a Splunk search
        1. Root search
        2. Calculation/evaluation
        3. Presentation/action
      2. Best practices with search anatomy
        1. The root search
        2. Calculation/evaluation
        3. Presentation/action
      3. Knowledge objects
        1. Eventtype Creation
          1. Creation through the Splunk UI
          2. Creation through the backend shell
        2. Field extractions
          1. Performing field extractions
            1. Pre-indexing field extractions (index time)
            2. Post-indexing field extractions (search time)
          2. Creating index time field extractions
          3. Creating search time field extractions
            1. Creating field extractions using IFX
            2. Creation through CLI
      4. Summary
    11. 5. Alerting
      1. Setting expectations
        1. Time is literal, not relative
          1. To quickly summarize
        2. Be specific
          1. To quickly summarize
        3. Predictions
          1. To quickly summarize
      2. Anatomy of an alert
        1. Search query results
        2. Alert naming
        3. The schedule
        4. The trigger
        5. The action
        6. Throttling
        7. Permissions
        8. Location of action scripts
          1. Example
      3. Custom commands/automated self-healing
        1. A word of warning
      4. Summary
    12. 6. Searching and Reporting
      1. General practices
        1. Core fields (root search)
          1. _time
          2. Index
          3. Sourcetype
          4. Host
          5. Source
        2. Case sensitivity
        3. Inclusive versus exclusive
      2. Search modes
        1. Fast Mode
        2. Verbose Mode
        3. Smart Mode (default)
      3. Advanced charting
        1. Overlay
          1. Host CPU / MEM utilization
        2. Xyseries
        3. Appending results
          1. timechart
          2. stats
          3. The Week-over-Week-overlay
        4. Day-over-day overlay
          1. SPL to overlay (the hard way)
          2. Timewrap (the easy way)
      4. Summary
    13. 7. Form-Based Dashboards
      1. Dashboards versus reports
        1. Reports
        2. Dashboards
          1. Form-based
          2. Drilldown
          3. Report/data model-based
          4. Search-based
      2. Modules
        1. Data input
        2. Chart
        3. Table
        4. Single value
        5. Map module
      3. Tokens
      4. Building a form-based dashboard
      5. Summary
    14. 8. Search Optimization
      1. Types of dashboard search panel
        1. Raw data search panel
        2. Shared search panel (base search)
        3. Report reference panel
        4. Data model/pivot reference panels
      2. Raw data search
      3. Shared searching using a base search
        1. Creating a base search
        2. Referencing a base search
      4. Report referenced panels
      5. Data model/pivot referenced panels
      6. Special notes
      7. Summary
    15. 9. App Creation and Consolidation
      1. Types of apps
        1. Search apps
        2. Deployment apps
        3. Indexer/cluster apps
        4. Technical add-ons
        5. Supporting add-ons
        6. Premium apps
      2. Consolidating search apps
        1. Creating a custom app
        2. App migrations
          1. Knowledge objects
          2. Dashboard consolidation
          3. Search app navigation
      3. Consolidating indexing/forwarding apps
        1. Forwarding apps
        2. Indexer/cluster apps
      4. Summary
    16. 10. Advanced Data Routing
      1. Splunk architecture
        1. Clustering
          1. Search head clustering
          2. Indexer cluster
            1. Multi-site redundancy
          3. Leveraging load balancers
            1. Failover methods
          4. Putting it all together
      2. Network segments
        1. Production
        2. Standard Integration Testing (SIT)
        3. Quality assurance
        4. Development
        5. The DMZ (App Tier)
      3. The data router
        1. Building roads and maps
        2. Building the UF input/output paths
        3. Building the HF input/output paths
        4. If you build it, they will come
      4. Summary

Product information

  • Title: Splunk Best Practices
  • Author(s): Travis Marlette
  • Release date: September 2016
  • Publisher(s): Packt Publishing
  • ISBN: 9781785281396