book

Implementing Splunk: Big Data Reporting and Development for Operational Intelligence

Name: Implementing Splunk: Big Data Reporting and Development for Operational Intelligence
Author: VINCENT BUMGARNER
ISBN: 9781849693288

by VINCENT BUMGARNER

January 2013

Beginner to intermediate

448 pages

9h 16m

English

Packt Publishing

Read now

Unlock full access

Implementing Splunk: Big Data Reporting and Development for Operational Intelligence
Table of Contents
Implementing Splunk: Big Data Reporting and Development for Operational Intelligence
Credits
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers and moreWhy Subscribe?Free Access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for

Conventions
Reader feedback
Customer support
Downloading the example codeErrataPiracyQuestions
1. The Splunk Interface
Logging in to Splunk
The Home app
The top bar
Search app
Data generatorThe Summary viewSearchActionsTimelineThe field pickerFieldsSearch resultsOptionsEvents viewer
Using the time picker
Using the field picker
Using Manager
Summary
2. Understanding Search
Using search terms effectively
Boolean and grouping operators
Clicking to modify your search
Event segmentationField widgetsTime
Using fields to search
Using the field picker
Using wildcards efficiently
Only trailing wildcards are efficientWildcards are tested lastSupplementing wildcards in fields
All about time
How Splunk parses timeHow Splunk stores timeHow Splunk displays timeHow time zones are determined and why it mattersDifferent ways to search against timeSpecifying time in-line in your search_indextime versus _time
Making searches faster
Sharing results with others
Saving searches for reuse
Creating alerts from searches
ScheduleActions
Summary
3. Tables, Charts, and Fields
About the pipe symbol
Using top to show common field values
Controlling the output of top
Using stats to aggregate values
Using chart to turn data
Using timechart to show values over time
timechart options
Working with fields
A regular expression primerCommands that create fieldsevalrexExtracting loglevelUsing the Extract Fields interfaceUsing rex to prototype a fieldUsing the admin interface to build a fieldIndexed fields versus extracted fieldsIndexed field case 1 – rare instances of a common termIndexed field case 2 – splitting wordsIndexed field case 3 – application from sourceIndexed field case 4 – slow requestsIndexed field case 5 – unneeded work
Summary
4. Simple XML Dashboards
The purpose of dashboards
Using wizards to build dashboards
Scheduling the generation of dashboards
Editing the XML directly
UI Examples app
Building forms
Creating a form from a dashboardDriving multiple panels from one formPost-processing search resultsPost-processing limitationsPanel 1Panel 2Panel 3Final XML
Summary
5. Advanced Search Examples
Using subsearches to find loosely related eventsSubsearchSubsearch caveatsNested subsearches
Using transaction
Using transaction to determine the session lengthCalculating the aggregate of transaction statisticsCombining subsearches with transaction
Determining concurrency
Using transaction with concurrencyUsing concurrency to estimate server loadCalculating concurrency with a by clause
Calculating events per slice of time
Using timechartCalculating average requests per minuteCalculating average events per minute, per hour
Rebuilding top
Summary
6. Extending Search
Using tags to simplify search
Using event types to categorize results
Using lookups to enrich data
Defining a lookup table fileDefining a lookup definitionDefining an automatic lookupTroubleshooting lookups
Using macros to reuse logic
Creating a simple macroCreating a macro with argumentsUsing eval to build a macro
Creating workflow actions
Running a new search using values from an eventLinking to an external siteBuilding a workflow action to show field contextBuilding the context workflow actionBuilding the context macro
Using external commands
Extracting values from XMLxmlkvXPathUsing Google to generate results
Summary
7. Working with Apps
Defining an app
Included apps
Installing apps
Installing apps from SplunkbaseUsing Geo Location Lookup ScriptUsing Google MapsInstalling apps from a file
Building your first app
Editing navigation
Customizing the appearance of your app
Customizing the launcher iconUsing custom CSSUsing custom HTMLCustom HTML in a simple dashboardUsing ServerSideInclude in a complex dashboard
Object permissions
How permissions affect navigationHow permissions affect other objectsCorrecting permission problems
App directory structure
Adding your app to Splunkbase
Preparing your appConfirming sharing settingsCleaning up our directoriesPackaging your appUploading your app
Summary
8. Building Advanced Dashboards
Reasons for working with advanced XML
Reasons for not working with advanced XML
Development process
Advanced XML structure
Converting simple XML to advanced XML
Module logic flow
Understanding layoutPanel
Panel placement
Reusing a query
Using intentions
stringreplaceaddterm
Creating a custom drilldown
Building a drilldown to a custom queryBuilding a drilldown to another panelBuilding a drilldown to multiple panels using HiddenPostProcess
Third-party add-ons
Google MapsSideview UtilsThe Sideview Search moduleLinking views with SideviewSideview URLLoaderSideview forms
Summary
9. Summary Indexes and CSV Files
Understanding summary indexesCreating a summary index
When to use a summary index
When to not use a summary index
Populating summary indexes with saved searches
Using summary index events in a query
Using sistats, sitop, and sitimechart
How latency affects summary queries
How and when to backfill summary data
Using fill_summary_index.py to backfillUsing collect to produce custom summary indexes
Reducing summary index size
Using eval and rex to define grouping fieldsUsing a lookup with wildcardsUsing event types to group results
Calculating top for a large time frame
Storing raw events in a summary index
Using CSV files to store transient data
Pre-populating a dropdownCreating a running calculation for a day
Summary
10. Configuring Splunk
Locating Splunk configuration files
The structure of a Splunk configuration file
Configuration merging logic
Merging orderMerging order outside of searchMerging order when searchingConfiguration merging logicConfiguration merging example 1Configuration merging example 2Configuration merging example 3Configuration merging example 4 (search)Using btool
An overview of Splunk .conf files
props.confCommon attributesSearch-time attributesIndex-time attributesParse-time attributesInput time attributesStanza typesPriorities inside a typeAttributes with classinputs.confCommon input attributesFiles as inputsUsing patterns to select rolled logsUsing blacklist and whitelistSelecting files recursivelyFollowing symbolic linksSetting the value of host from sourceIgnoring old data at installationWhen to use crcSaltDestructively indexing filesNetwork inputsNative Windows inputsScripts as inputstransforms.confCreating indexed fieldsCreating a loglevel fieldCreating a session field from sourceCreating a "tag" fieldCreating host categorization fieldsModifying metadata fieldsOverriding hostOverriding sourceOverriding sourcetypeRouting events to a different indexLookup definitionsWildcard lookupsCIDR wildcard lookupsUsing time in lookupsUsing REPORTCreating multivalue fieldsCreating dynamic fieldsChaining transformsDropping eventsfields.confoutputs.confindexes.confauthorize.confsavedsearches.conftimes.confcommands.confweb.conf
User interface resources
Views and navigationAppserver resourcesMetadata
Summary
11. Advanced Deployments
Planning your installation
Splunk instance types
Splunk forwardersSplunk indexerSplunk search
Common data sources
Monitoring logs on serversMonitoring logs on a shared driveConsuming logs in batchReceiving syslog eventsReceiving events directly on the Splunk indexerUsing a native syslog receiverReceiving syslog with a Splunk forwarderConsuming logs from a databaseUsing scripts to gather data
Sizing indexers
Planning redundancy
Indexer load balancingUnderstanding typical outages
Working with multiple indexes
Directory structure of an indexWhen to create more indexesTesting dataDiffering longevityDiffering permissionsUsing more indexes to increase performanceThe lifecycle of a bucketSizing an indexUsing volumes to manage multiple indexes
Deploying the Splunk binary
Deploying from a tar fileDeploying using msiexecAdding a base configurationConfiguring Splunk to launch at boot
Using apps to organize configuration
Separate configurations by purpose
Configuration distribution
Using your own deployment systemUsing Splunk deployment serverStep 1 – Deciding where your deployment server will runStep 2 – Defining your deploymentclient.conf configurationStep 3 – Defining our machine types and locationsStep 4 – Normalizing our configurations into apps appropriatelyStep 5 – Mapping these apps to deployment clients in serverclass.confStep 6 – Restarting the deployment serverStep 7 – Installing deploymentclient.conf
Using LDAP for authentication
Using Single Sign On
Load balancers and Splunk
websplunktcpdeployment server
Multiple search heads
Summary
12. Extending Splunk
Writing a scripted input to gather dataCapturing script output with no dateCapturing script output as a single eventMaking a long-running scripted input
Using Splunk from the command line
Querying Splunk via REST
Writing commands
When not to write a commandWhen to write a commandConfiguring commandsAdding fieldsManipulating dataTransforming dataGenerating data
Writing a scripted lookup to enrich data
Writing an event renderer
Using specific fieldsTable of fields based on field valuePretty print XML
Writing a scripted alert action to process results
Summary
Index

Content preview from Implementing Splunk: Big Data Reporting and Development for Operational Intelligence

Using top to show common field values

A very common question to answer is, "What values are most common?" When looking for errors, you are probably interested in what piece of code has the most errors. The top command provides a very simple way to answer this question. Let's step through a few examples.

First, run a search for errors:

source="impl_splunk_gen" error

Using our sample data, we find events containing the word error, a sampling of which is listed here:

2012-03-03T19:36:23.138-0600 ERROR Don't worry, be happy. [logger=AuthClass, user=mary, ip=1.2.3.4]
2012-03-03T19:36:22.244-0600 ERROR error, ERROR, Error! [logger=LogoutClass, user=mary, ip=3.2.4.5, network=green]
2012-03-03T19:36:21.158-0600 WARN error, ERROR, Error! [logger=LogoutClass, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Big Data Analytics Using Splunk: Deriving Operational Intelligence from Social Media, Machine Data, Existing Data Warehouses, and Other Real-Time Streaming Sources

Peter Zadrozny, Raghu Kodali

Splunk Operational Intelligence Cookbook

Publisher Resources

ISBN: 9781849693288

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Implementing Splunk: Big Data Reporting and Development for Operational Intelligence

by VINCENT BUMGARNER

Using top to show common field values

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.