book

Splunk: Enterprise Operational Intelligence Delivered

Name: Splunk: Enterprise Operational Intelligence Delivered
ISBN: 9781787288256

by Betsy Page Sigman, Erickson Delgado, Josh Diakun, Paul R Johnson, Derek Mock, Ashish Kumar Tulsiram Yadav

February 2017

Beginner to intermediate

962 pages

21h 26m

English

Packt Publishing

Read now

Unlock full access

Splunk: Enterprise Operational Intelligence Delivered
Table of Contents
Splunk: Enterprise Operational Intelligence Delivered
Credits
Preface
What this learning path covers
What you need for this learning path
Who this learning path is for
Reader feedback
Customer support
Downloading the example codeErrataPiracyQuestions
1. Module 1

1. Splunk in Action
Your Splunk.com accountObtaining a Splunk.com account
Installing Splunk on Windows
Logging in the first timeRun a simple search
Creating a Splunk app
Populating data with Eventgen
Installing an add-on
Controlling Splunk
Configuring Eventgen
Viewing the Destinations app
Creating your first dashboard
Summary
2. Bringing in Data
Splunk and big dataStreaming dataLatency of dataSparseness of data
Splunk data sources
Machine dataWeb logsData filesSocial media dataOther data types
Creating indexes
Buckets
Data inputs
Splunk events and fields
Extracting new fields
Summary
3. Search Processing Language
Anatomy of a searchSearch pipeline
Time modifiers
Filtering search results
Search command - stats
Search command - top/rare
Search commands - chart and timechart
Search command - eval
Search command - rex
Summary
4. Data Models and Pivot
Creating a data modelAdding attributes to objectsCreating child objectsCreating an attribute based on a regular expression
Data model acceleration
The Pivot EditorCreating a chart from a PivotCreating an area chartCreating a pie chart showing destination details by airport codeSingle value with trending sparkline
Rearranging your dashboard
Summary
5. Data Optimization, Reports, Alerts, and Accelerating Searches
Data classification with event types
Data normalization with tags
Data enrichment with lookups
Creating reports
Creating alerts
Search and report acceleration
Scheduling best practices
Summary indexing
Summary
6. Panes of Glass
Creating effective dashboards
Types of dashboard
Gathering information and business requirementsDynamic form-based dashboardCreating a Status Distribution panelCreating the Status Types Over Time panelCreating the Hits vs Response Time panelArranging the dashboardPanel optionsPie chart - status distributionStacked area chart - Status Types Over TimeColumn with line overlay combo chart - Hits vs Response Time
Form inputs
Creating a time range input
Creating a radio input
Creating a dropdown input
Static Real-Time dashboard
Single Value Panels with color rangesCreating panels by cloningSingle Value Panels with trendsReal-time column charts with line overlays
Creating a map called a choropleth
Summary
7. Splunk SDK for JavaScript and D3.js
Introduction to Splunk SDKs
Practical applications of Splunk's SDK
PrerequisitesCreating a CRON JobCreating a saved search
Creating the final dashboard\jobs.js
HTTP serverRendering the chart
Summary
8. HTTP Event Collector
What is the HEC?
How does the HEC work?
How data flows to the HEC?
Logging in dataUsing a token with dataSending out the data requestVerifying the tokenIndexing the dataEnabling the HECGenerating an HEC authentication tokenHow to test the HEC with cURL and PowerShellUsing the HEC with dynamic UI eventsJavaScript logging with the HEC
Summary
9. Best Practices and Advanced Queries
Temporary indexes and oneshot indexing
Searching within an index
Search within a limited time frame
Quick searches via fast mode
Using event sampling
Splunk Universal Forwarders
Advanced queries
SubsearchUsing appendUsing joinUsing eval and ifUsing eval and match with a case function
How to improve logs
Including clear key-value pairsCreating events that are understandable to human readersRemember to use timestamps for all eventsBe sure your identifiers are uniqueLog using text format, not binaryUse formats that developers can use easilyLog what you think might be useful at some pointCreate use categories with meaningInclude the source of the log eventMinimize the number of multi-line events
Summary
2. Module 2
1. Play Time – Getting Data In
Introduction
Indexing files and directories
Getting readyHow to do it…How it works…There's more…Adding a file or directory data input via the CLIAdding a file or directory input via inputs.confOne-time indexing of data files via the Splunk CLIIndexing the Windows event logsSee also
Getting data through network ports
Getting readyHow to do it…How it works…There's more…Adding a network input via the CLIAdding a network input via inputs.confSee also
Using scripted inputs
Getting readyHow to do it…How it works…See also
Using modular inputs
Getting readyHow to do it…How it works…There's more…See also
Using the Universal Forwarder to gather data
Getting readyHow to do it…How it works…There's more…Add the receiving indexer via outputs.conf
Loading the sample data for this book
Getting readyHow to do it…How it works…See also
Defining field extractions
Getting readyHow to do it…How it works…See also
Defining event types and tags
Getting readyHow to do it…How it works…There's more…Adding event types and tags via eventtypes.conf and tags.confSee also
2. Diving into Data – Search and Report
Introduction
Making raw event data readable
Getting readyHow to do it…How it works…There's more…Tabulating every fieldRemoving fields, then tabulating everything else
Finding the most accessed web pages
Getting readyHow to do it…How it works…There's more…Searching for the top 10 accessed web pagesSearching for the most accessed pages by userSee also
Finding the most used web browsers
Getting readyHow to do it…How it works…There's more…Searching for the web browser data for the most used OS typesSee also
Identifying the top-referring websites
Getting readyHow to do it…How it works…There's more…Searching for the top 10 using stats instead of topSee also
Charting web page response codes
Getting readyHow to do it…How it works…There's more…Totaling success and error web page response codesSee also
Displaying web page response time statistics
Getting readyHow to do it…How it works…There's more…Displaying web page response time by actionSee also
Listing the top viewed products
Getting readyHow to do it…How it works…There's more…Searching for the percentage of cart additions from product viewsSee also
Charting the application's functional performance
Getting readyHow to do it…How it works…There's more…See also
Charting the application's memory usage
Getting readyHow to do it…How it works…See also
Counting the total number of database connections
Getting readyHow to do it…How it works…See also
3. Dashboards and Visualizations – Making Data Shine
Introduction
Creating an Operational Intelligence dashboard
Getting readyHow to do it…How it works…There's more…Changing dashboard permissions
Using a pie chart to show the most accessed web pages
Getting readyHow to do it…How it works…There's more…Searching for the top 10 accessed web pagesSee also
Displaying the unique number of visitors
Getting readyHow to do it…How it works…There's more…Coloring the value based on rangesAdding trends and sparklines to the valuesSee also
Using a gauge to display the number of errors
Getting readyHow to do it…How it works…There's more…See also
Charting the number of method requests by type and host
Getting readyHow to do it…How it works…See also
Creating a timechart of method requests, views, and response times
Getting readyHow to do it…How it works…There's more…Method requests, views, and response times by hostSee also
Using a scatter chart to identify discrete requests by size and response time
Getting readyHow to do it…How it works…There's more…Using time series data points with a scatter chartSee also
Creating an area chart of the application's functional statistics
Getting readyHow to do it…How it works…See also
Using a bar chart to show the average amount spent by category
Getting readyHow to do it…How it works…See also
Creating a line chart of item views and purchases over time
Getting readyHow to do it…How it works…See also
4. Building an Operational Intelligence Application
Introduction
Creating an Operational Intelligence application
Getting readyHow to do it…How it works…There's more…Creating an application from another applicationDownloading and installing a Splunk appSee also
Adding dashboards and reports
Getting readyHow to do it…How it works…There's more…Changing permissions of saved reportsSee also
Organizing the dashboards more efficiently
Getting readyHow to do it…How it works…There's more…Modifying the Simple XML directlySee also
Dynamically drilling down on activity reports
Getting readyHow to do it…How it works…There's more…Disabling the drilldown feature in tables and chartsSee also
Creating a form for searching web activity
Getting readyHow to do it…How it works…There's more…Adding a Submit button to your formSee also
Linking web page activity reports to the form
Getting readyHow to do it…How it works…There's more…Adding an overlay to the Sessions Over Time chartSee also
Displaying a geographical map of visitors
Getting readyHow to do it…How it works…There's more…Adding a map panel using Simple XMLMapping different distributions by areaSee also
Scheduling PDF delivery of a dashboard
Getting readyHow to do it…How it works…See also
5. Extending Intelligence – Data Models and Pivoting
Introduction
Creating a data model for web access logs
Getting readyHow to do it…How it works…There's more…Searching data models using the search interfaceSee also
Creating a data model for application logs
Getting readyHow to do it…How it works…See also
Accelerating data models
Getting readyHow to do it…How it works…There's more…Viewing data model and acceleration summary informationAdvanced configuration of data model accelerationSee also
Pivoting total sales transactions
Getting readyHow to do it…How it works…There's more…Pivot searching using the pivot command and search interfaceSee also
Pivoting purchases by geographic location
Getting readyHow to do it…How it works…See also
Pivoting slowest responding web pages
Getting readyHow to do it…How it works…See also
Pivot charting top error codes
Getting readyHow to do it…How it works…See also
6. Diving Deeper – Advanced Searching
Introduction
Calculating the average session time on a website
Getting readyHow to do it…How it works…There's more…Starts with a website visit, ends with a checkoutDefining maximum pause, span, and events in a transactionSee also
Calculating the average execution time for multi-tier web requests
Getting readyHow to do it…How it works…There's more…Calculating the average execution time without using a joinSee also
Displaying the maximum concurrent checkouts
Getting readyHow to do it…How it works…See also
Analyzing the relationship of web requests
Getting readyHow to do it…How it works…There's more…Analyzing relationships of DB actions to memory utilizationSee also
Predicting website traffic volumes
Getting readyHow to do it…How it works…There's more…Predicting the total number of items purchasedPredicting the average response time of function callsSee also
Finding abnormally-sized web requests
Getting readyHow to do it…How it works…There's more…The anomalies commandThe anomalousvalues commandThe anomalydetection commandThe cluster commandSee also
Identifying potential session spoofing
Getting readyHow to do it…How it works…There's more…Creating logic for urgencySee also
7. Enriching Data – Lookups and Workflows
Introduction
Looking up product code descriptions
Getting readyHow to do it…How it works…There's more…Manually adding the lookup to SplunkSee also
Flagging suspect IP addresses
Getting readyHow to do it…How it works…There's more…Modifying an existing saved search to populate a lookup tableSee also
Creating a session state table
Getting readyHow to do it…How it works…There's more…Use the Splunk KV store to maintain the session state tableSee also
Adding hostnames to IP addresses
Getting readyHow to do it…How it works…There's more…Enabling automatic external field lookupsSee also
Searching ARIN for a given IP address
Getting readyHow to do it…How it works…There's more…Limiting workflow actions by event typesSee also
Triggering a Google search for a given error
Getting readyHow to do it…How it works…There's more…Triggering a Google search from the chart drilldown optionsSee also
Creating a ticket for application errors
Getting readyHow to do it…How it works…There's more…Adding a workflow action manually in SplunkSee also
Looking up inventory from an external database
Getting readyHow to do it…How it works…There's more…Use DB Connect for direct external DB lookupsSee also
8. Being Proactive – Creating Alerts
Introduction
Alerting on abnormal web page response times
Getting readyHow to do it…How it works…There's more…Viewing triggered alerts in Splunk's Alert managerSee also
Alerting on errors during checkout in real time
Getting readyHow to do it…How it works…There's more…Building alerts via a configuration fileEditing alert configuration attributes using Advanced editIdentify the real-time searches that are runningSee also
Alerting on abnormal user behavior
Getting readyHow to do it…How it works…There's more…Alerting on abnormal user purchases without checkoutsSee also
Alerting on failure and triggering a scripted response
Getting readyHow to do it…How it works…There's more…See also
Alerting when predicted sales exceed inventory
Getting readyHow to do it…How it works…There's more…Adding an RSS feed notification action to an alertSee also
9. Speeding Up Intelligence – Data Summarization
Introduction
Calculating an hourly count of sessions versus completed transactions
Getting readyHow to do it…How it works…There's more…Generating the summary more frequentlyAvoiding summary index overlaps and gapsSee also
Backfilling the number of purchases by city
Getting readyHow to do it…How it works…There's more…Backfilling a summary index from within a search directlySee also
Displaying the maximum number of concurrent sessions over time
Getting readyHow to do it…How it works…There's more…Viewing the status of an accelerated reportSee also
10. Above and Beyond – Customization, Web Framework, REST API, HTTP Event Collector, and SDKs
Introduction
Customizing the application navigation
Getting readyHow to do it...How it works...There's more…
Adding a force-directed graph of web hits
Getting readyHow to do it...How it works...There's more…Changing the time range on the search managerSee also
Adding a calendar heatmap of product purchases
Getting readyHow to do it...How it works...See also
Adding cell highlighting of average product price
Getting readyHow to do it...How it works...There's more…See also
Remotely querying Splunk's REST API for unique page views
Getting readyHow to do it...How it works...There's more…Authenticating with a session tokenSee also
Creating a Python application to return unique IP addresses
Getting readyHow to do it...How it works...There's more...Paginating the results of your searchSee also
Creating a custom search command to format product names
Getting readyHow to do it...How it works...See also
Collecting data from remote scanning devices
Getting readyHow to do it...How it works...See also
3. Module 3
1. What's New in Splunk 6.3?
Splunk's architectureThe need for parallelizationIndex parallelization
Search parallelization
Pipeline parallelizationThe search schedulerSummary parallelization
Data integrity control
Intelligent job scheduling
The app key-value store
System requirementsUses of the key-value storeComponents of the key-value storeManaging key-value store collections via RESTExamplesReplication of the key-value store
Splunk Enterprise Security
Enabling HTTPS for Splunk WebEnabling HTTPS for the Splunk forwarderSecuring a password with SplunkThe access control list
Authentication using SAML
Summary
2. Developing an Application on Splunk
Splunk apps and technology add-onsWhat is a Splunk app?What is a technology add-on?
Developing a Splunk app
Creating the Splunk application and technology add-onPackaging the applicationInstalling a Splunk app via Splunk WebInstalling the Splunk app manually
Developing a Splunk add-on
Building an add-onInstalling a technology add-on
Managing Splunk apps and add-ons
Splunk apps from the app store
Summary
3. On-boarding Data in Splunk
Deep diving into various input methods and sourcesData sourcesStructured dataWeb and cloud servicesIT operations and network securityDatabasesApplication and operating system dataData input methodsFiles and directoriesNetwork sourcesWindows data
Adding data to Splunk – new interfaces
HTTP Event Collector and configurationHTTP Event CollectorConfiguration via Splunk WebManaging the Event Collector tokenThe JSON API formatAuthenticationMetadataEvent data
Data processing
Event configurationCharacter encodingEvent line breakingTimestamp configurationHost configurationConfiguring a static host value – files and directoriesConfiguring a dynamic host value – files and directoriesConfiguring a host value – events
Managing event segmentation
Improving the data input process
Summary
4. Data Analytics
Data and indexesAccessing dataThe index commandThe eventcount commandThe datamodel commandThe dbinspect commandThe crawl commandManaging dataThe input commandThe delete commandThe clean commandSummary indexing
Search
The search commandThe sendmail commandThe localop command
Subsearch
The append commandThe appendcols commandThe appendpipe commandThe join command
Time
The reltime commandThe localize command
Fields
The eval commandThe xmlkv commandThe spath commandThe makemv commandThe fillnull commandThe filldown commandThe replace command
Results
The fields commandThe searchtxn commandThe head / tail commandThe inputcsv commandThe outputcsv command
Summary
5. Advanced Data Analytics
ReportsThe makecontinuous commandThe addtotals commandThe xyseries command
Geography and location
The iplocation commandThe geostats command
Anomalies
The anomalies commandThe anomalousvalue commandThe cluster commandThe kmeans commandThe outlier commandThe rare command
Predicting and trending
The predict commandThe trendline commandThe x11 command
Correlation
The correlate commandThe associate commandThe diff commandThe contingency command
Machine learning
Summary
6. Visualization
Prerequisites – configuration settings
Tables
Tables – Data overlayTables – SparklineSparkline – Filling and changing colorSparkline – The max value indicatorSparkline – A bar styleTables – An icon set
Single value
Charts
Charts – ColoringChart overlayBubble charts
Drilldown
Dynamic drilldownThe x-axis or y-axis value as a token to a formDynamic drilldown to pass a respective row's specific column valueDynamic drilldown to pass a fieldname of a clicked valueContextual drilldownThe URL field value drilldownSingle value drilldown
Summary
7. Advanced Visualization
Sunburst sequenceWhat is a sunburst sequence?ExampleImplementation
Geospatial visualization
ExampleSyntaxSearch queryImplementation
Punchcard visualization
ExampleSearch queryImplementation
Calendar heatmap visualization
ExampleSearch queryImplementation
The Sankey diagram
ExampleImplementation
Parallel coordinates
ExampleSearch queryImplementation
The force directed graph
ExampleImplementation
Custom chart overlay
ExampleImplementation
Custom decorations
ExampleWhat is the use of such custom decorations?Implementation
Summary
8. Dashboard Customization
Dashboard controlsHTML dashboardDisplay controlsExample and implementationSyntaxForm input controlsExample and implementationPanel controlsExample and implementationEnabling/disabling refresh timeDisabling the manual refresh linkEnabling auto refresh
Multi-search management
ExampleImplementation
Tokens
Eval tokensSyntax of the eval tokenExampleImplementationCustom tokensExampleImplementation
Null search swapper
ExampleImplementation
Switcher
Link switcherExample and implementationButton switcherExample and implementation
Summary
9. Advanced Dashboard Customization
Layout customizationPanel widthExampleImplementationGroupingExampleSingle-value groupingVisualization groupingImplementationPanel toggleExampleImplementationImage overlayExampleWhat is the use of image overlay?Where can image overlay be used?Implementation
Custom look and feel
Example and implementation
The custom alert action
What is alerting?AlertingThe featuresImplementationExample
Summary
10. Tweaking Splunk
Index replicationStandalone environmentDistributed environmentReplicationSearchingFailures
Indexer auto-discovery
ExampleImplementation
Sourcetype manager
Field extractor
Accessing field extractorUsing field extractorExampleRegular expressionDelimiter
Search history
Event pattern detection
Data acceleration
Need for data accelerationData model acceleration
Splunk buckets
Search optimizations
Time rangeSearch modesScope of searchingSearch terms
Splunk health
splunkd logSearch log
Summary
11. Enterprise Integration with Splunk
The Splunk SDK
Installing the Splunk SDK
The Splunk SDK for Python
Importing the Splunk API in PythonConnecting and authenticating the Splunk serverSplunk APIsCreating and deleting an indexCreating inputUploading filesSaved searchesSplunk searches
Splunk with R for analytics
The setupUsing R with Splunk
Splunk with Tableau for visualization
The setupUsing Tableau with Splunk
Summary
12. What Next? Splunk 6.4
Storage optimization
Machine learning
Management and admin
Indexer and search head enhancement
Visualizations
Multi-search management
Enhanced alert actions
Summary
Biblography
Index

Content preview from Splunk: Enterprise Operational Intelligence Delivered

Alerting on failure and triggering a scripted response

By now, you have used every different type of alert available and many of the more common alert actions such as e-mailing. However, one extremely powerful alert action feature we are yet to touch upon is the ability to execute a script when an alert triggers.

In this recipe, you will create a simple real-time per-result alert that triggers when any 503 HTTP web server errors are detected. Upon triggering, the alert will execute a script that will write the details of the event to a local file on the server.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.