Splunk 7.x Quick Start Guide

Splunk logs

Splunk logs all of its activities in the $SPLUNK_HOME/var/log/splunk/ and .../var/log/introspection directories. Splunk ingests its own activity logs and stores them in the _internal (for 30 days) and _audit (6 years) indexes; resource usage, storage data, and KV store performance are indexed in _introspection for 14 days.

The _internal index and splunkd sourcetype is your go-to place for most troubleshooting. If you run a search for the sourcetypes and sources (source log files) in this index, you'll get a hint about the spectrum of information stored there; the most common sources are metrics.log and splunkd.log and messages with log_level of ERROR, WARN, or WARNING:

index=_internal | stats count by sourcetype, sourceindex=_internal ...