book

Prometheus: Up & Running

by Brian Brazil

July 2018

Beginner to intermediate

386 pages

9h 6m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Expanding the KnownConventions Used in This BookUsing Code ExamplesO’Reilly SafariHow to Contact UsAcknowledgments
What Is Monitoring?A Brief and Incomplete History of MonitoringCategories of MonitoringPrometheus ArchitectureClient LibrariesExportersService DiscoveryScrapingStorageDashboardsRecording Rules and AlertsAlert ManagementLong-Term StorageWhat Prometheus Is Not
Running PrometheusUsing the Expression BrowserRunning the Node ExporterAlerting
A Simple ProgramThe CounterCounting ExceptionsCounting SizeThe GaugeUsing GaugesCallbacksThe SummaryThe HistogramBucketsUnit Testing InstrumentationApproaching InstrumentationWhat Should I Instrument?How Much Should I Instrument?What Should I Name My Metrics?
PythonWSGITwistedMultiprocess with GunicornGoJavaHTTPServerServletPushgatewayBridgesParsersExposition FormatMetric TypesLabelsEscapingTimestampscheck metrics
What Are Labels?Instrumentation and Target LabelsInstrumentationMetricMultiple LabelsChildAggregatingLabel PatternsEnumInfoWhen to Use LabelsCardinality
InstallationData SourceDashboards and PanelsAvoiding the Wall of GraphsGraph PanelTime ControlsSinglestat PanelTable PanelTemplate Variables

CPU CollectorFilesystem CollectorDiskstats CollectorNetdev CollectorMeminfo CollectorHwmon CollectorStat CollectorUname CollectorLoadavg CollectorTextfile CollectorUsing the Textfile CollectorTimestamps
Service Discovery MechanismsStaticFileConsulEC2RelabellingChoosing What to ScrapeTarget LabelsHow to Scrapemetric_relabel_configsLabel Clashes and honor_labels
cAdvisorCPUMemoryLabelsKubernetesRunning in KubernetesService Discoverykube-state-metrics
ConsulHAProxyGrok ExporterBlackboxICMPTCPHTTPDNSPrometheus Configuration
Other Monitoring SystemsInfluxDBStatsD
Consul TelemetryCustom CollectorsLabelsGuidelines
Aggregation BasicsGaugeCounterSummaryHistogramSelectorsMatchersInstant VectorRange VectorOffsetHTTP APIqueryquery_range
GroupingwithoutbyOperatorssumcountavgstddev and stdvarmin and maxtopk and bottomkquantilecount_values
Working with ScalarsArithmetic OperatorsComparison OperatorsVector MatchingOne-to-OneMany-to-One and group_leftMany-to-Many and Logical OperatorsOperator Precedence
Changing TypevectorscalarMathabsln, log2, and log10expsqrtceil and floorroundclamp_max and clamp_minTime and Datetimeminute, hour, day_of_week, day_of_month, days_in_month, month, and yeartimestampLabelslabel_replacelabel_joinMissing Series and absentSorting with sort and sort_descHistograms with histogram_quantileCountersrateincreaseirateresetsChanging Gaugeschangesderivpredict_lineardeltaideltaholt_wintersAggregation Over Time
Using Recording RulesWhen to Use Recording RulesReducing CardinalityComposing Range Vector FunctionsRules for APIsHow Not to Use RulesNaming of Recording Rules
Alerting RulesforAlert LabelsAnnotations and TemplatesWhat Are Good Alerts?Configuring AlertmanagersExternal Labels
Notification PipelineConfiguration FileRouting TreeReceiversInhibitionsAlertmanager Web Interface
Planning a RolloutGrowing PrometheusGoing Global with FederationLong-Term StorageRunning PrometheusHardwareConfiguration ManagementNetworks and AuthenticationPlanning for FailureAlertmanager ClusteringMeta- and Cross-MonitoringManaging PerformanceDetecting a ProblemFinding Expensive Metrics and TargetsReducing LoadHorizontal ShardingManaging ChangeGetting Help

Content preview from Prometheus: Up & Running

Chapter 1. What Is Prometheus?

Prometheus is an open source, metrics-based monitoring system. Of course, Prometheus is far from the only one of those out there, so what makes it notable?

Prometheus does one thing and it does it well. It has a simple yet powerful data model and a query language that lets you analyse how your applications and infrastructure are performing. It does not try to solve problems outside of the metrics space, leaving those to other more appropriate tools.

Since its beginnings with no more than a handful of developers working in SoundCloud in 2012, a community and ecosystem has grown around Prometheus. Prometheus is primarily written in Go and licensed under the Apache 2.0 license. There are hundreds of people who have contributed to the project itself, which is not controlled by any one company. It is always hard to tell how many users an open source project has, but I estimate that as of 2018, tens of thousands of organisations are using Prometheus in production. In 2016 the Prometheus project became the second member¹ of the Cloud Native Computing Foundation (CNCF).

For instrumenting your own code, there are client libraries in all the popular languages and runtimes, including Go, Java/JVM, C#/.Net, Python, Ruby, Node.js, Haskell, Erlang, and Rust. Software like Kubernetes and Docker are already instrumented with Prometheus client libraries. For third-party software that exposes metrics in a non-Prometheus format, there are hundreds of integrations available. ...