book

Logs and Telemetry

by Phil Wilkins

December 2024

Intermediate to advanced

392 pages

12h 17m

English

Manning Publications

Read now

Unlock full access

How this book is organized: A road mapAbout the codeliveBook discussion forum

1.1 Why is Fluent Bit so important?1.1.1 The value of event distribution1.1.2 Fluent’s place in CNCF1.2 Core Fluent Bit concepts1.2.1 Payload structure1.2.2 Logical architecture1.3 Drivers of Fluent Bit adoption1.3.1 Small footprint, efficiency, and speed1.3.2 Effect of OpenTelemetry and how Fluent Bit relates to It1.3.3 Extending Fluent Bit with C, Go, WebAssembly, and Lua1.3.4 Fluent Bit and stream processing1.3.5 OTel vs. Fluent Bit and Fluentd1.4 Is Fluent Bit a child or a successor of Fluentd?1.5 How we’re going to discover Fluent Bit1.5.1 How much Kubernetes will this book involve?1.5.2 Logging in ActionSummary
2.1 Multiple ways to configure Fluent Bit2.1.1 Configuration formats2.1.2 CLI controls2.1.3 Defining a monitoring pipeline using the CLI2.1.4 Fluent Bit prebuilt Docker container2.2 Fluent Bit configuration in two forms2.2.1 Fluent Bit vs. Fluentd configuration comparison2.2.2 Comparing Classic and YAML configuration2.3 Checking configuration with a dry run2.3.1 Exercise: Using - -dry-run to help fix a conf file2.4 Configuring file inclusions2.4.1 Creating dynamic configuration by using inclusions2.4.2 Proving stub inclusions2.5 Environment variables in the configuration2.5.1 Applying environment variables2.5.2 Setting environment variables2.6 Monitoring Fluent Bit’s healthSummary
3.1 Fluent Bit plugins3.2 OS and device sources3.2.1 Monitoring infrastructure with native executables3.2.2 Tuning monitoring sources3.2.3 Device sources3.3 Using stdout3.3.1 The twelve-factor app and Fluent Bit3.3.2 Running the containerized Log Simulator3.4 File-based log events3.5 Capturing log files3.5.1 Simple file consumption3.5.2 Supporting long-running processes3.5.3 Capturing logs from short-lived applications3.6 Network events and communication between Fluent Bit and Fluentd3.6.1 Network input sources3.6.2 HTTP source3.6.3 Securing communication with SSL/TLS3.6.4 forward source3.6.5 Beyond network ports3.6.6 Internode communication3.6.7 OpenTelemetry3.7 Fluent Bit buffers and chunks3.8 Other sources3.8.1 Container-related plugins3.8.2 Getting data from other processes3.8.3 Observing the observersSummary
4.1 Architectural context4.2 Fluent Bit capturing Docker events and metrics4.2.1 Docker Events4.2.2 Docker Metrics4.3 Using Podman as a Docker alternative4.4 Other containers4.5 Container logging drivers4.6 Application direct to Fluent Bit4.6.1 OpenTelemetry’s approach to containerized applications4.6.2 Deploying for application direct logging4.6.3 Enriching log events with Pod context by injection4.6.4 Enriching log events with Pod context by filter4.7 Kubernetes and observability4.7.1 Understanding Kubernetes’ position on logging4.7.2 Kubernetes auditing4.7.3 Kubernetes events input4.7.4 The many parts of the Kubernetes ecosystem4.7.5 Container Images4.7.6 Helm charts4.8 Kubernetes operator4.9 Observations on Fluent Bit with Kubernetes4.10 The next frontier of observability with Fluent Bit: eBPFSummary
5.1 Architectural context5.2 Common characteristics of Fluent Bit output plugins5.2.1 Output resilience through retries5.2.2 Network controls5.2.3 Worker threads5.2.4 Considerations for using threads5.3 Null output5.3.1 Monitoring with Fluent Bit5.3.2 Configuring null output5.4 Sending log events to the console5.4.1 Formatting outputs5.4.2 Seeing matching at work5.5 Writing to files5.6 Prometheus outputs5.6.1 Prometheus Node Exporter5.6.2 Running our Prometheus configuration5.6.3 Prometheus Fluent Bit Exporter5.6.4 Prometheus remote writer5.7 PostgreSQL output5.8 HTTP output5.9 Forwarding to other Fluent nodes5.10 OpenTelemetry5.11 Hyperscaler native and SaaS observabilitySummary
6.1 Architectural context6.2 The goal of parsing6.3 Relationship between parsers and filters6.4 Prebuilt parsers6.5 Parsing an Apache log file6.6 Multiline parsing6.7 Custom parsing6.8 Processing JSON6.8.1 Changing the log event timestamp6.8.2 Diagnosing the unhappy paths6.9 Other types of parsers6.9.1 logfmt6.9.2 LTSV6.10 Decoders6.11 Parsing shortcut for file inputsSummary
7.1 Architectural context7.2 Integrating and enriching with filters7.2.1 Directing and securing logs with GeoIP7.2.2 Using the CheckList filter7.3 Extending and amending with filters7.3.1 Taking a brief look at the nest filter7.3.2 Illustrating the record_modifier filter7.3.3 Illustrating the modify filter7.3.4 Bringing it together7.3.5 Testing filters7.4 Routing and controlling7.4.1 Using the record accessor7.4.2 Rewriting the tag filter example7.4.3 Explicitly including and excluding events with grep7.5 Controlling events7.5.1 throttle7.5.2 log_to_metrics7.5.3 Advanced use of matching7.6 Custom filtering with Lua7.6.1 Background of Lua7.6.2 Implementing a Lua filterSummary
8.1 Architectural context8.2 Key ideas8.3 Basic query8.4 Stream-processing windows8.4.1 Hopping windows8.4.2 Tumbling windows8.4.3 Setting window durations8.4.4 Deciding which window to use8.5 Selecting multiple attributes and naming8.6 Streams vs. tags8.7 Creating streams8.8 Chaining and creating new streams8.9 Typical use cases for streaming8.9.1 Forecasting8.9.2 Intermittent error tolerance8.9.3 Spurious data values8.9.4 Absence of events8.9.5 Cross-referencing streamsSummary
9.1 Architectural context9.2 Fluent Bit processor: Changing the behavior of existing plugins9.2.1 Processor with Lua for logs9.2.2 Content modifier processor9.2.3 Processor for traces9.2.4 Processor to metrics9.2.5 Processor using SQL9.3 Why we need to extend Fluent Bit9.4 C language9.4.1 Considerations9.4.2 Benefits9.4.3 Drawbacks9.4.4 Tools for the job9.5 Go language9.5.1 Benefits9.5.2 Drawbacks9.6 WebAssembly9.6.1 Benefits9.6.2 Drawbacks9.7 Selecting an extension strategySummary
10.1 Architectural context10.2 Why Go?10.3 Plugin objective10.4 Go plugin approach10.4.1 Simplifying our build process10.4.2 Code structure10.4.3 Fluent Bit feature switches10.4.4 The build process for plugins10.5 Understanding the plugin life cycle10.5.1 Input life cycle10.5.2 Output life cycle10.6 Implementing the plugin10.6.1 Setting up MySQL10.6.2 Input plugin10.6.3 Building the code10.6.4 Output plugin10.7 Deploying the custom plugin10.8 Configuring our scenario10.9 Executing the build10.10 Running the custom pluginsSummary
11.1 Use case11.2 Deployment needs11.3 Customer dashboards11.3.1 Customer dashboards with Fluent Bit11.3.2 Customer dashboard containers11.3.3 Customer dashboard innovation11.4 Development pipelines11.5 Core services11.6 Central accounting needs11.7 Operational processes11.8 Tool choices11.9 ConclusionSummary
A.1 Tool installation overviewA.2 Downloading book resourcesA.3 Prepping LinuxA.4 Fluent BitA.4.1 Linux InstallsA.4.2 macOSA.4.3 Windows installsA.5 DockerA.5.1 WindowsA.5.2 Verifying the installationA.5.3 Linux (including macOS)A.5.4 macOSA.6 KubernetesA.7 LogSimulatorA.7.1 Running as a downloaded imageA.7.2 Running as a locally built Docker imageA.7.3 Java and GroovyA.7.4 Post-LogSimulator useA.8 WireMockA.9 PostmanA.10 PostgresA.11 MySQLA.12 PrometheusA.13 jq
B.1 Standard plugins based on platformB.1.1 Input pluginsB.1.2 Output pluginsB.1.3 Filter pluginsB.1.4 ProcessorsB.2 Predefined parsersB.2.1 parser.conf fileB.2.2 parsers_ambassador fileB.2.3 parsers_cinder fileB.2.4 parsers_extraB.2.5 parsers_java fileB.2.6 parsers_kafka fileB.2.7 parsers_openstack fileB.3 Multiline parsersB.4 Sources of predefined regular expressionsB.5 Plugins supporting record accessorB.6 Stream processor functionsB.7 Reserved attribute namesB.8 Expressing timeB.9 Expressing data sizesB.10 Fluent Bit formattersB.11 Useful third-party toolsB.12 ObservabilityB.13 Helpful logging practices and resourcesB.14 Additional readingB.15 Web resourcesB.15.1 Formal and de facto standardsB.15.2 Additional web resourcesB.16 Fluent Bit resourcesB.17 LuaB.18 WASM and WASIB.19 C development resourcesB.20 Logging format definitions
C.1 Technology differencesC.2 Configuration capabilitiesC.3 Inputs and outputsC.3.1 Support for logging frameworksC.3.2 Plugin choiceC.3.3 Secondary/fallback output optionsC.3.4 OpenTelemetryC.3.5 Customization with embedded codeC.4 RoutingC.5 Buffering and internal data structureC.6 Streaming processingC.7 Conclusion

Overview

Build cloud native observability pipelines with minimal footprints and high-performance throughput—all with Fluent Bit, Kubernetes, and your favorite visualization and analytics tools.

Logs and Telemetry is an all-practical guide to monitoring both cloud-native and traditional environments with the Fluent Bit observability tool. It takes you from the basics of collecting app logs, all the way to filtering, routing, enriching and transforming logs, metrics, and traces.

Inside Logs and Telemetry you’ll learn how to:

Deploy Fluent Bit for telemetry (log, metric, and trace) collection
Configure pipelines to filter, route, and transform data
Integrate Fluent Bit with containers and Kubernetes
Configure Fluent Bit to work with OpenTelemetry, Prometheus, and other open source tech
Monitor applications at scale with minimal footprint
Address challenges in Kubernetes-based ecosystems using Fluent Bit
Utilize Fluent Bit for real-time event analytics to derive new metrics and insights
Develop custom filters, inputs, and outputs for unique or reusable use cases

Logs and Telemetry draws on both the input and support of key committers and founders of Fluent Bit, and author Phil Wilkins’ years of experience in DevOps. Inside, you’ll see how you can integrate Fluent Bit with Prometheus, OpenTelemetry, FluentD deployments, and more. Learn how Fluent Bit can not only meet all the demands of cloud-native use cases, but also more traditional deployments as well.

About the Technology
Fluent Bit is a super-fast lightweight observability tool that’s perfect for Kubernetes and containers, as well as traditional IT environments. Fluent Bit makes it a snap to extract meaning from the logs, traces, and other performance metrics generated by your applications and infrastructure. It’s also a great way to route telemetry to analysis tools like Prometheus and Grafana.

About the Book
Logs and Telemetry shows you how to turn systems data into actionable insights using Fluent Bit. You’ll start by learning the pre-built plugins for common use cases and progress to integration with powerful tools like OpenTelemetry and real-time analytical event processing. You’ll use plugins to configure routing, filtering and processing, automate your observability with Lua scripts, and configure Fluent Bit to meet the demands of highly scalable environments.

What's Inside

Deploy Fluent Bit for telemetry collection
Configure pipelines to filter, route, and transform data
Integrate Fluent Bit with containers and Kubernetes
Monitor applications at scale

About the Reader
For developers, DevOps engineers, and SREs working with observability.

About the Author
Phil Wilkins has spent over 25 years in the software industry from multinationals to software startups. He is the author of Logging in Action.

The technical editor on this book was Karthik Gaekwad.

Quotes
An essential, targeted book for anyone working on cloud native applications.
- Jay Vyas, Tesla Motors

Extensive real-life examples and comprehensive coverage! It’s a great resource for architects, developers, and SREs.
- Sambasiva Andaluri, IBM

A must read for anyone managing a critical IT-system. You will truly understand what’s going on in your applications and infrastructure.
- Hassan Ajan, Gain Momentum

A detailed dive into building observability and monitoring.
- Jamie Riedesel, author of Software Telemetry