book

Logs and Telemetry

Name: Logs and Telemetry
Author: Phil Wilkins
ISBN: 9781633437470

by Phil Wilkins

December 2024

Intermediate to advanced

392 pages

12h 17m

English

Manning Publications

Read now

Unlock full access

Logs and Telemetry
copyright
dedication
contents
foreword
preface
acknowledgments
about this book
How this book is organized: A road mapAbout the codeliveBook discussion forum
about the author
about the cover illustration

Part 1 From concepts to running Fluent Bit
1 Introduction to Fluent Bit
1.1 Why is Fluent Bit so important?1.1.1 The value of event distribution1.1.2 Fluent’s place in CNCF1.2 Core Fluent Bit concepts1.2.1 Payload structure1.2.2 Logical architecture1.3 Drivers of Fluent Bit adoption1.3.1 Small footprint, efficiency, and speed1.3.2 Effect of OpenTelemetry and how Fluent Bit relates to It1.3.3 Extending Fluent Bit with C, Go, WebAssembly, and Lua1.3.4 Fluent Bit and stream processing1.3.5 OTel vs. Fluent Bit and Fluentd1.4 Is Fluent Bit a child or a successor of Fluentd?1.5 How we’re going to discover Fluent Bit1.5.1 How much Kubernetes will this book involve?1.5.2 Logging in ActionSummary
2 From zero to “Hello, World”
2.1 Multiple ways to configure Fluent Bit2.1.1 Configuration formats2.1.2 CLI controls2.1.3 Defining a monitoring pipeline using the CLI2.1.4 Fluent Bit prebuilt Docker container2.2 Fluent Bit configuration in two forms2.2.1 Fluent Bit vs. Fluentd configuration comparison2.2.2 Comparing Classic and YAML configuration2.3 Checking configuration with a dry run2.3.1 Exercise: Using - -dry-run to help fix a conf file2.4 Configuring file inclusions2.4.1 Creating dynamic configuration by using inclusions2.4.2 Proving stub inclusions2.5 Environment variables in the configuration2.5.1 Applying environment variables2.5.2 Setting environment variables2.6 Monitoring Fluent Bit’s healthSummary
Part 2 Digging deeper
3 Capturing inputs
3.1 Fluent Bit plugins3.2 OS and device sources3.2.1 Monitoring infrastructure with native executables3.2.2 Tuning monitoring sources3.2.3 Device sources3.3 Using stdout3.3.1 The twelve-factor app and Fluent Bit3.3.2 Running the containerized Log Simulator3.4 File-based log events3.5 Capturing log files3.5.1 Simple file consumption3.5.2 Supporting long-running processes3.5.3 Capturing logs from short-lived applications3.6 Network events and communication between Fluent Bit and Fluentd3.6.1 Network input sources3.6.2 HTTP source3.6.3 Securing communication with SSL/TLS3.6.4 forward source3.6.5 Beyond network ports3.6.6 Internode communication3.6.7 OpenTelemetry3.7 Fluent Bit buffers and chunks3.8 Other sources3.8.1 Container-related plugins3.8.2 Getting data from other processes3.8.3 Observing the observersSummary
4 Getting inputs from containers and Kubernetes
4.1 Architectural context4.2 Fluent Bit capturing Docker events and metrics4.2.1 Docker Events4.2.2 Docker Metrics4.3 Using Podman as a Docker alternative4.4 Other containers4.5 Container logging drivers4.6 Application direct to Fluent Bit4.6.1 OpenTelemetry’s approach to containerized applications4.6.2 Deploying for application direct logging4.6.3 Enriching log events with Pod context by injection4.6.4 Enriching log events with Pod context by filter4.7 Kubernetes and observability4.7.1 Understanding Kubernetes’ position on logging4.7.2 Kubernetes auditing4.7.3 Kubernetes events input4.7.4 The many parts of the Kubernetes ecosystem4.7.5 Container Images4.7.6 Helm charts4.8 Kubernetes operator4.9 Observations on Fluent Bit with Kubernetes4.10 The next frontier of observability with Fluent Bit: eBPFSummary
5 Outputting events
5.1 Architectural context5.2 Common characteristics of Fluent Bit output plugins5.2.1 Output resilience through retries5.2.2 Network controls5.2.3 Worker threads5.2.4 Considerations for using threads5.3 Null output5.3.1 Monitoring with Fluent Bit5.3.2 Configuring null output5.4 Sending log events to the console5.4.1 Formatting outputs5.4.2 Seeing matching at work5.5 Writing to files5.6 Prometheus outputs5.6.1 Prometheus Node Exporter5.6.2 Running our Prometheus configuration5.6.3 Prometheus Fluent Bit Exporter5.6.4 Prometheus remote writer5.7 PostgreSQL output5.8 HTTP output5.9 Forwarding to other Fluent nodes5.10 OpenTelemetry5.11 Hyperscaler native and SaaS observabilitySummary
6 Parsing to extract more meaning
6.1 Architectural context6.2 The goal of parsing6.3 Relationship between parsers and filters6.4 Prebuilt parsers6.5 Parsing an Apache log file6.6 Multiline parsing6.7 Custom parsing6.8 Processing JSON6.8.1 Changing the log event timestamp6.8.2 Diagnosing the unhappy paths6.9 Other types of parsers6.9.1 logfmt6.9.2 LTSV6.10 Decoders6.11 Parsing shortcut for file inputsSummary
7 Filtering and transforming events
7.1 Architectural context7.2 Integrating and enriching with filters7.2.1 Directing and securing logs with GeoIP7.2.2 Using the CheckList filter7.3 Extending and amending with filters7.3.1 Taking a brief look at the nest filter7.3.2 Illustrating the record_modifier filter7.3.3 Illustrating the modify filter7.3.4 Bringing it together7.3.5 Testing filters7.4 Routing and controlling7.4.1 Using the record accessor7.4.2 Rewriting the tag filter example7.4.3 Explicitly including and excluding events with grep7.5 Controlling events7.5.1 throttle7.5.2 log_to_metrics7.5.3 Advanced use of matching7.6 Custom filtering with Lua7.6.1 Background of Lua7.6.2 Implementing a Lua filterSummary
Part 3 Plugins and queries
8 Stream processors for time series calculations and filtering
8.1 Architectural context8.2 Key ideas8.3 Basic query8.4 Stream-processing windows8.4.1 Hopping windows8.4.2 Tumbling windows8.4.3 Setting window durations8.4.4 Deciding which window to use8.5 Selecting multiple attributes and naming8.6 Streams vs. tags8.7 Creating streams8.8 Chaining and creating new streams8.9 Typical use cases for streaming8.9.1 Forecasting8.9.2 Intermittent error tolerance8.9.3 Spurious data values8.9.4 Absence of events8.9.5 Cross-referencing streamsSummary
9 Building processors and Fluent Bit extension options
9.1 Architectural context9.2 Fluent Bit processor: Changing the behavior of existing plugins9.2.1 Processor with Lua for logs9.2.2 Content modifier processor9.2.3 Processor for traces9.2.4 Processor to metrics9.2.5 Processor using SQL9.3 Why we need to extend Fluent Bit9.4 C language9.4.1 Considerations9.4.2 Benefits9.4.3 Drawbacks9.4.4 Tools for the job9.5 Go language9.5.1 Benefits9.5.2 Drawbacks9.6 WebAssembly9.6.1 Benefits9.6.2 Drawbacks9.7 Selecting an extension strategySummary
10 Building plugins
10.1 Architectural context10.2 Why Go?10.3 Plugin objective10.4 Go plugin approach10.4.1 Simplifying our build process10.4.2 Code structure10.4.3 Fluent Bit feature switches10.4.4 The build process for plugins10.5 Understanding the plugin life cycle10.5.1 Input life cycle10.5.2 Output life cycle10.6 Implementing the plugin10.6.1 Setting up MySQL10.6.2 Input plugin10.6.3 Building the code10.6.4 Output plugin10.7 Deploying the custom plugin10.8 Configuring our scenario10.9 Executing the build10.10 Running the custom pluginsSummary
11 Putting Fluent Bit into action: An enterprise use case
11.1 Use case11.2 Deployment needs11.3 Customer dashboards11.3.1 Customer dashboards with Fluent Bit11.3.2 Customer dashboard containers11.3.3 Customer dashboard innovation11.4 Development pipelines11.5 Core services11.6 Central accounting needs11.7 Operational processes11.8 Tool choices11.9 ConclusionSummary
appendix A Installations
A.1 Tool installation overviewA.2 Downloading book resourcesA.3 Prepping LinuxA.4 Fluent BitA.4.1 Linux InstallsA.4.2 macOSA.4.3 Windows installsA.5 DockerA.5.1 WindowsA.5.2 Verifying the installationA.5.3 Linux (including macOS)A.5.4 macOSA.6 KubernetesA.7 LogSimulatorA.7.1 Running as a downloaded imageA.7.2 Running as a locally built Docker imageA.7.3 Java and GroovyA.7.4 Post-LogSimulator useA.8 WireMockA.9 PostmanA.10 PostgresA.11 MySQLA.12 PrometheusA.13 jq
appendix B Useful resources
B.1 Standard plugins based on platformB.1.1 Input pluginsB.1.2 Output pluginsB.1.3 Filter pluginsB.1.4 ProcessorsB.2 Predefined parsersB.2.1 parser.conf fileB.2.2 parsers_ambassador fileB.2.3 parsers_cinder fileB.2.4 parsers_extraB.2.5 parsers_java fileB.2.6 parsers_kafka fileB.2.7 parsers_openstack fileB.3 Multiline parsersB.4 Sources of predefined regular expressionsB.5 Plugins supporting record accessorB.6 Stream processor functionsB.7 Reserved attribute namesB.8 Expressing timeB.9 Expressing data sizesB.10 Fluent Bit formattersB.11 Useful third-party toolsB.12 ObservabilityB.13 Helpful logging practices and resourcesB.14 Additional readingB.15 Web resourcesB.15.1 Formal and de facto standardsB.15.2 Additional web resourcesB.16 Fluent Bit resourcesB.17 LuaB.18 WASM and WASIB.19 C development resourcesB.20 Logging format definitions
appendix C Comparing Fluent Bit and Fluentd
C.1 Technology differencesC.2 Configuration capabilitiesC.3 Inputs and outputsC.3.1 Support for logging frameworksC.3.2 Plugin choiceC.3.3 Secondary/fallback output optionsC.3.4 OpenTelemetryC.3.5 Customization with embedded codeC.4 RoutingC.5 Buffering and internal data structureC.6 Streaming processingC.7 Conclusion
index

Content preview from Logs and Telemetry

7 Filtering and transforming events

This chapter covers

Creating content and attribute-based filtering
Modifying event content to enrich, structure, and route events
Transforming content from logs to metrics
Rewriting tags to help downstream matching
Grepping events to identify events for filtering
Limiting events to prevent backend overloading
Coding filters using Lua

Chapter 6 looked at parsers and the filter that enables them. This chapter looks at the rest of the filters available to us. We will look carefully at filters that can manipulate the tag value, as this is central to how we route logs, metrics, and traces to different output destinations.

7.1 Architectural context

Figure 7.1 shows how filters fit into our logical Fluent ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781633437470Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Logs and Telemetry

by Phil Wilkins