book

Logging in Action

Name: Logging in Action
Author: Phil Wilkins
ISBN: 9781617298356

by Phil Wilkins

April 2022

Intermediate to advanced

392 pages

12h 2m

English

Manning Publications

Audiobook available

Read now

Unlock full access

inside front cover
Logging in Action
Copyright
contents
front matter
forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organizedAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1. From zero to “Hello World”
1 Introduction to Fluentd
1.1 Elevator pitch for Fluentd1.1.1 What is a log event?1.1.2 Fluentd compared to middleware1.2 Why do we produce logs?1.3 Evolving ideas1.3.1 Four golden signals1.3.2 Three pillars of observability1.4 Log unification1.4.1 Unifying logs vs. log analytics1.5 Software stacks1.5.1 ELK stack1.5.2 Comparing Fluentd and Logstash1.5.3 The relationship between Fluentd and Fluent Bit1.5.4 The relationship between Logstash and Beats1.6 Log routing as a vehicle for security1.7 Log event life cycle1.8 Evolution of Fluentd1.8.1 Treasure Data1.8.2 CNCF1.8.3 Relationship to major cloud vendors PaaS/IaaS1.9 Where can Fluentd and Fluent Bit be used?1.9.1 Platform constraints1.10 Fluentd UI-based editing1.11 Plugins1.12 How Fluentd can be used to make operational tasks easier1.12.1 Actionable log events1.12.2 Making logs more meaningful1.12.3 Polyglot environments1.12.4 Multiple targets1.12.5 Controlling log data costs1.12.6 Logs to metrics1.12.7 Rapid operational consolidationSummary
2 Concepts, architecture, and deployment of Fluentd
2.1 Architecture and core concepts2.1.1 The makeup of a log event2.1.2 Handling time2.1.3 Architecture of Fluentd2.1.4 Fluent configuration execution order2.1.5 Directives2.1.6 Putting timing requirements into action2.2 Deployment of Fluentd2.2.1 Deploying Fluentd for the book’s examples2.2.2 Deployment considerations for Fluentd2.2.3 Fluentd minimum footprint2.2.4 Simple deployment of Ruby2.2.5 Simple deployment of Fluentd2.2.6 Deploying a log generator2.2.7 Installing Postman2.3 Bringing Fluentd to life with “Hello World”2.3.1 “Hello World” scenario2.3.2 “Hello World” configuration2.3.3 Starting Fluentd2.4 “Hello World” with Fluent Bit2.4.1 Starting Fluent Bit2.4.2 Alternate Fluent Bit startup options2.4.3 Fluent Bit configuration file comparison2.4.4 Fluent Bit configuration file in detail2.4.5 Putting the dummy plugin into action2.5 Fluentd deployment with Kubernetes and containers2.5.1 Fluentd DaemonSet2.5.2 Dockerized Fluentd2.6 Using Fluentd UI2.6.1 Installing Fluentd with UISummary
Part 2. Fluentd in depth
3 Using Fluentd to capture log events
3.1 Dry running to check a configuration3.1.1 Putting validating Fluentd configuration into action3.2 Reading log files3.2.1 Putting the adaption of a Fluentd configuration to Fluent Bit into action3.2.2 Rereading and resuming reading of log files3.2.3 Configuration considerations for tracking position3.2.4 Wildcards in the path attribute3.2.5 Expressing time3.2.6 Controlling the impact of wildcards in filenames3.2.7 Replacing wildcards with delimited lists in action3.2.8 Handling log rotation3.3 Self-monitoring3.3.1 HTTP interface check3.4 Imposing structure on log events3.4.1 Standard parsers3.4.2 Third-party parsers3.4.3 Applying a Regex parser to a complex log3.4.4 Putting parser configuration into actionSummary

4 Using Fluentd to output log events
4.1 File output plugin4.1.1 Basic file output4.1.2 Basics of buffering4.1.3 Chunks and Controlling Buffering4.1.4 Retry and backoff4.1.5 Putting configuring buffering size settings into action4.2 Output formatting options4.2.1 out_file4.2.2 json4.2.3 ltsv4.2.4 csv4.2.5 msgpack4.2.6 Applying formatters4.2.7 Putting JSON formatter configuration into action4.3 Sending log events to MongoDB4.3.1 Deploying MongoDB Fluentd plugin4.3.2 Configuring the Mongo output plugin for Fluentd4.3.3 Putting MongoDB connection configuration strings into action4.4 Actionable log events4.4.1 Actionable log events through service invocation4.4.2 Actionable through user interaction tools4.5 Slack to demonstrate the social output4.5.1 Handling tokens and credentials more carefully4.5.2 Externalizing Slack configuration attributes in action4.6 The right tool for the right jobSummary
5 Routing log events
5.1 Reaching multiple outputs by copying5.1.1 Copy by reference or by value5.1.2 Handling errors when copying5.2 Configuration reuse and extension through inclusion5.2.1 Place holding with null output5.2.2 Putting inclusions with a MongoDB output into action5.3 Injecting context into log events5.3.1 Extraction of values5.4 Tag-based routing5.4.1 Using exec output plugin5.4.2 Putting tag naming conventions into action5.4.3 Putting dynamic tagging with extract into action5.5 Tag plugins5.6 Labels: Taking tags to a new level5.6.1 Using a stdout filter to see what is happening5.6.2 Illustrating label and tag routing5.6.3 Connecting pipelines5.6.4 Label sequencing5.6.5 Special labels5.6.6 Putting a common pipeline into actionSummary
6 Filtering and extrapolation
6.1 Application of filters6.1.1 All is well events do not need to be distributed6.1.2 Spotting the needle in a haystack6.1.3 False urgency6.1.4 Releveling6.1.5 Unimplemented housekeeping6.2 Why change log events?6.2.1 Easier to process meaning downstream6.2.2 Add context6.2.3 Record when we have reacted to a log event6.2.4 Data redaction/masking6.3 Applying filters and parsers6.3.1 Filter plugins6.3.2 Applying grep filters6.3.3 Changing log events with the record_transformer plugin6.3.4 Filter parser vs. record transformer6.4 Demonstrating change impact with stdout in action6.4.1 A solution demonstrating change impact with stdout in action6.5 Extract to set key values6.6 Deriving new data values with the record_transformer6.6.1 Putting the incorporation of calculations into a log event transformation into action6.7 Generating simple Fluentd metrics6.7.1 Putting log event counting into actionSummary
Part 3. Beyond the basics
7 Performance and scaling
7.1 Threading and processes to scale with workers7.1.1 Seeing workers in action7.1.2 Worker constraints7.1.3 Controlling output plugin threads7.1.4 Memory management optimization7.2 Scaling and moving workloads7.2.1 Fan-in/log aggregation and consolidation7.2.2 Fan-out and workload distribution7.2.3 High availability7.2.4 Putting a high-availability comparison into action7.3 Fluentd scaling in containers vs. native and virtual environments7.3.1 Kubernetes worker node configuration7.3.2 Per-cluster configuration7.3.3 Container as virtualization7.3.4 Sidecar pattern7.3.5 Options comparison7.4 Securing traffic between Fluentd nodes7.4.1 TLS configuration7.4.2 TLS not just for encryption7.4.3 Certificate and private key storage7.4.4 Security is more than certificates7.5 Credentials management7.5.1 Simple credentials use case7.5.2 Putting certification into actionSummary
8 Driving logs with Docker and Kubernetes
8.1 Fluentd out of the box from Docker Hub8.1.1 Official Docker images8.1.2 Docker log drivers8.1.3 Getting set up for Docker log drivers8.2 Using Docker log drivers8.2.1 Docker drivers via the command line8.2.2 A quick check of network connections8.2.3 Running Docker command line8.2.4 Switching to driver configuration through a configuration file8.3 Kubernetes components logging and the use of Fluentd8.3.1 Kubernetes components and structured logging8.3.2 Kubernetes default log retention and log rotation8.3.3 kubectl with logging8.4 Demonstrating logging with Kubernetes8.4.1 Kubernetes setup8.4.2 Creating logs to capture8.4.3 Understanding how Fluentd DaemonSets are put together8.5 Getting a peek at host logs8.6 Configuring a Kubernetes logging DaemonSet8.6.1 Getting the Fluentd configuration ready to be used8.6.2 Creating our Kubernetes deployment configuration8.6.3 Putting the implementation of a Fluentd for Kubernetes into action8.6.4 Deploying to minikube8.6.5 Tidying up8.7 Kubernetes configuration in action8.7.1 Answer8.8 More Kubernetes monitoring and logging to watch for8.8.1 Node monitoring8.8.2 Termination messagesSummary
9 Creating custom plugins
9.1 Plugin source code9.2 What is Redis, and why build a plugin with the Redis list capability?9.2.1 Redis list over RedisTimeSeries9.3 Illustrating our objective using Redis CLI9.4 Preparing for development9.5 Plugin frameworks9.5.1 Creating the skeleton plugin9.5.2 Plugin life cycle9.6 Implementing the plugin core9.6.1 How configuration attributes work9.6.2 Starting up and shutting down9.6.3 Getting the plugin to work with our Fluentd installation9.6.4 Putting additional configuration validation into action9.6.5 Implementing the Redis output logic9.6.6 Putting the testing of synchronous output into action9.7 Implementing the Redis input plugin9.7.1 Testing input and output plugin execution9.8 Extending output with buffering9.8.1 Improving our scenario by putting maintainability into action9.9 Unit testing9.10 Putting the development of unit tests into action9.10.1 Answer9.11 Package and deployment9.11.1 Documentation9.11.2 Complete metadata aka manifest9.11.3 Building the gem package9.11.4 Rerun without the plugin paths9.12 Extending to be an enterprise-class solutionSummary
Part 4. Good logging practices and frameworks to maximize log value
10 Logging best practices
10.1 Audit events vs. log events10.2 Log levels and severities10.2.1 Trace10.2.2 Debug10.2.3 Info(rmation)10.2.4 Warn(ing)10.2.5 Error10.2.6 Fatal10.2.7 Extending or creating your own log levels10.3 Clear language10.4 Human and machine-readable10.5 Context is key10.5.1 Context: What10.5.2 Context: When10.5.3 Context: Where10.5.4 Context: Why10.5.5 Context: Who10.5.6 A practical checklist for capturing context10.6 Error codes10.6.1 Using standard errors10.6.2 Codes can be for more than errors10.7 Too little logging or too much?10.7.1 What qualifies as sensitive?10.7.2 GDPR is only the start10.8 Log structure and format10.8.1 Putting making log entries ready for application shipping into action10.9 Use frameworks if you can10.10 Development practices10.10.1 Rethrowing exceptions10.10.2 Using standard exceptions and error structures10.10.3 String construction as a reason not to logSummary
11 Logging frameworks
11.1 Value of logging frameworks11.2 Typical structure of a logging framework11.2.1 Logger context11.2.2 Appender11.2.3 Logger11.2.4 Filter11.2.5 Formatter11.2.6 Configuration11.2.7 Logger config11.3 Appender structures11.4 Logging framework landscape11.5 Choosing a framework11.5.1 Putting optimizing application logging into action11.6 Fluentd’s own logging and appenders11.7 Illustrations of an application logging directly to Fluentd11.7.1 Python with logging framework: Using the Fluentd library11.7.2 Invoking Fluentd appender directly11.7.3 Illustration with only Python’s logging11.7.4 Illustration without Python’s logging or Fluentd library11.7.5 Porting the Fluentd calls to another language into action11.7.6 Using generic appenders: The takeawaysSummary
Appendix A. Installation of additional tools and services
A.1 Tool installation overviewA.2 Creating environment variables and amending PATHA.2.1 WindowsA.2.2 LinuxA.3 Java and GroovyA.4 PostmanA.5 ElasticsearchA.5.1 Core Elasticsearch installationA.5.2 Elasticsearch UI installationA.5.3 Fluentd plugin for ElasticsearchA.6 Mongo databaseA.6.1 Mongo DB installationA.6.2 MongoDB configurationA.7 SlackA.8 Setting up Docker and KubernetesA.8.1 Windows PrerequisitesA.8.2 VirtualBox approachA.8.3 Ubuntu image preparation for working with DockerA.8.4 Kubernetes installationA.9 Support Ruby development libraries and toolsA.10 RedisA.10.1 Redis gemA.11 PythonA.12 Vault
Appendix B. Processing times and dates, regular expressions, and other configuration values
B.1 Expressing relative timeB.2 Expressing dates and timesB.3 Expressing sizesB.4 Regular expressionsB.4.1 Escape CodesB.4.2 Repetition/selectionB.4.3 Anchors, groups, and alternatesB.5 Docker tag customization
Appendix C. Plugins summary
C.1 Formatter pluginsC.2 Extract and inject plugin supportC.3 Filter pluginsC.4 Tag manipulation pluginsC.5 Preventing alert stormsC.6 Analytical and metrics pluginsC.7 Plugin Interfaces
Appendix D. Real-world use case
D.1 Fluentd use in a real-world use caseD.2 Setting the sceneD.2.1 The operational challengesD.3 Introducing monitoringD.3.1 Extending monitoringD.3.2 Finessing of monitoringD.4 Cloud dilemmaD.5 SolutionD.6 Conclusion
Appendix E. Useful resources
E.1 Helpful Fluentd resourcesE.2 Helpful Fluentd third-party toolsE.3 Helpful logging practices resourcesE.4 Common log formats and descriptionsE.5 Helpful Ruby resourcesE.6 Docker and KubernetesE.7 ElasticsearchE.8 RedisE.9 SSL/TLS and securityE.10 Environment setupE.11 Logging frameworksE.12 Information portals on legislationE.13 Other handy sources of informationE.14 Supporting Fluentd resourcesE.15 Related reading
index
inside back cover

Content preview from Logging in Action

3 Using Fluentd to capture log events

This chapter covers

Configuring Fluentd for the input of log files
Examining the impact of stopping and starting during file reading by Fluentd
Using parsers to extract more meaning from log events
Self-monitoring and external monitoring of Fluentd using APIs

With the conceptual and architectural foundations set up, and having run a simple configuration, we’re ready to start looking at the capture of log events in more detail. In this chapter, we’re going to focus on capturing log events. But before we do, let’s look at how we can check that our Fluentd configuration is correct.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617298356Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Logging in Action

by Phil Wilkins

3 Using Fluentd to capture log events

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.