book

Logging in Action

by Phil Wilkins

April 2022

Intermediate to advanced

392 pages

12h 2m

English

Manning Publications

Read now

Unlock full access

forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organizedAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1.1 Elevator pitch for Fluentd1.1.1 What is a log event?1.1.2 Fluentd compared to middleware1.2 Why do we produce logs?1.3 Evolving ideas1.3.1 Four golden signals1.3.2 Three pillars of observability1.4 Log unification1.4.1 Unifying logs vs. log analytics1.5 Software stacks1.5.1 ELK stack1.5.2 Comparing Fluentd and Logstash1.5.3 The relationship between Fluentd and Fluent Bit1.5.4 The relationship between Logstash and Beats1.6 Log routing as a vehicle for security1.7 Log event life cycle1.8 Evolution of Fluentd1.8.1 Treasure Data1.8.2 CNCF1.8.3 Relationship to major cloud vendors PaaS/IaaS1.9 Where can Fluentd and Fluent Bit be used?1.9.1 Platform constraints1.10 Fluentd UI-based editing1.11 Plugins1.12 How Fluentd can be used to make operational tasks easier1.12.1 Actionable log events1.12.2 Making logs more meaningful1.12.3 Polyglot environments1.12.4 Multiple targets1.12.5 Controlling log data costs1.12.6 Logs to metrics1.12.7 Rapid operational consolidationSummary
2.1 Architecture and core concepts2.1.1 The makeup of a log event2.1.2 Handling time2.1.3 Architecture of Fluentd2.1.4 Fluent configuration execution order2.1.5 Directives2.1.6 Putting timing requirements into action2.2 Deployment of Fluentd2.2.1 Deploying Fluentd for the book’s examples2.2.2 Deployment considerations for Fluentd2.2.3 Fluentd minimum footprint2.2.4 Simple deployment of Ruby2.2.5 Simple deployment of Fluentd2.2.6 Deploying a log generator2.2.7 Installing Postman2.3 Bringing Fluentd to life with “Hello World”2.3.1 “Hello World” scenario2.3.2 “Hello World” configuration2.3.3 Starting Fluentd2.4 “Hello World” with Fluent Bit2.4.1 Starting Fluent Bit2.4.2 Alternate Fluent Bit startup options2.4.3 Fluent Bit configuration file comparison2.4.4 Fluent Bit configuration file in detail2.4.5 Putting the dummy plugin into action2.5 Fluentd deployment with Kubernetes and containers2.5.1 Fluentd DaemonSet2.5.2 Dockerized Fluentd2.6 Using Fluentd UI2.6.1 Installing Fluentd with UISummary
3.1 Dry running to check a configuration3.1.1 Putting validating Fluentd configuration into action3.2 Reading log files3.2.1 Putting the adaption of a Fluentd configuration to Fluent Bit into action3.2.2 Rereading and resuming reading of log files3.2.3 Configuration considerations for tracking position3.2.4 Wildcards in the path attribute3.2.5 Expressing time3.2.6 Controlling the impact of wildcards in filenames3.2.7 Replacing wildcards with delimited lists in action3.2.8 Handling log rotation3.3 Self-monitoring3.3.1 HTTP interface check3.4 Imposing structure on log events3.4.1 Standard parsers3.4.2 Third-party parsers3.4.3 Applying a Regex parser to a complex log3.4.4 Putting parser configuration into actionSummary

4.1 File output plugin4.1.1 Basic file output4.1.2 Basics of buffering4.1.3 Chunks and Controlling Buffering4.1.4 Retry and backoff4.1.5 Putting configuring buffering size settings into action4.2 Output formatting options4.2.1 out_file4.2.2 json4.2.3 ltsv4.2.4 csv4.2.5 msgpack4.2.6 Applying formatters4.2.7 Putting JSON formatter configuration into action4.3 Sending log events to MongoDB4.3.1 Deploying MongoDB Fluentd plugin4.3.2 Configuring the Mongo output plugin for Fluentd4.3.3 Putting MongoDB connection configuration strings into action4.4 Actionable log events4.4.1 Actionable log events through service invocation4.4.2 Actionable through user interaction tools4.5 Slack to demonstrate the social output4.5.1 Handling tokens and credentials more carefully4.5.2 Externalizing Slack configuration attributes in action4.6 The right tool for the right jobSummary
5.1 Reaching multiple outputs by copying5.1.1 Copy by reference or by value5.1.2 Handling errors when copying5.2 Configuration reuse and extension through inclusion5.2.1 Place holding with null output5.2.2 Putting inclusions with a MongoDB output into action5.3 Injecting context into log events5.3.1 Extraction of values5.4 Tag-based routing5.4.1 Using exec output plugin5.4.2 Putting tag naming conventions into action5.4.3 Putting dynamic tagging with extract into action5.5 Tag plugins5.6 Labels: Taking tags to a new level5.6.1 Using a stdout filter to see what is happening5.6.2 Illustrating label and tag routing5.6.3 Connecting pipelines5.6.4 Label sequencing5.6.5 Special labels5.6.6 Putting a common pipeline into actionSummary
6.1 Application of filters6.1.1 All is well events do not need to be distributed6.1.2 Spotting the needle in a haystack6.1.3 False urgency6.1.4 Releveling6.1.5 Unimplemented housekeeping6.2 Why change log events?6.2.1 Easier to process meaning downstream6.2.2 Add context6.2.3 Record when we have reacted to a log event6.2.4 Data redaction/masking6.3 Applying filters and parsers6.3.1 Filter plugins6.3.2 Applying grep filters6.3.3 Changing log events with the record_transformer plugin6.3.4 Filter parser vs. record transformer6.4 Demonstrating change impact with stdout in action6.4.1 A solution demonstrating change impact with stdout in action6.5 Extract to set key values6.6 Deriving new data values with the record_transformer6.6.1 Putting the incorporation of calculations into a log event transformation into action6.7 Generating simple Fluentd metrics6.7.1 Putting log event counting into actionSummary
7.1 Threading and processes to scale with workers7.1.1 Seeing workers in action7.1.2 Worker constraints7.1.3 Controlling output plugin threads7.1.4 Memory management optimization7.2 Scaling and moving workloads7.2.1 Fan-in/log aggregation and consolidation7.2.2 Fan-out and workload distribution7.2.3 High availability7.2.4 Putting a high-availability comparison into action7.3 Fluentd scaling in containers vs. native and virtual environments7.3.1 Kubernetes worker node configuration7.3.2 Per-cluster configuration7.3.3 Container as virtualization7.3.4 Sidecar pattern7.3.5 Options comparison7.4 Securing traffic between Fluentd nodes7.4.1 TLS configuration7.4.2 TLS not just for encryption7.4.3 Certificate and private key storage7.4.4 Security is more than certificates7.5 Credentials management7.5.1 Simple credentials use case7.5.2 Putting certification into actionSummary
8.1 Fluentd out of the box from Docker Hub8.1.1 Official Docker images8.1.2 Docker log drivers8.1.3 Getting set up for Docker log drivers8.2 Using Docker log drivers8.2.1 Docker drivers via the command line8.2.2 A quick check of network connections8.2.3 Running Docker command line8.2.4 Switching to driver configuration through a configuration file8.3 Kubernetes components logging and the use of Fluentd8.3.1 Kubernetes components and structured logging8.3.2 Kubernetes default log retention and log rotation8.3.3 kubectl with logging8.4 Demonstrating logging with Kubernetes8.4.1 Kubernetes setup8.4.2 Creating logs to capture8.4.3 Understanding how Fluentd DaemonSets are put together8.5 Getting a peek at host logs8.6 Configuring a Kubernetes logging DaemonSet8.6.1 Getting the Fluentd configuration ready to be used8.6.2 Creating our Kubernetes deployment configuration8.6.3 Putting the implementation of a Fluentd for Kubernetes into action8.6.4 Deploying to minikube8.6.5 Tidying up8.7 Kubernetes configuration in action8.7.1 Answer8.8 More Kubernetes monitoring and logging to watch for8.8.1 Node monitoring8.8.2 Termination messagesSummary
9.1 Plugin source code9.2 What is Redis, and why build a plugin with the Redis list capability?9.2.1 Redis list over RedisTimeSeries9.3 Illustrating our objective using Redis CLI9.4 Preparing for development9.5 Plugin frameworks9.5.1 Creating the skeleton plugin9.5.2 Plugin life cycle9.6 Implementing the plugin core9.6.1 How configuration attributes work9.6.2 Starting up and shutting down9.6.3 Getting the plugin to work with our Fluentd installation9.6.4 Putting additional configuration validation into action9.6.5 Implementing the Redis output logic9.6.6 Putting the testing of synchronous output into action9.7 Implementing the Redis input plugin9.7.1 Testing input and output plugin execution9.8 Extending output with buffering9.8.1 Improving our scenario by putting maintainability into action9.9 Unit testing9.10 Putting the development of unit tests into action9.10.1 Answer9.11 Package and deployment9.11.1 Documentation9.11.2 Complete metadata aka manifest9.11.3 Building the gem package9.11.4 Rerun without the plugin paths9.12 Extending to be an enterprise-class solutionSummary
10.1 Audit events vs. log events10.2 Log levels and severities10.2.1 Trace10.2.2 Debug10.2.3 Info(rmation)10.2.4 Warn(ing)10.2.5 Error10.2.6 Fatal10.2.7 Extending or creating your own log levels10.3 Clear language10.4 Human and machine-readable10.5 Context is key10.5.1 Context: What10.5.2 Context: When10.5.3 Context: Where10.5.4 Context: Why10.5.5 Context: Who10.5.6 A practical checklist for capturing context10.6 Error codes10.6.1 Using standard errors10.6.2 Codes can be for more than errors10.7 Too little logging or too much?10.7.1 What qualifies as sensitive?10.7.2 GDPR is only the start10.8 Log structure and format10.8.1 Putting making log entries ready for application shipping into action10.9 Use frameworks if you can10.10 Development practices10.10.1 Rethrowing exceptions10.10.2 Using standard exceptions and error structures10.10.3 String construction as a reason not to logSummary
11.1 Value of logging frameworks11.2 Typical structure of a logging framework11.2.1 Logger context11.2.2 Appender11.2.3 Logger11.2.4 Filter11.2.5 Formatter11.2.6 Configuration11.2.7 Logger config11.3 Appender structures11.4 Logging framework landscape11.5 Choosing a framework11.5.1 Putting optimizing application logging into action11.6 Fluentd’s own logging and appenders11.7 Illustrations of an application logging directly to Fluentd11.7.1 Python with logging framework: Using the Fluentd library11.7.2 Invoking Fluentd appender directly11.7.3 Illustration with only Python’s logging11.7.4 Illustration without Python’s logging or Fluentd library11.7.5 Porting the Fluentd calls to another language into action11.7.6 Using generic appenders: The takeawaysSummary
A.1 Tool installation overviewA.2 Creating environment variables and amending PATHA.2.1 WindowsA.2.2 LinuxA.3 Java and GroovyA.4 PostmanA.5 ElasticsearchA.5.1 Core Elasticsearch installationA.5.2 Elasticsearch UI installationA.5.3 Fluentd plugin for ElasticsearchA.6 Mongo databaseA.6.1 Mongo DB installationA.6.2 MongoDB configurationA.7 SlackA.8 Setting up Docker and KubernetesA.8.1 Windows PrerequisitesA.8.2 VirtualBox approachA.8.3 Ubuntu image preparation for working with DockerA.8.4 Kubernetes installationA.9 Support Ruby development libraries and toolsA.10 RedisA.10.1 Redis gemA.11 PythonA.12 Vault
B.1 Expressing relative timeB.2 Expressing dates and timesB.3 Expressing sizesB.4 Regular expressionsB.4.1 Escape CodesB.4.2 Repetition/selectionB.4.3 Anchors, groups, and alternatesB.5 Docker tag customization
C.1 Formatter pluginsC.2 Extract and inject plugin supportC.3 Filter pluginsC.4 Tag manipulation pluginsC.5 Preventing alert stormsC.6 Analytical and metrics pluginsC.7 Plugin Interfaces
D.1 Fluentd use in a real-world use caseD.2 Setting the sceneD.2.1 The operational challengesD.3 Introducing monitoringD.3.1 Extending monitoringD.3.2 Finessing of monitoringD.4 Cloud dilemmaD.5 SolutionD.6 Conclusion
E.1 Helpful Fluentd resourcesE.2 Helpful Fluentd third-party toolsE.3 Helpful logging practices resourcesE.4 Common log formats and descriptionsE.5 Helpful Ruby resourcesE.6 Docker and KubernetesE.7 ElasticsearchE.8 RedisE.9 SSL/TLS and securityE.10 Environment setupE.11 Logging frameworksE.12 Information portals on legislationE.13 Other handy sources of informationE.14 Supporting Fluentd resourcesE.15 Related reading

Content preview from Logging in Action

front matter

foreword

Software is the lifeblood of most industries today and can be a differentiator for those companies that can iterate quickly and find customer value before their competitors. Some of the recent trends that allow large organizations to move fast include the adoption of cloud platforms and microservice architectures. While some of the trends have evolved, one thing has remained constant: when things go wrong, we need to quickly understand where to look to fix the problem. Microservices and ephemeral cloud infrastructure (containers, etc.) exacerbate this problem.

I vividly remember working on a particularly nasty distributed problem for a client a few years back wherein a set of services would communicate with each other to ...