book

Software Telemetry

Name: Software Telemetry
Author: Jamie Riedesel
ISBN: 9781617298141

by Jamie Riedesel

August 2021

Intermediate to advanced

560 pages

18h 5m

English

Manning Publications

Audiobook available

Read now

Unlock full access

inside front cover
Software Telemetry
Copyright
dedication
brief contents
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumOther online resourcesabout the authorabout the cover illustration
1 Introduction
1.1 Defining the styles of telemetry1.1.1 Defining centralized logging1.1.2 Defining metrics1.1.3 Defining distributed tracing1.1.4 Defining SIEM1.2 How telemetry is consumed by different teams1.2.1 Telemetry use by Operations, DevOps, and SRE teams1.2.2 Telemetry use by Security and Compliance teams1.2.3 Telemetry use by Software Engineering and SRE teams1.2.4 Telemetry use by Customer Support teams1.2.5 Telemetry use by business intelligence1.3 Challenges facing telemetry systems1.3.1 Chronic underinvestment harms decision-making1.3.2 Diverse needs resist standardization1.3.3 Information spills and cleaning them up to avoid legal problems1.3.4 Court orders break your assumptions1.4 What you will learnSummary
Part 1. Telemetry system architecture
2 The Emitting stage: Creating and submitting telemetry
2.1 Emitting from production code2.1.1 Emitting telemetry into a log file2.1.2 Emitting telemetry into the system log2.1.3 Emitting telemetry into standard output2.1.4 Formatting telemetry for emissions2.2 Emitting from hardware2.2.1 Explaining SNMP2.2.2 Ingesting telemetry from a Cisco ASA firewall2.3 Emitting from as-a-Service systems2.3.1 Emitting events from SaaS systems2.3.2 Emitting events from IaaS systemsSummary

3 The Shipping stage: Moving and storing telemetry
3.1 Emitter/shipper functions, telemetry from production code3.1.1 Shipping directly into storage3.1.2 Shipping through queues and streams3.1.3 Shipping to SaaS systems3.2 Shipping between SaaS systems3.3 Tipping points in Shipping-stage architectureSummary
4 The Shipping stage: Unifying diverse telemetry formats
4.1 Shipping locally-emitted telemetry4.1.1 Shipping telemetry from a log file4.1.2 Shipping telemetry from the system logger4.1.3 Shipping telemetry from standard output4.2 Unifying diverse emitting formats4.2.1 Encoding telemetry into strings4.2.2 Picking a shipping format4.2.3 Converting Syslog to JSON or other object-encoding formats4.2.4 Designing with cardinality in mindSummary
5 The Presentation stage: Displaying telemetry
5.1 Displaying telemetry in metrics systems5.1.1 Making pretty pictures with telemetry5.1.2 Feeding the graphs with aggregation functions5.1.3 Using aggregations with pdf_pages5.2 Displaying telemetry in centralized logging systems5.2.1 Selecting needed features in a display system for centralized logging5.2.2 Demonstrating centralized logging display5.3 Displaying telemetry in security systems5.4 Displaying telemetry distributed tracing systems5.5 Displaying telemetry in large organizationsSummary
6 Marking up and enriching telemetry
6.1 Markup in the Emitting stage6.2 Markup and enrichment in the Shipping stage6.2.1 Applying context-related telemetry in the Shipping stage6.2.2 Extracting and enriching telemetry in-flight6.2.3 Converting field types during the Shipping stage6.3 Enrichment in the Presentation stage6.4 How telemetry style affects markup and enrichment6.4.1 Markup and enrichment with centralized logging6.4.2 Markup and enrichment with SIEM systems6.4.3 Markup and enrichment with metrics6.4.4 Markup and enrichment with distributed tracing systemsSummary
7 Handling multitenancy
7.1 How multitenant architectures come about7.1.1 Evolving multitenancy in an early-stage startup7.1.2 Evolving multitenancy in a culture of free sharing7.1.3 Evolving multitenancy in a culture of strong separation7.2 Designing multitenant telemetry systems7.2.1 Multitenancy in the Shipping stage7.2.2 Multitenancy in the Presentation stageSummary
Part 2. Use cases revisited: Applying architecture concepts
8 Growing cloud-based startup
8.1 Telemetry at the small-company stage8.1.1 Describing the small company’s telemetry system8.1.2 Analyzing the small company’s telemetry system8.2 Telemetry at the medium-size company stage8.2.1 Describing the medium-size company’s telemetry system8.2.2 Analyzing the medium-size company’s telemetry system8.3 Telemetry at the large-company stage8.3.1 Describing the large company’s telemetry system8.3.2 Analyzing the large company’s telemetry system8.4 Telemetry at the enterprise stage8.5 Looking back at all this growthSummary
9 Nonsoftware business
9.1 Telemetry use in small organizations9.2 Telemetry use in medium-size organizations9.3 Telemetry use in large organizations9.4 Telemetry use in enterprise organizationsSummary
10 Long-established business IT
10.1 Telemetry use in medium-size organizations10.1.1 Telemetry use in office IT10.1.2 Telemetry use in production systems10.2 Telemetry use in large organizations10.3 Telemetry use in global organizations10.3.1 Telemetry use in the Booking and Passenger Manifest department10.3.2 Telemetry use in the Loyalty Programs departmentSummary
Part 3. Techniques for handling telemetry
11 Optimizing for regular expressions at scale
11.1 Anchoring expressions for speed11.2 Building expressions to fail fast11.3 Digging into the Cisco ASA firewall telemetry11.4 Refining emissions to speed regular-expression performance11.5 Additional regular-expression resourcesSummary
12 Standardized logging and event formats
12.1 Implementing structured logging in your code12.2 Implementing standards in your code12.3 Implementing standards in the Shipping stageSummary
13 Using more nonfile emitting techniques
13.1 Designing for socket- and datagram-based emitters13.2 Emitting and shipping for container- and serverless-based code13.2.1 Emitting and shipping from containerd-based code13.2.2 Emitting and shipping from serverless-based code13.3 Encrypting UDP-based telemetrySummary
14 Managing cardinality in telemetry
14.1 Identifying cardinality problems14.1.1 Cardinality in time-series databases14.1.2 Cardinality in logging databases14.2 Lowering the cost of cardinality14.2.1 Use logging standards to contain cardinality14.2.2 Using storage-side methods to tame cardinality14.2.3 Make cardinality someone else’s problemSummary
15 Ensuring telemetry integrity
15.1 Getting telemetry out of reach of an attacker15.1.1 Move telemetry too fast to catch15.1.2 Use ACLs to enforce write-only telemetry15.1.3 Durable telemetry when using SaaS providers15.2 Making telemetry harder to mess with15.2.1 Using access control requirements to defend against attacks15.2.2 Ensuring configuration integrity in your telemetry systems15.2.3 Making changes obviousSummary
16 Redacting and reprocessing telemetry
16.1 Identifying toxic data and where it comes from16.2 Redacting toxic information spills16.3 Reprocessing telemetry to support upgrades16.4 Isolating toxic data to reduce cleanup costsSummary
17 Building policies for telemetry retention and aggregation
17.1 Creating a retention policy17.1.1 Building a policy for centralized logging17.1.2 Building a policy for metrics17.1.3 Building a policy for distributed tracing17.1.4 Building a policy for SIEM systems17.2 Creating an aggregation policy17.3 Using sampling to reduce costs and increase retentionSummary
18 Surviving legal processes
18.1 Defining the eDiscovery process18.2 Dealing with records-retention requests18.2.1 Examining an ELK-based centralized logging system18.2.2 Examining a Sumo Logic-based centralized logging system18.3 Dealing with document-production requests18.3.1 Telemetry in the collection phase18.3.2 Telemetry in the review phase18.3.3 Telemetry in the production phase18.4 Working with lawyersSummary
Appendix A. Telemetry storage systems
A.1 Analyzing ElasticsearchA.1.1 What Elasticsearch is good atA.1.2 What is challenging for ElasticsearchA.2 Analyzing Apache CassandraA.2.1 What Cassandra is good atA.2.2 What is challenging for CassandraA.3 Analyzing Grafana Labs’ LokiA.3.1 What Loki is good atA.3.2 What is challenging for LokiA.4 Analyzing MongoDBA.4.1 What MongoDB is good atA.4.2 What is challenging for MongoDBA.5 Analyzing PrometheusA.5.1 What Prometheus is good atA.5.2 What is challenging for PrometheusA.6 Analyzing InfluxDBA.6.1 What InfluxDB is good atA.6.2 What is challenging for InfluxDBA.7 Analyzing JaegerA.7.1 What Jaeger is good atA.7.2 What is challenging for Jaeger
Appendix B. Recommendation checklist reference
B.1 Telemetry standards, structure, and setting policiesSection 4.2.2: Setting standardized telemetry formatsSection 4.2.4: Designing telemetry formats with cardinality in mindSection 6.4.1: When and where to mark up or enrich telemetry in centralized logging systemsSection 6.4.3: When and where to mark up or enrich telemetry in metrics systemsSection 7.2.1: How parasitic is that parasitic load?Chapter 11: Making regular expressions fastSection 11.4: The project phases for optimizing your logging statements for regular expressionsChapter 12: The benefits of using a structured loggerSection 13.1: In-memory networking and how it eases telemetrySection 14.2.1: Enforcing logging standards through development processSection 17.1.3: Recommendations on setting a tracing retention policySection 17.1.4: Recommendations on setting a SIEM retention policySection 17.3: Considerations when picking a sampling rateB.2 Presentation-stage recommendationsSection 5.1.1: The features of a good metrics systemSection 5.1.1: Considerations for building dashboardsSection 5.2.1: The features of a good centralized logging systemSection 5.3: Extending centralized logging to SIEM workSection 7.2.2: Adding multitenancyB.3 Cardinality managementSection 4.2.4: Designing telemetry formats with cardinality in mindSection 14.1: The symptoms of high cardinalitySection 14.2.1: Healthy low-cardinality context-related telemetrySection 14.2.2: How sharding affects cardinality managementSection 14.2.3: When to make cardinality someone else’s problemB.4 Telemetry safety and effectsChapter 15: The two principles of secure telemetrySection 15.1.1: Moving telemetry too fast to catchSection 15.2.1: The three Linux Mandatory Access Control systemsSection 15.2.1: Places to use ACLs in a telemetry pipelineSection 15.2.3: How encryption and digital signatures support telemetrySection 15.2.3: How encryption and digital signatures make telemetry more fragileSection 16.1: The three types of toxic dataSection 16.1: The penalties for mishandling toxic dataSection 16.3: What drives periodic reprocessingSection 16.4: Why isolating telemetry helps youSection 16.4: Tips to avoid false-positive toxic-data detectionsB.5 Legal topicsSection 18.2: Questions to ask when assessing a telemetry system to handle legal hold ordersSection 18.4: How to work with lawyers
Appendix C. Exercise answers
index
inside back cover

Content preview from Software Telemetry

11 Optimizing for regular expressions at scale

This chapter covers

Knowing where regular expressions are used in telemetry
Optimizing your regular-expression use
Changing your emissions to speed regular expressions

The best way to optimize regular expressions (regexes) at scale is to not use regular expressions. But the telemetry tooling we have available sometimes doesn’t give us the option of avoiding regular expressions, and we have to deal with them anyway. This chapter is about making the regexes you must use perform the best they can. Much like programming, regexes are used for two reasons in a telemetry pipeline:

Control program flow—If a string matches a regex, do something, such as attempt parsing a certain way.
Extract fields ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617298141Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Software Telemetry

by Jamie Riedesel

11 Optimizing for regular expressions at scale

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.