Monitoring with Graphite

Preface

Twenty years ago, it was considered bleeding edge to monitor your internet router with SNMP and a Perl script. These tools gave network administrators a new level of insight into their operations and user activity. As the popularity of the internet soared, so too did the number of successful online businesses and services. This, predictably, led to better monitoring and trending software. On the one hand, businesses had to defend their investment against service and network outages; on the other, they needed improved methods for capacity planning as their systems scaled to meet user demand.

Graphite is one of the most powerful monitoring tools available today. Its popularity is driven not only by open source availability, but also by its ease of use, rapid graph prototyping abilities, and friendly rendering API that allows anyone to embed Graphite charts in their applications and websites. The Graphite user community is huge and routinely feeding enhancements back into the core project, such as new statistical and transformative functions, output formats like JSON for client-side chart rendering, and even pluggable storage backends to leverage the growing ecosystem of distributed database systems.

Who Should Read This Book

Monitoring with Graphite is for anyone who wants to learn more about monitoring systems, services, or applications. The book makes few assumptions about your background or experiences other than that you have access to a computer for setting up and interacting with your own Graphite instance.

System administrators and users who wish to gain advanced skills related to scaling Graphite should be comfortable with a Linux or UNIX-style command-line environment. You won’t be dropped in the morass and expected to fend for yourself, but you’ll get more out of this experience if you already know how to navigate a UNIX filesystem and interact with package managers, and what to do when a disk fills up due to an excess of Whisper files.

Everyone should be prepared to have fun learning about monitoring best practices, time-series data storage and retrieval, and constructing charts with a toolkit full of rendering functions and statistical primitives. You will impress your friends and family with the skills you take away from this volume. I almost guarantee it.

Last but certainly not least, understanding how to use and administer Graphite is a valuable skill. Search any online technical job board and you’re sure to find a number of opportunities for individuals who know how to use Graphite effectively or—better yet—install and administer it.

Why I Wrote This Book

In spite of Graphite’s popularity, advanced knowledge about the maintenance and “scaling out” of its software components tends to be concentrated among a handful of experienced users and developers. As a core developer and maintainer of the Graphite project, I’ve seen the effects of this knowledge gap firsthand as businesses and individuals approach me routinely for scaling advice and demonstrations. Unfortunately, I don’t scale nearly as well as the project I love. Therefore, it only makes sense for me to relent and communicate my experiences in this book.

Some of my friends refer to me as “the Graphite Whisperer.” As ridiculous as the title may sound, I admit that I feel a certain pride arranging an elaborate sequence of statistical functions into a graph that accurately represents the source data. I hope that I’m able to imbue you, the reader, with as much Graphite mastery as you can endure. Understandably, not everyone gets as excited talking about time-series line charts as I do. I’ll do my best to keep the forthcoming content instructional yet entertaining, relevant and applicable to the most common use cases.

A Word on Monitoring Today

Three years ago I was writing about how “#monitoringsucks” and we need to reconsider our approach to monitoring architectures. Traditional monitoring systems were monolithic and unwieldy. Nagios already had a largely negative connotation, and there were few alternatives on the horizon.

Either by luck or instinct, I correctly predicted that the monitoring ecosystem would evolve toward a composable model of well-defined services and compatible interfaces. Look at the open source monitoring community and this is precisely what you’ll find today.

In particular, the emergence of NoSQL databases has resulted in enormous momentum and competition among the various time-series databases. Users and businesses have embraced Etsy’s “Measure Anything, Measure Everything” mantra, which has driven the need for larger and faster data stores. A hugely successful conference series arose out of the desire for more collaboration and discussion around our monitoring toolchain. People are more open and transparent about their processes and tooling, and these shared experiences benefit everyone.

In other words, software moves fast and Graphite is no exception. Fortunately, Graphite has reached such a high level of adoption among users and businesses that you’d be hard-pressed to find tools and services that don’t integrate with it directly through its robust APIs or any number of third-party bridge utilities. In many regards, Graphite has become a time-series specification that other software projects leverage for themselves.

Navigating This Book

This book is organized roughly as follows:

• Chapters 1 and 2 provide a basic introduction to monitoring and trending concepts and terminologies. There’s a large vocabulary of terms shared among Graphite users; it helps to speak the same language.

• Chapters 3 and 4 introduce the components that make up Graphite and their respective features and functionality. You will learn how to install Graphite and configure it for a basic setup.

• Chapters 5 and 6 cover the typical user workflow for creating your first line chart. By the end of these chapters you should be very comfortable building complex charts with chained functions and multiple axes, and interacting with the rendering API directly.

• Chapter 7 introduces the native Graphite dashboard and some of the more popular third-party external dashboards. You’ll also get a more thorough coverage of the render API and how to leverage it for client-side chart rendering with JavaScript frameworks like D3.js.

• Chapters 8 and 9 are targeted at system administrators and power users who want to master the art of scaling and troubleshooting high-performance or highly available Graphite clusters.

• Appendix A details the internal statistics reported by Carbon and Graphite-Web. These are invaluable measures for understanding the health of your Graphite cluster.

If you’re like me, you don’t read books from front to back. If you’re really like me, you usually don’t read the Preface at all. However, on the off chance that you will see this in time, here are a few suggested online resources:

• http://graphiteapp.org/

• http://obfuscurity.com/Tags/Graphite

• http://graphite.readthedocs.org/en/latest/

• http://graphite.readthedocs.org/en/0.9.15/

• https://answers.launchpad.net/graphite/

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This icon signifies a tip or suggestion.

Note

This element signifies a general note.

Warning

This icon indicates a warning or caution.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Monitoring with Graphite by Jason Dixon (O’Reilly). Copyright 2017 Jason Dixon, 978-1-491-91643-8.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

O’Reilly Safari

Note

Safari (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals.

Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others.

For more information, please visit http://oreilly.com/safari.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book where we list errata, examples, and any additional information. You can access this page at http://bit.ly/monitoring_with_graphite.

To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

First, I’d like to thank the people who understood the mission in bringing this book to fruition. To the people at O’Reilly Media who reached out to me over the years, encouraging me to move forward with this project—Mike Loukides, Courtney Nash, and Brian Anderson—your persistence and patience is greatly appreciated.

To Chris M. Davis and the team at Orbitz for creating the fantastic Graphite project and releasing it as open source. Your work has been a source of inspiration and provides me with the opportunity to share my love for the project in talks, work, and this book.

To Mike Julian, Pete Cheslock, and Michael Gorsuch for providing valuable feedback and putting up with my pestering and cajoling every half chapter.

To Tom Cameron, who offered his technical expertise for the Anycast discussions.

To Michael Leinartas for his contributions to Graphite, including his work as the project maintainer, and being a mentor to me as I became familiar with the codebase.

Finally, to my wife Christa and my wonderful kids, Alyssa and Nathan, thank you for your unending support and encouragement throughout the writing of this book. You’re the best family a guy could ask for and I hope this stack of paper serves as a small testament of my gratitude for you all being a part of my life.