The Metrics Manifesto

CHAPTER 9ABC A/B Testing and Security ROI

Get Better ROI for Security

This chapter exists because of a request from a fellow entrepreneur. He came to me, looking for a method for calculating ROI for his security product. His hope was to use something in the sales process that might motivate action.

What started out as a side project for a friend has evolved into a useful tool.¹ In the first half of this chapter we will focus on the higher-level concepts. The latter half will exclusively focus on the code.

There were two resources that heavily influenced this chapter. The first and most influential is a video/article turned into a tutorial on DataCamp by Rasmus Bååth.² And of course, my first book that I co-authored with Doub Hubbard.³ Both merge impact as dollars with Bayesian concepts and Monte Carlo–based simulations.

Buying Security with Predictive Analytics

I'm proposing a simple approach for making faster and more informed security buying decisions. I call it ABC A/B testing, short for Approximate Bayesian Computation A/B testing⁴ (heavy emphasis on the word approximate).⁵ We have already covered ABC. Also, we are using ABC in conjunction with beta-binomial and gamma-poisson models. So, in one sense, much of this should be conceptually familiar territory.

ABC A/B testing uses subject matter expert (SME) beliefs and small sample data to compare product effectiveness. It specifically forecasts costs based on “proof of concept” (POC) outcomes, financial inputs, and ...