Security Engineering: A Guide to Building Dependable Distributed Systems, Second Edition

Chapter 26. System Evaluation and Assurance

If it's provably secure, it probably isn't.

—— Lars Knudsen

I think any time you expose vulnerabilities it's a good thing.

—— Attorney General Janet Reno [1068]

Open source is good for security because it prevents you from even trying to violate Kerckhoffs's Law.

—— Eric Raymond

Introduction

I've covered a lot of material in this book, some of it quite difficult. But I've left the hardest parts to the last. These are the questions of assurance — whether the system will work — and evaluation — how you convince other people of this. How do you make a decision to ship the product, and how do you sell the safety case to your insurers?

Assurance fundamentally comes down to the question of whether capable motivated people have beat up on the system enough. But how do you define 'enough'? And how do you define the 'system'? How do you deal with people who protect the wrong thing, because their model of the requirements is out-of-date or plain wrong? And how do you allow for human failures? There are many systems which can be operated just fine by alert experienced professionals, but are unfit for purpose because they're too tricky for ordinary folk to use or are intolerant of error.

But if assurance is hard, evaluation is even harder. It's about how you convince your boss, your clients — and, in extremis, a jury — that the system is indeed fit for purpose; that it does indeed work (or that it did work at some particular time in the past). The reason that ...