PostgreSQL 10 High Performance - Third Edition

Sources of bad results and variation

There are many ways to get bad results from pgbench, ones that don't mean anything valid. And there are even more ways to get results that vary so much from one pgbench run to another that they don't mean what you would imagine them to. Normally you should run any pgbench test at least three times and observe the average and variation before presuming that test is valid. Taking the middle of three results is a common median technique for filtering those results into a single one for comparison.

You must let the test run for a while to get useful results. As a general rule of thumb, if you haven't executed a minute's worth of runtime, you're unlikely to have gotten useful results at all. The main exception ...