Ceph: Designing and Implementing Scalable Storage Systems

MON settings

In Ceph releases beginning with Jewel, the MONs do a pretty good job at managing their databases. This was not necessarily the case in older releases. MON DB size is a function of both the numbers of OSDs and PGs in your cluster and of how much topology churn is going on. During periods of heavy recovery or node maintenance, the /var/lib/ceph/mon DB can grow to tens of GBs. This can become problematic on systems that provision meager filesystem space, and in some cases, it can impact MON responsiveness and thus overall cluster snappiness.

Even on Jewel and later releases, this setting is recommended; it directs MON daemons to sweep stale entries from their DBs at startup-time:

[mon]mon_compact_on_start = true

Another valuable ...