The main tuning options that should be looked at first are the ones that define the size limit to the top tier, when it should flush and when it should evict.
The following two configuration options configure the maximum size of the data to be stored in the top-tier pool:
target_max_bytes target_max_objects
The size is either specified in bytes or the number of objects and does not have to be the same size as the actual pool – but it cannot be larger. The size is also based on the available capacity after replication of the RADOS pool, so for a 3x replica pool, this will be one-third of your raw capacity. If the number of bytes or objects in this pool goes above this limit, I/O will block; therefore, it's important ...
