As a highly-available database, etcd tolerates minority failures, which means a partial failure where the majority of cluster members are still available and healthy; however, it is a good practice to replace the failed members as soon as possible in order to improve the overall cluster health and minimize the risk of majority failure. It is also highly recommended that you always keep the cluster size greater than two members in production. In order to recover from a minority failure, you need to perform two steps:
- Remove the failed member from the cluster.
- Add a new replacement member. If there is more than one failed member, replace them sequentially.