Monitoring is the key. I use Zabbix, but essentially you want to gather metrics and report on issues.

Once things are set up and working, even with 10s of VMs and applications, it’s quite reliable. The biggest things that catch you out are updates breaking functionality, updates requiring additional manual steps, running out of disk space or expired certificates.

I find I get a spurt of energy to recreate or implement a new system every few months but things just tick over in the meantime.