I think it would be a good idea to have a chaos monkey on a long running environment, such as e.g. the pocops env.
This monkey (for clarity: I expect this to be a script, not a person 馃槃) should periodically cause havoc by restarting processes.
If not too many processes are restarted at once the env should happily keep on chugging along.
Maybe combine it with some fio jobs that are continuously running (with verification enabled).
I think it would be a good idea to have a chaos monkey on a long running environment, such as e.g. the pocops env.
This monkey (for clarity: I expect this to be a script, not a person 馃槃) should periodically cause havoc by restarting processes.
If not too many processes are restarted at once the env should happily keep on chugging along.
Maybe combine it with some fio jobs that are continuously running (with verification enabled).