Let the software fail on purpose to see if it can recover.
I heard someone telling about Chaos Monkey, even before I've heard of Netflix, and really liked the idea of letting infrastrucrute and applications crash on purpose to see if failure is handle correctly. In the Agile world they say: if its hard you should do it more often. Software failures are hard so let them happen more often.
See also: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html