We all knew that we are risking with MMM. Risking, and placing availability as a more important like consistency. But non of us can risk loosing data forever but we show using it, regarding to our conversations think: "I can fix my data later on, but I can’t turn back time and prevent the downtime. (Pascal Hofmann@xaprb.com)".
As I wrote before about staying online, now let me write about how to stay consistent.
We all know, mmm is not like a key of salvation, but its getting close to it . While MySQL doesn't support multi-master-slave environments from it's source code, we will sleep badly wondering on the safety of our precious databases.
But its not just about MMM, a few days ago we ran in to a well known InnoDB "feature". Its about the auto increment counter determination on restart. InnoDB try to count the next auto increment value on MySQL restart what can screw up things in the replication as in your data integrity too (http://dev.mysql.com/doc/refman/5.1/en/innodb-auto-increment-handling.html) what could be risky when you are about to use mmm, because you can restart your masters whenever you just want to.
In this post I'm using maatkit to verify and restore the rows without locking or downtime.