On this post I just want to let you know that I will speak up at Froscon in Cologne, Germany next week (21st, August).

My speech will cover an architecture what I have used to "keep your mysql backend online, no matter what", what is the title of the presentation too.

This architecture includes MMM as a very important part of it.

Please visit the conference's website and join us there if you can.

www.froscon.de

http://code.google.com/p/check-mysql-all/wiki/check_mmm

You can call this plugin over nrpe. I'm already working on to fork a version which more useful with passive checks.

This plugin was developed by Ryan Lowe (Percona).

So, here is a little workaround that I did and I can't find the same anywhere.

How does it work?

This is an unofficial and unsupported solution. Let's have a master and one or more slave hosts. On your master, you have to create a database lets say slave_auth. Create the same tables in this database as you have in the 'mysql' database and needed for authentication (more details below), and set up a user who can read/write those tables.

When you are done, create a temporary database on your slaves lets say auth_temp with the exact structure of the tables on your master, but with federated engine connected to your master with the user you added previously. Shut down mysql, overwrite the mysql system tables with those you got in auth_temp db (or don't shut down, just overwrite), flush privileges, and there you go, its done. Funny humm?

What are the requirements?

• It needs federated engine supported on your slaves

(no, there is no more requirements)

How to make it work:

Let's move on to some direct working example:

1. Moves on your master(s):

1.1 Creating tables for slaves:

mysql>create database auth_slaves;
mysql>create table auth_slaves.columns_priv like mysql.columns_priv;
mysql>create table auth_slaves.db like mysql.db;
mysql>create table auth_slaves.tables_priv like mysql.tables_priv;
mysql>create table auth_slaves.user like mysql.user;

(If you already have different privileges on your slaves, I would recommend a mysql dump instead)

1.2 Create a user for your slaves:

mysql>grant select,insert,update on auth_slaves.* to 'auth_slave_user'@'192.168.1.%' identified by 'auth_4_sl4v3s';

You are able to set up different kind of auth users, for example if you only wan't on some of your slaves to be just read only, don't grant other rights, but select. Up to you.

All done on  your master..

2. Moves on your slave(s):

2.1 Create a temporary database

mysql>create database dropme_auth;

2.2 Create the tables

Now, I don't paste all the table definitions, but execute show create table on each table printed above, copy the result to a txt or something and modify the command to work with federated like in this example:

mysql#host:master#db:auth_slaves>show create table user\G
*************************** 1. row ***************************
Table: user
Create Table: CREATE TABLE `user` (
......
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Users and global privileges'
1 row in set (0.00 sec)

Now all you have to do is modify the engine like: engine = federated CONNECTION='mysql://auth_slave_user:auth_4_sl4v3s@master.host:3306/auth_slaves/user'

So, just create a table driven by federated engine connected to your masters auth_slaves db with the same table names with the same table structure.

3. Replace the system tables:

In this scenerio you must overwrite the tables in the mysql database folder (for example /var/lib/mysql/mysql) with the tables from your temporary database where you built the federated tables (for example /var/lib/mysql/dropme_auth) like this:  cp /var/lib/mysql/dropme_auth/*.frm /var/lib/mysql/mysql/ .

Don't use '*' in the cp, it's even better if you copy every table one by one.
Take care of the privileges after the copy.

4. Execute a simple select query like select * from mysql.user. If you can see the same what is on your master, you are done. If not, hmm. Thats a kind of a problem

What if connection broken?

Nothing will happen unless you execute flush privileges. Because as we all know, mysql cache the authentication tables and doesn't load the content again until flushing of privileges.

What about flush privileges?

The question you will consider soon if you build this up: Ok wait, how can I modify authentication settings now?! The answer is so simple, just make it on any of your slaves and it will affect the tables stored on your master. When you done, if the documentation correct, execute flush privileges command on your master and all your slaves will reload the modified tables.

Benefits:

The most important is what used to be when you move on to centralized authentication.

• You only need to maintain the privileges on one of your slaves instead of a lots.
• In this case, you don't have to keep the same privileges for your masters and slaves and you don't have to replicate the mysql database.

As I wrote before about staying online, now let me write about how to stay consistent.

We all know, mmm is not like a key of salvation, but its getting close to it . While MySQL doesn't support multi-master-slave environments from it's source code, we will sleep badly wondering on the safety of our precious databases.

But its not just about MMM, a few days ago we ran in to a well known InnoDB "feature". Its about the auto increment counter determination on restart. InnoDB try to count the next auto increment value on MySQL restart what can screw up things in the replication as in your data integrity too (http://dev.mysql.com/doc/refman/5.1/en/innodb-auto-increment-handling.html) what could be risky when you are about to use mmm, because you can restart your masters whenever you just want to.

In this post I'm using maatkit to verify and restore the rows without locking or downtime.

Maatkit:

This is not a success of MMM, but success of maatkit.
I don't want to write about maatkit. Everybody who is able to use MMM were met with maatkit. Great stuff and if you didn't met with it yet, this is the right time. http://www.maatkit.org

Verification between your masters:

This is the hardest part, you CAN'T and/or you SHOULD NOT to avoid manual overview.

I'm using mk-table-sync for this scenerio. As the about say: mk-table-sync finds and fixes data differences between MySQL tables.
This is what we are going to do, but lets consider the possibilities. If we goes trough the help menu, we can see there is a lots of options. Worth going trough of them.

Here is the command I used to execute.

mk-table-sync --chunksize 100 --databases mytestdb --lock 1 --transaction --skipbinlog --print  db02 db01

And here is the explain:

--chunksize 100 : Chunksize means the size of the chunks maatkit compare to each other. Mean rows. In this command maatkit compare the tables by chunks which are contain 100 rows. Its an affordable size, Its wont be faster and as I benchmarked neider become slower, but as you will see later, its start a transaction and lock the table. More about locking and transactions later but to understand why I recommend a small chunksize, you have to know that this means that the locking of the table is it in will be a short lock.

--database mytestdb: Allows you to give a comma-separated list of databases should be taken care of. In this example I use only one database but as I mentioned you can pass a list of databases you want to scan or what you don't want to scan.

--lock 1: As I mentioned before, maatkit provide you two kind of "locking". The first one is a real table lock and the second one is the way of using transactions. With innodb tables using transaction is a good choice, more about later, but with MyIsam tables, I think the best way is lock 1 regarding to what I want to do here, syncing online masters to each other. mk-table-sync provide 3 different ways of locking here: "0=none, 1=per sync cycle, 2=per table, or 3=globally" as far as I know, 0 means no locks, thats not a good way if you are about to sync online masters under traffic. Locking each table could be a wrong decision too, in case of a larger table, and globally is the same under usage. So this is why I just picked lock 1 what will lock while it's take care of my chunk (this is why I lowered the chunk size to 100).

--transaction: This is what i was talking about above. Instead of locking, mk-table-sync able to use transactions . Starting a transaction with a larger chunk size could cause lock-wait-timeout messages as locking can. This is why there is a small chunk size again. You have to listen to your isolation level too. As you know, its could be unsafe under read-committed.

--skipbinlog: This is the most important part of it. In a multi-master environment syncing without this option is like a suicide. With this, mk-table-sync will execute the fixing queries without logging in to binlog. I should not to mention, that logging to binlog on the slave could really mess up your databases as the other master will replicate and execute it.

--print: mk-table-sync provide three different (and some of them able to combine) ways to run. First I would recommend is --print. Print will just print to the stdout (what you can forward to file with 1>/your/file/path) but when you redirect the stdout, don't forget when using with --ask-pass that this message won't show up for your . You can use --execute instead. That will execute the queries immediately. I'm always scared by that a little bit so I rather get the output, go over it and execute what I think I should.

db02/db01: In my example these two hosts are the writable master (at first) and the passive master as a second one.You can use whatever hosts you want. Commands will be executed on the second host.

What to do with your output: When its done and its find differences, you can find where ever your stdout goes. In thet file or on your screen you can see the queries what mk-table-sync would execute to make your passive master consistent. Now its up to you what to do with this information. You can execute each of the or re-run the sync with --execute instead of --print.

Syncing slaves

If you done with your masters, you can sync the slaves on the same way, but you can use the already consistent passive master of your to sync from. Both maatkit and mmm provide some very useful scripts to rebuild slaves and you always got a passive master to do from. With this you are free to rebuild your cluster whenever you want.

So let's see  how can MMM help:

Modifications in your database, backup or maintenance:

For a lot of people these words could mean different things, so what I would like to mention is how MMM can help with that.

Just imagine a typical issue: a new column should be added to a large table.

If you have DRBD, NDB or actually nothing at all, this will make you cry if that table is large enough. This could mean different size for everyone, but lets say 25GB with 1,5 million rows. You can see each row size is not small at all. Adding a column even with a pretty fast, but standalone (not clustered like ndb or different sharding techniques) this could take up 4-5 minutes. Taking your site down for minutes is just not a possibility.

What would you or your developers do to avoid this scenario? A few times, I have seen them create a new table and later execute joins or multiple queries on that. That's really not good for us. (here you can find more answers on performance impacts). So lets say you have a chance to alter the table without downtime. MMM can do it for you.

All you have to do is:

• Define the new column (at least in the beginning) with a possible NULL (but that isn't really good for us too) or with a default value to make it possible that your table will work with the current application and replication together without failed replication or failed queries. What I mean is: let's say you have a table with the following columns: id int(8) auto_increment, sometext varchar(32) NOT NULL and you want to add a column to the end say insertdate datetime default NOW() at the end.

So lets see our possibilities: At first, if the developers would handle it, you would have another table, lets say date_of_join which would contain the same id that you can find in the table above and every time when they want to insert in to the first table, they would write the first and our new table too. And when they want to select something that would mean at least two queries which is really not so good for us. So the first thing you have to take care is: Always made changes what would work perfectly with your current architecture/application.

• Start executing alter command on each server of yours one-by-one (except your current active master) and on those servers which serve as a master (too), use sql_log_bin = 0 for this query! And this is important.  With this option, you can alter each server and finally move the writer role with MMM to another server and alter the last one too.

I already went trough this for a few times. Pretty smooth.

Upgrade:

Another problem with MySQL that is its a single-point of failure. In the most cases (except ndb) you cant  afford to loose your master. So every single modification will cost a lot. You can't upgrade. Now this problem is solved with MMM. All you have to take care is the correct, planned and exact ORDER of upgrading.

What I mean is you have to start upgrading from the bottom. First upgrade your slaves and when they done, upgrade the passive master and finally, move the writer role with MMM by the command set_offline. And set_offline is (could be) very important. Because if you move the writer rule to your passive, but upgraded master, maybe its execute queries in a different way what your lower version MySQL don't understand. Moving roles with set_offline will stop replication and you have time to upgrade safely. But I have to mention that you are risking downtime again but on a different way. Moving to a newer version (5.0>5.1) could be VERY risky without proper testing. Take care about that.

I hope I gave you some good advices.

So, your question is of course: what is MMM for MySQL and why haven't I heard of this before?

Well, MMM for MySQL is a project that strives to provide a HA Solution for MySQL, with automatic failover between multiple masters and slaves. If set up properly, you don't have to worry anymore over downtime for upgrading to a new version of MySQL, or a new version of your application (=> large alter tables, addition of indexes, etc.)

The project has recently reached it's version 2, which is a complete rewrite from scratch. We are now slowly seeing this being put into production, with some success stories making us happy already.

I won't make it too long here, but look forward to hear more from us!