MySQL Fabric setup document

MySQL Fabric is a framework for managing groups of MySQL Servers and using those servers to provide services. It is designed to be extensible so that over time many different services can be added.
In the current version the services provided are High Availability (built on top of MySQL Replication) and scale-out (by sharding the data).

MySQL Fabric is implemented as a MySQL Fabric node/process (which performs management functions) and Fabric-aware connectors that are able to route queries and transactions directly to the most appropriate MySQL Server.
The MySQL Fabric node stores state and routing information in its State Store (which is a MySQL database).

The MySQL Servers that are being managed by MySQL Fabric continue to use InnoDB (and in the future NDB/MySQL Cluster may also be supported).
MySQL Fabric manages one or more HA-Groups where each HA-Group contains one or more MySQL Servers. For High Availability, a HA Group contains a Primary and one or more Secondary MySQL Servers.
The Primary is currently a MySQL Replication master that replicates to each of the secondaries (MySQL Replication slaves) within the group.

By default, the Fabric-aware connectors route writes to the Primary and load balance reads across the available secondaries.
Should the Primary fail, MySQL Fabric will promote one of the Secondaries to be the new Primary (automatically promoting the MySQL Server to be the replication Master and updating the routing performed by the Fabric-aware connectors).

The MySQL Fabric node has a built-in monitoring function that checks on the status of the master. In addition, the Fabric-aware connectors report to MySQL Fabric when the Primary becomes unavailable to them.
The administrator can configure how many problems need to be reported (and over what time period) before the failover is initiated
MySQL Fabric is available for use under the GPL v2 Open Source license or it can be commercially licensed as part of MySQL Enterprise Edition or MySQL Cluster Carrier Grade Edition.
For development, the MySQL Fabric node and all of the managed MySQL Servers can be hosted on a single machine.
For deployment, the minimal HA configuration would need 3 or more machines:

2 to host MySQL Servers
1 to host the MySQL Fabric process (that machine could also be running application code)
More MySQL Fabric FAQs are at : https://www.mysql.com/products/enterprise/fabric/faq.html#2

Building the MySQL Fabric Framework :

Pre-requisites :

Python Connector :http://dev.mysql.com/downloads/connector/python/
MySQL 5.6+ :http://dev.mysql.com/downloads/fabric/
fabric1 dbversity-fabric-1 (xx.xx.34.35) – fabric
mysql1 dbversity-mysql-2 (xx.xx.87.36) – mysql master
mysql2 dbversity-mysql-3 (xx.xx.87.37) – mysql slave
mysql3 dbversity-mysql-4 (xx.x.45.124) – mysql slave

[ root @ dbversity-fabric-1: /tmp ] rpm -ivh mysql-connector-python-2.0.4-1.el6.noarch.rpm
Preparing… ########################################### [100%]
package mysql-connector-python-2.0.4-1.el6.noarch is already installed
[ root @ dbversity-fabric-1: /tmp ]
[ root @ dbversity-fabric-1: /tmp ] yum install MySQL-server-5.6.25-1.el6.x86_64.rpm
Setting up Install Process
Examining MySQL-server-5.6.25-1.el6.x86_64.rpm: MySQL-server-5.6.25-1.el6.x86_64
Marking MySQL-server-5.6.25-1.el6.x86_64.rpm to be installed
Resolving Dependencies
–> Running transaction check
—> Package MySQL-server.x86_64 0:5.6.25-1.el6 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================================================================================================
Installing:
MySQL-server x86_64 5.6.25-1.el6 /MySQL-server-5.6.25-1.el6.x86_64 238 M

Transaction Summary
============================================================================================================================================================================================================================================
Install 1 Package(s)

Total size: 238 M
Installed size: 238 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
** Found 1 pre-existing rpmdb problem(s), ‘yum check’ output follows:
crypto-utils-2.4.1-24.2.el6.x86_64 has missing requires of perl(Newt)
Installing : MySQL-server-5.6.25-1.el6.x86_64 1/1
Verifying : MySQL-server-5.6.25-1.el6.x86_64 1/1

Installed:
MySQL-server.x86_64 0:5.6.25-1.el6

Complete!
[ root @ dbversity-fabric-1: /tmp ]

MySQL Fabric needs access to a MySQL Database to store state and routing information for the farm of servers.

[ root @ dbversity-fabric-1: /tmp ] cat /etc/my.cnf
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock

[mysqld]
datadir=/var/lib/mysql
binlog-format=ROW
log-slave-updates=true
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
port=3306
report-host=10.116.34.35
report-port=3306
server-id=1
log-bin=vm-5cc5-bad6.log

[mysqld_safe]
log-error=/var/lib/mysql/mysqld.log
pid-file=/var/lib/mysql/mysqld.pid

[ root @ dbversity-fabric-1: /tmp ]

[ root @ dbversity-fabric-1: /var/lib/mysql ] mysql_install_db –defaults-file=/etc/my.cnf –datadir=/var/lib/mysql &
[ root @ dbversity-fabric-1: /tmp ] service mysql restart
Shutting down MySQL…. SUCCESS!
Starting MySQL. SUCCESS!
[ root @ dbversity-fabric-1: /tmp ]
[ root @ dbversity-fabric-1: /tmp ]
[ root @ dbversity-fabric-1: /tmp ] cd /var/lib/mysql
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ] ll -lhtr
total 175M
drwx—— 2 mysql mysql 4.0K Jun 29 04:13 test
-rw-rw—- 1 mysql mysql 48M Jun 29 04:13 ib_logfile1
-rw-rw—- 1 mysql mysql 64K Jun 29 04:13 vm-5cc5-bad6.000001
drwx—— 2 mysql mysql 4.0K Jun 29 04:13 mysql
-rw-rw—- 1 mysql mysql 1.2M Jun 29 04:13 vm-5cc5-bad6.000002
-rw-rw—- 1 mysql mysql 63K Jun 29 04:13 vm-5cc5-bad6.000003
drwx—— 2 mysql mysql 4.0K Jun 29 04:13 performance_schema
-rw-rw—- 1 mysql mysql 1.2M Jun 29 04:13 vm-5cc5-bad6.000004
-rw-rw—- 1 mysql mysql 56 Jun 29 04:14 auto.cnf
-rw-rw—- 1 mysql mysql 201 Jun 29 04:21 vm-5cc5-bad6.000005
-rw-r—– 1 mysql root 877 Jun 29 04:21 mysqld.log-20150629.gz
drwx—— 2 mysql mysql 4.0K Jun 29 04:22 fabric
-rw-rw—- 1 mysql mysql 19K Jun 29 04:33 vm-5cc5-bad6.000006
-rw-rw—- 1 mysql mysql 5 Jun 29 04:33 vm-5cc5-bad6.pid
-rw-rw—- 1 mysql mysql 154 Jun 29 04:33 vm-5cc5-bad6.index
-rw-rw—- 1 mysql mysql 191 Jun 29 04:33 vm-5cc5-bad6.000007
srwxrwxrwx 1 mysql mysql 0 Jun 29 04:33 mysql.sock
-rw-r—– 1 mysql root 6.7K Jun 29 04:33 mysqld.log
-rw-rw—- 1 mysql mysql 76M Jun 29 04:33 ibdata1
-rw-rw—- 1 mysql mysql 48M Jun 29 04:33 ib_logfile0
[ root @ dbversity-fabric-1: /var/lib/mysql ]
MySQL Fabric needs to be able to access this state store and so a dedicated user is created (note that the fabric database hasn’t yet been created – that will be done soon using the mysqlfabric command):

[ root @ dbversity-fabric-1: /var/lib/mysql ] mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql>
mysql>
mysql> GRANT ALL PRIVILEGES ON fabric.* TO fabric@localhost;
Query OK, 0 rows affected (0.00 sec)

mysql>
.

MySQL Fabric has its own configuration file (note that its location can vary depending on your platform and how MySQL Utilities were installed). The contents of this configuration file should be reviewed before starting the MySQL Fabric process (in this case, the mysqldump_program and mysqlclient_program settings needed to be changed as MySQL was installed in a user’s directory) and the MySQL Fabric management port was changed to 8080 and authentication for the management interface was disabled:
[ root @ dbversity-fabric-1: /var/lib/mysql ] cat /etc/mysql/fabric.cfg
[DEFAULT]
prefix = /usr/local
sysconfdir = /usr/local/etc
logdir = /var/log

[logging]
level = INFO
url = file:///tmp/fabric.log

[storage]
address = localhost:3306
user = fabric
password =
database = fabric
auth_plugin = mysql_native_password
connection_timeout = 6
connection_attempts = 6
connection_delay = 1

[servers]
user = fabric
password =
unreachable_timeout = 5

[protocol.xmlrpc]
address = localhost:32274
threads = 5
user = admin
password = admin
disable_authentication = no
realm = MySQL Fabric
ssl_ca =
ssl_cert =
ssl_key =

[protocol.mysql]
address = localhost:32275
user = admin
password =
disable_authentication = no
ssl_ca =
ssl_cert =
ssl_key =

[executor]
executors = 5

[sharding]
mysqldump_program = /usr/bin/mysqldump
mysqlclient_program = /usr/bin/mysql

[statistics]
prune_time = 3600

[failure_tracking]
notifications = 300
notification_clients = 50
notification_interval = 60
failover_interval = 1
detections = 3
detection_interval = 6
detection_timeout = 1
prune_time = 3600

[connector]
ttl = 1

[ root @ dbversity-fabric-1: /var/lib/mysql ]

The final step before starting the MySQL Fabric process is to create the MySQL Fabric schema within the state store:

[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric manage setup –param=storage.user=fabric
[INFO] 1435566172.304603 – MainThread – Initializing persister: user (fabric), server (localhost:3306), database (fabric).
[INFO] 1435566173.626526 – MainThread – Initial password for admin/mysql set
Password set for admin/mysql from configuration file.
[INFO] 1435566173.629744 – MainThread – Password set for admin/mysql from configuration file.
[INFO] 1435566173.630340 – MainThread – Initial password for admin/xmlrpc set
Password set for admin/xmlrpc from configuration file.
[INFO] 1435566173.633207 – MainThread – Password set for admin/xmlrpc from configuration file.
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]

An optional step is then to check for yourself that the schema is indeed there:
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysql

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 17
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql>
mysql> show databases;
+——————–+
| Database |
+——————–+
| information_schema |
| fabric |
| mysql |
| performance_schema |
| test |
+——————–+
5 rows in set (0.00 sec)

mysql> use fabric;show tables;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
+——————-+
| Tables_in_fabric |
+——————-+
| checkpoints |
| error_log |
| group_replication |
| group_view |
| groups |
| log |
| machines |
| permissions |
| proc_view |
| providers |
| role_permissions |
| roles |
| servers |
| shard_maps |
| shard_ranges |
| shard_tables |
| shards |
| user_roles |
| users |
+——————-+
19 rows in set (0.00 sec)

mysql>
mysql> ^DBye
The MySQL Fabric process can now be started; in this case the process will run from the terminal from which it’s started but the –daemonize option can be used to make it run as a daemon.
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric manage start
[INFO] 1435566240.743661 – MainThread – Initializing persister: user (fabric), server (localhost:3306), database (fabric).
[WARNING] 1435566240.752884 – MainThread – Provider error: No module named novaclient.
[INFO] 1435566240.753186 – MainThread – Loading Services.
[INFO] 1435566240.766341 – MainThread – MySQL-RPC protocol server started, listening on localhost:32275
[INFO] 1435566240.778824 – MainThread – Fabric node starting.
[INFO] 1435566240.788106 – MainThread – Starting Executor.
[INFO] 1435566240.788199 – MainThread – Setting 5 executor(s).
[INFO] 1435566240.788591 – Executor-0 – Started.
[INFO] 1435566240.789377 – Executor-1 – Started.
[INFO] 1435566240.790558 – Executor-2 – Started.
[INFO] 1435566240.792450 – Executor-3 – Started.
[INFO] 1435566240.794814 – MainThread – Executor started.
[INFO] 1435566240.796965 – Executor-4 – Started.
[INFO] 1435566240.802330 – MainThread – Starting failure detector.
[INFO] 1435566240.804435 – XML-RPC-Server – XML-RPC protocol server (‘127.0.0.1’, 32274) started.
[INFO] 1435566240.805010 – XML-RPC-Server – Setting 5 XML-RPC session(s).
[INFO] 1435566240.805483 – XML-RPC-Session-0 – Started XML-RPC-Session.
[INFO] 1435566240.808021 – XML-RPC-Session-2 – Started XML-RPC-Session.
[INFO] 1435566240.808133 – XML-RPC-Session-1 – Started XML-RPC-Session.
[INFO] 1435566240.809423 – XML-RPC-Session-3 – Started XML-RPC-Session.
[INFO] 1435566240.811768 – XML-RPC-Session-4 – Started XML-RPC-Session.
If the process had been run as a daemon then it’s useful to be able to check if it’s actually running:

[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric manage ping
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
[ root @ dbversity-fabric-1: /var/lib/mysql ]
============================================================================================================================================================

Adding MySQL Servers to Create a HA Farm :

At this point, MySQL Fabric is up and running but it has no MySQL Servers to manage.
mysql1 12d4-dl585-03 (xx.xx.87.36) – mysql master
mysql2 dbversity-mysql-4 (xx.xx.87.37) – mysql slave
mysql3 dbversity-mysql-3 (xx.xx.45.124) – mysql slave

Three MySQL Servers will make up the managed HA group – each running on a different machine – these are the configuration files for each

[root@dbversity-mysql-2 mysql]# cat /etc/my.cnf
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock

[mysqld]
log-bin=dbversity-mysql-2.log
server-id = 1
binlog-format=ROW
log-slave-updates=true
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
report-host=xx.xx.87.36
report-port=3306

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysqld.pid
[root@dbversity-mysql-2 mysql]#

[root@dbversity-mysql-4 mysql]# cat /etc/my.cnf
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]

# Replication settings

server-id=2

binlog-format=ROW
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
report-host=xx.xx.87.36
report-port=3306
log-bin=dbversity-mysql-4.log
log-slave-updates=true

[mysqld_safe]
log-error=/var/lib/mysql/mysqld.log
pid-file=/var/lib/mysql/mysqld.pid

[root@dbversity-mysql-4 mysql]#
[root@dbversity-mysql-3 sybase]# cat /etc/my.cnf
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]

# Replication settings

server-id=3
binlog-format=ROW
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
report-host=xx.xx.45.124
report-port=3306
log-bin=dbversity-mysql-3.log
log-slave-updates=true

[mysqld_safe]
log-error=/var/lib/mysql/mysqld.log
pid-file=/var/lib/mysql/mysqld.pid
[root@dbversity-mysql-3 sybase]#
[root@dbversity-mysql-2 mysql]# ps -ef | grep mysql | grep -v grep | grep -v java
root 10804 1 0 03:41 pts/2 00:00:00 /bin/sh /usr/bin/mysqld_safe –datadir=/var/lib/mysql –pid-file=/var/lib/mysql/dbversity-mysql-2.pid
mysql 11067 10804 0 03:41 pts/2 00:00:11 /usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql –plugin-dir=/usr/lib64/mysql/plugin –user=mysql –log-error=/var/log/mysqld.log –pid-file=/var/lib/mysql/dbversity-mysql-2.pid
[root@dbversity-mysql-2 mysql]#

[root@dbversity-mysql-4 mysql]# ps -ef | grep mysql | grep -v grep | grep -v java
root 9054 1 0 03:10 pts/0 00:00:00 /bin/sh /usr/bin/mysqld_safe –datadir=/var/lib/mysql –pid-file=/var/lib/mysql/dbversity-mysql-4.pid
mysql 9313 9054 0 03:10 pts/0 00:00:03 /usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql –plugin-dir=/usr/lib64/mysql/plugin –user=mysql –log-error=/var/lib/mysql/mysqld.log –pid-file=/var/lib/mysql/dbversity-mysql-4.pid
[root@dbversity-mysql-4 mysql]#
[root@dbversity-mysql-3 sybase]# ps -ef | grep mysql | grep -v grep | grep -v java
root 25785 1 0 03:39 pts/1 00:00:00 /bin/sh /usr/bin/mysqld_safe –datadir=/var/lib/mysql –pid-file=/var/lib/mysql/dbversity-mysql-3.pid
mysql 26066 25785 0 03:39 pts/1 00:00:02 /usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql –plugin-dir=/usr/lib64/mysql/plugin –user=mysql –log-error=/var/lib/mysql/mysqld.log –pid-file=/var/lib/mysql/dbversity-mysql-3.pid
[root@dbversity-mysql-3 sybase]#

Create fabric user in all the three hosts.

GRANT ALL PRIVILEGES ON *.* TO fabric@’%’;

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 14489
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql>
mysql>
mysql>
mysql>
mysql> GRANT ALL PRIVILEGES ON *.* TO fabric@’%’;
Query OK, 0 rows affected (0.00 sec)
mysql>
mysql> GRANT ALL PRIVILEGES ON *.* TO root@’%’;
Query OK, 0 rows affected (0.00 sec)
[root@dbversity-mysql-4 mysql]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql>
mysql>
mysql> GRANT ALL PRIVILEGES ON *.* TO fabric@’%’;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO root@’%’;
Query OK, 0 rows affected (0.00 sec)
[root@dbversity-mysql-3 mysql]# mysql -u root -p”DBvers1ty”
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql>
mysql>
mysql> GRANT ALL PRIVILEGES ON *.* TO fabric@’%’;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO root@’%’;
Query OK, 0 rows affected (0.00 sec)
At this point, the MySQL Fabric process (and its associate state store) is up and running, as are the MySQL Servers that will become part of the HA group.
MySQL Fabric is now able to access and manipulate those MySQL Servers and so they can now be added to a HA group called mysql-ha-group

mysqlfabric group create mysql-ha-group

[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric group create mysql-ha-group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
———————————— ——– ——- ——
88ba9a9a-7cc4-44ca-bd96-a10d0646ba09 1 1 1

state success when description
—– ——- ————- ————————————————————-
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x17314d0>.
4 2 1.43557e+09 Executing action (_create_group).
5 2 1.43557e+09 Executed action (_create_group).
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric group add mysql-ha-group xx.xx.87.36:3306
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
———————————— ——– ——- ——
5f75f636-264e-4c21-8e25-ee00fe5f6d09 1 1 1

state success when description
—– ——- ————- ————————————————————-
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x1731890>.
4 2 1.43557e+09 Executing action (_add_server).
5 2 1.43557e+09 Executed action (_add_server).
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
———————————— ——– ——- ——
54239862-2e5e-46b1-a358-5cce7bf6288d 1 1 1

state success when description
—– ——- ————- ————————————————————-
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x1731890>.
4 2 1.43557e+09 Executing action (_add_server).
5 2 1.43557e+09 Executed action (_add_server).
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric group add mysql-ha-group xx.xx.45.124:3306
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
———————————— ——– ——- ——
ec03dfdd-3161-4a4a-8f0f-949410ab69b3 1 1 1

state success when description
—– ——- ————- ————————————————————-
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x1731890>.
4 2 1.43557e+09 Executing action (_add_server).
5 2 1.43557e+09 Executed action (_add_server).
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric group lookup_servers mysql-ha-group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
———————————— —————– ——— ——— ——
42aba790-1e32-11e5-926e-001635823fd4 xx.xx.87.36:3306 SECONDARY READ_ONLY 1.0
df9f3814-1e2d-11e5-9251-001a4bbee66e xx.xx.87.37:3306 SECONDARY READ_ONLY 1.0
ffe5315f-1e3e-11e5-92c1-005056b84054 xx.xx.45.124:3306 SECONDARY READ_ONLY 1.0
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]

Note that all of the MySQL Servers are reported as being Secondaries (in other words, none of them is acting as the MySQL Replication master).
The next step is to promote one of the servers to be the Primary; in this case the uuid of the server we want to promote is provided but that isn’t required – in which case MySQL Fabric will select one.
You can also check each server UUID from below location. Please note that prior to MySQL 5.6, you won’t see this parameter & thus you need MySQL 5.6+ for MySQL fabric set-up.

[root@dbversity-mysql-2 mysql]# pwd
/var/lib/mysql
[root@dbversity-mysql-2 mysql]# cat auto.cnf
[auto]
server-uuid=42aba790-1e32-11e5-926e-001635823fd4
[root@dbversity-mysql-2 mysql]#

Now let’s MySQL fabric select Primary for us ….. as below.
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric group promote mysql-ha-group
[WARNING] 1435569823.841939 – Executor-4 – Candidate (df9f3814-1e2d-11e5-9251-001a4bbee66e) cannot become a master due to the following reasons: issues to become a master ({‘is_gtid_not_enabled’: False, ‘is_slave_updates_not_enabled’: False, ‘is_not_running’: False, ‘is_binlog_not_enabled’: False, ‘no_rpl_user’: False}), prerequistes as a slave ({}), valid master (True).
[WARNING] 1435569824.716206 – Executor-4 – Candidate (ffe5315f-1e3e-11e5-92c1-005056b84054) cannot become a master due to the following reasons: issues to become a master ({‘is_gtid_not_enabled’: False, ‘is_slave_updates_not_enabled’: False, ‘is_not_running’: False, ‘is_binlog_not_enabled’: False, ‘no_rpl_user’: False}), prerequistes as a slave ({}), valid master (True).
[WARNING] 1435569826.207743 – Executor-4 – Error trying to process transactions in the relay log for candidate (<server(uuid=42aba790-1e32-11e5-926e-001635823fd4, address=xx.xx.87.36:3306, mode=READ_ONLY, status=SECONDARY>): Command (START SLAVE SQL_THREAD, ()) failed accessing (xx.xx.87.36:3306). 1200 (HY000): The server is not configured as slave; fix in config file or with CHANGE MASTER TO..
[INFO] 1435569827.178752 – Executor-4 – Master has changed from None to 42aba790-1e32-11e5-926e-001635823fd4.
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
———————————— ——– ——- ——
bebe6e35-0427-40d3-9e94-5455ed29e4d9 1 1 1

state success when description
—– ——- ————- ————————————————————-
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x1581590>.
4 2 1.43557e+09 Executing action (_define_ha_operation).
5 2 1.43557e+09 Executed action (_define_ha_operation).
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x16b6190>.
4 2 1.43557e+09 Executing action (_find_candidate_fail).
5 2 1.43557e+09 Executed action (_find_candidate_fail).
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x16a2c50>.
4 2 1.43557e+09 Executing action (_check_candidate_fail).
5 2 1.43557e+09 Executed action (_check_candidate_fail).
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x16a2f10>.
4 2 1.43557e+09 Executing action (_wait_slave_fail).
5 2 1.43557e+09 Executed action (_wait_slave_fail).
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x16b5050>.
4 2 1.43557e+09 Executing action (_change_to_candidate).
5 2 1.43557e+09 Executed action (_change_to_candidate).
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric group lookup_servers mysql-ha-group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
———————————— —————– ——— ———- ——
42aba790-1e32-11e5-926e-001635823fd4 xx.xx.87.36:3306 PRIMARY READ_WRITE 1.0
df9f3814-1e2d-11e5-9251-001a4bbee66e xx.xx.87.37:3306 SECONDARY READ_ONLY 1.0
ffe5315f-1e3e-11e5-92c1-005056b84054 xx.xx.45.124:3306 SECONDARY READ_ONLY 1.0
[ root @ dbversity-fabric-1: /var/lib/mysql ]

Now Let’s see how to promote a server by it’s UUID manually ( i.e., not by MySQL Fabric select one for us )
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric group promote mysql-ha-group –slave_id df9f3814-1e2d-11e5-9251-001a4bbee66e
[INFO] 1435569972.823658 – Executor-0 – Master has changed from 42aba790-1e32-11e5-926e-001635823fd4 to None.
[INFO] 1435569975.806162 – Executor-0 – Master has changed from None to df9f3814-1e2d-11e5-9251-001a4bbee66e.
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
———————————— ——– ——- ——
5d234c62-3c42-4226-a262-b9eba3b2a03c 1 1 1

state success when description
—– ——- ————- ————————————————————-
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x1581590>.
4 2 1.43557e+09 Executing action (_define_ha_operation).
5 2 1.43557e+09 Executed action (_define_ha_operation).
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x16a2e50>.
4 2 1.43557e+09 Executing action (_check_candidate_switch).
5 2 1.43557e+09 Executed action (_check_candidate_switch).
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x16a2e90>.
4 2 1.43557e+09 Executing action (_block_write_switch).
5 2 1.43557e+09 Executed action (_block_write_switch).
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x16a2a10>.
4 2 1.43557e+09 Executing action (_wait_slaves_switch).
5 2 1.43557e+09 Executed action (_wait_slaves_switch).
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x16b5050>.
4 2 1.43557e+09 Executing action (_change_to_candidate).
5 2 1.43557e+09 Executed action (_change_to_candidate).
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ]
[ root @ dbversity-fabric-1: /var/lib/mysql ] mysqlfabric group lookup_servers mysql-ha-group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
———————————— —————– ——— ———- ——
42aba790-1e32-11e5-926e-001635823fd4 xx.xx.87.36:3306 SECONDARY READ_ONLY 1.0
df9f3814-1e2d-11e5-9251-001a4bbee66e xx.xx.87.37:3306 PRIMARY READ_WRITE 1.0
ffe5315f-1e3e-11e5-92c1-005056b84054 xx.xx.45.124:3306 SECONDARY READ_ONLY 1.0
[ root @ dbversity-fabric-1: /var/lib/mysql ]

Note that xx.xx.87.37 is now showing as the Primary; any of the Secondary servers can also be queried to confirm that they are indeed MySQL replication slaves of the Primary.

[root@dbversity-mysql-4 mysql]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 13
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql>
mysql>
mysql> show master status\G
*************************** 1. row ***************************
File: dbversity-mysql-4.000004
Position: 851
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: 42aba790-1e32-11e5-926e-001635823fd4:1-2, df9f3814-1e2d-11e5-9251-001a4bbee66e:1-2
1 row in set (0.00 sec)

mysql>
Let’s check “show slave status” at SECONDARIES
[root@dbversity-mysql-2 mysql]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 18866
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: xx.xx.87.37
Master_User: fabric
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: dbversity-mysql-4.000004
Read_Master_Log_Pos: 851
Relay_Log_File: dbversity-mysql-2-relay-bin.000002
Relay_Log_Pos: 770
Relay_Master_Log_File: dbversity-mysql-4.000004
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 851
Relay_Log_Space: 982
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2
Master_UUID: df9f3814-1e2d-11e5-9251-001a4bbee66e
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: df9f3814-1e2d-11e5-9251-001a4bbee66e:1-2
Executed_Gtid_Set: 42aba790-1e32-11e5-926e-001635823fd4:1-2,
df9f3814-1e2d-11e5-9251-001a4bbee66e:1-2
Auto_Position: 1
1 row in set (0.00 sec)

mysql>
mysql>
[root@dbversity-mysql-3 mysql]# mysql -u root -p”DBvers1ty”
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: xx.xx.87.37
Master_User: fabric
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: dbversity-mysql-4.000004
Read_Master_Log_Pos: 851
Relay_Log_File: dbversity-mysql-3-relay-bin.000002
Relay_Log_Pos: 420
Relay_Master_Log_File: dbversity-mysql-4.000004
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 851
Relay_Log_Space: 632
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2
Master_UUID: df9f3814-1e2d-11e5-9251-001a4bbee66e
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 42aba790-1e32-11e5-926e-001635823fd4:1-2,
df9f3814-1e2d-11e5-9251-001a4bbee66e:1-2,
ffe5315f-1e3e-11e5-92c1-005056b84054:1-3
Auto_Position: 1
1 row in set (0.00 sec)
At this stage, the MySQL replication relationship is configured and running but there isn’t yet High Availability as MySQL Fabric is not monitoring the state of the servers – the final configuration step fixes that:
Let’s activate the mysqlfabric group.

[ root @ dbversity-fabric-1: ~ ] mysqlfabric group activate mysql-ha-group
[INFO] 1435570418.576711 – Executor-1 – Monitoring group (mysql-ha-group).
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
———————————— ——– ——- ——
b83bda94-aa0d-4e76-968a-277c393cdcb5 1 1 1

state success when description
—– ——- ————- ————————————————————-
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x1731990>.
4 2 1.43557e+09 Executing action (_activate_group).
5 2 1.43557e+09 Executed action (_activate_group).
[ root @ dbversity-fabric-1: ~ ]
Everything is now set up to detect if the Primary (master) should fail and in the event that it does, promote one of the Secondaries to be the new Primary.

let’s stop primary & see what will happen

[root@dbversity-mysql-4 ~]# service mysql stop
Shutting down MySQL………… [ OK ]
[root@dbversity-mysql-4 ~]#
At MySQL fabric server
[ root @ dbversity-fabric-1: ~ ] mysqlfabric group lookup_servers mysql-ha-group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
———————————— —————– ——— ———- ——
42aba790-1e32-11e5-926e-001635823fd4 xx.xx.87.36:3306 SECONDARY READ_ONLY 1.0
df9f3814-1e2d-11e5-9251-001a4bbee66e xx.xx.87.37:3306 PRIMARY READ_WRITE 1.0
ffe5315f-1e3e-11e5-92c1-005056b84054 xx.xx.45.124:3306 SECONDARY READ_ONLY 1.0
[ root @ dbversity-fabric-1: ~ ]
[ root @ dbversity-fabric-1: ~ ]
[ root @ dbversity-fabric-1: ~ ] [WARNING] 1435570547.075298 – FailureDetector(mysql-ha-group) – Server (df9f3814-1e2d-11e5-9251-001a4bbee66e) in group (mysql-ha-group) is unreachable.
[WARNING] 1435570549.500174 – FailureDetector(mysql-ha-group) – Server (df9f3814-1e2d-11e5-9251-001a4bbee66e) in group (mysql-ha-group) is unreachable.
[WARNING] 1435570551.924605 – FailureDetector(mysql-ha-group) – Server (df9f3814-1e2d-11e5-9251-001a4bbee66e) in group (mysql-ha-group) is unreachable.
[WARNING] 1435570551.942220 – Executor-3 – Reported issue (FAULTY) for server (df9f3814-1e2d-11e5-9251-001a4bbee66e).
[INFO] 1435570551.944626 – Executor-3 – Master (df9f3814-1e2d-11e5-9251-001a4bbee66e) in group (mysql-ha-group) has been lost.
[INFO] 1435570556.528646 – Executor-3 – Master has changed from df9f3814-1e2d-11e5-9251-001a4bbee66e to ffe5315f-1e3e-11e5-92c1-005056b84054.

[ root @ dbversity-fabric-1: ~ ]
[ root @ dbversity-fabric-1: ~ ] mysqlfabric group lookup_servers mysql-ha-group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
———————————— —————– ——— ———- ——
42aba790-1e32-11e5-926e-001635823fd4 xx.xx.87.36:3306 SECONDARY READ_ONLY 1.0
df9f3814-1e2d-11e5-9251-001a4bbee66e xx.xx.87.37:3306 FAULTY READ_WRITE 1.0
ffe5315f-1e3e-11e5-92c1-005056b84054 xx.xx.45.124:3306 PRIMARY READ_WRITE 1.0
[ root @ dbversity-fabric-1: ~ ]

Now let’s stop the current Primary
[root@dbversity-mysql-3 ~]# hostname -i
xx.xx.45.124
[root@dbversity-mysql-3 ~]#
[root@dbversity-mysql-3 ~]# service mysql stop
Shutting down MySQL………… SUCCESS!
[root@dbversity-mysql-3 ~]#

At MySQL Fabric server

[ root @ dbversity-fabric-1: ~ ] [WARNING] 1435570675.134683 – FailureDetector(mysql-ha-group) – Server (ffe5315f-1e3e-11e5-92c1-005056b84054) in group (mysql-ha-group) is unreachable.
[WARNING] 1435570677.367771 – FailureDetector(mysql-ha-group) – Server (ffe5315f-1e3e-11e5-92c1-005056b84054) in group (mysql-ha-group) is unreachable.
[WARNING] 1435570679.602582 – FailureDetector(mysql-ha-group) – Server (ffe5315f-1e3e-11e5-92c1-005056b84054) in group (mysql-ha-group) is unreachable.
[WARNING] 1435570679.618963 – Executor-2 – Reported issue (FAULTY) for server (ffe5315f-1e3e-11e5-92c1-005056b84054).
[INFO] 1435570679.621658 – Executor-2 – Master (ffe5315f-1e3e-11e5-92c1-005056b84054) in group (mysql-ha-group) has been lost.
[INFO] 1435570683.234218 – Executor-2 – Master has changed from ffe5315f-1e3e-11e5-92c1-005056b84054 to 42aba790-1e32-11e5-926e-001635823fd4.

[ root @ dbversity-fabric-1: ~ ]
[ root @ dbversity-fabric-1: ~ ] mysqlfabric group lookup_servers mysql-ha-group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
———————————— —————– ——- ———- ——
42aba790-1e32-11e5-926e-001635823fd4 xx.xx.87.36:3306 PRIMARY READ_WRITE 1.0
df9f3814-1e2d-11e5-9251-001a4bbee66e xx.xx.87.37:3306 FAULTY READ_WRITE 1.0
ffe5315f-1e3e-11e5-92c1-005056b84054 xx.xx.45.124:3306 FAULTY READ_WRITE 1.0
[ root @ dbversity-fabric-1: ~ ]

To configure server as a Slave ….in case of server crash.

CHANGE MASTER TO MASTER_HOST=’xx.xx.87.36′,
MASTER_PORT=3306,
MASTER_USER=’fabric’,
MASTER_LOG_FILE = ‘dbversity-mysql-2.000006’,
MASTER_LOG_POS = 271;

To deactivate a group …..

[ root @ dbversity-fabric-1: ~ ] mysqlfabric group deactivate mysql-ha-group
[INFO] 1435574067.865310 – Executor-2 – Stop monitoring group (mysql-ha-group).
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
———————————— ——– ——- ——
5e7b2395-8674-48d6-bc84-560f1cbc0446 1 1 1

state success when description
—– ——- ————- ————————————————————-
3 2 1.43557e+09 Triggered by <mysql.fabric.events.Event object at 0x1731b10>.
4 2 1.43557e+09 Executing action (_deactivate_group).
5 2 1.43557e+09 Executed action (_deactivate_group).
[ root @ dbversity-fabric-1: ~ ]
[ root @ dbversity-fabric-1: ~ ] mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 5.6.25-log MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql>
mysql>
mysql> show databases;
+——————–+
| Database |
+——————–+
| information_schema |
| fabric |
| mysql |
| performance_schema |
| test |
+——————–+
5 rows in set (0.00 sec)

mysql>
mysql> use fabric;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql>
mysql>
mysql> select * from servers;
+————————————–+——————-+——+——–+——–+—————-+
| server_uuid | server_address | mode | status | weight | group_id |
+————————————–+——————-+——+——–+——–+—————-+
| 42aba790-1e32-11e5-926e-001635823fd4 | xx.xx.87.36:3306 | 3 | 0 | 1 | mysql-ha-group |
| df9f3814-1e2d-11e5-9251-001a4bbee66e | xx.xx.87.37:3306 | 3 | 0 | 1 | mysql-ha-group |
| ffe5315f-1e3e-11e5-92c1-005056b84054 | xx.xx.45.124:3306 | 3 | 0 | 1 | mysql-ha-group |
+————————————–+——————-+——+——–+——–+—————-+
3 rows in set (0.00 sec)
mysql>
mysql> select * from groups;
+—————-+————-+————————————–+—————————-+——–+
| group_id | description | master_uuid | master_defined | status |
+—————-+————-+————————————–+—————————-+——–+
| mysql-ha-group | NULL | 42aba790-1e32-11e5-926e-001635823fd4 | 2015-06-29 09:38:03.000000 | |
| MySQL_HA_Grp | NULL | NULL | NULL | |
+—————-+————-+————————————–+—————————-+——–+
2 rows in set (0.00 sec)

mysql>
mysql> select * from error_log;
+————————————–+—————————-+———————————+——–+
| server_uuid | reported | reporter | error |
+————————————–+—————————-+———————————+——–+
| df9f3814-1e2d-11e5-9251-001a4bbee66e | 2015-06-29 09:35:51.000000 | FailureDetector(mysql-ha-group) | FAULTY |
| ffe5315f-1e3e-11e5-92c1-005056b84054 | 2015-06-29 09:37:59.000000 | FailureDetector(mysql-ha-group) | FAULTY |
| 42aba790-1e32-11e5-926e-001635823fd4 | 2015-06-29 09:39:30.000000 | FailureDetector(mysql-ha-group) | FAULTY |
+————————————–+—————————-+———————————+——–+
3 rows in set (0.00 sec)

mysql> select * from group_replication;
Empty set (0.00 sec)

mysql> select * from group_view;
+—————-+—————+————–+
| group_id | promote_count | demote_count |
+—————-+—————+————–+
| mysql-ha-group | 4 | 1 |
+—————-+—————+————–+
1 row in set (0.04 sec)

mysql> select * from log;
+———————-+—————————-+——————————+——————————————————————————————————-+———-+——+
| subject | reported | reporter | message | category | type |
+———————-+—————————-+——————————+——————————————————————————————————-+———-+——+
| 0 | 2015-06-29 08:24:00.000000 | mysql.fabric.services.manage | Fabric node starting. | 0 | 0 |
| manage.ping | 2015-06-29 08:24:48.000000 | mysql.fabric.command | Started command (manage, ping). | 1 | 0 |
| manage.ping | 2015-06-29 08:24:48.000000 | mysql.fabric.command | Finished command (manage, ping). | 1 | 1 |
| manage.ping | 2015-06-29 08:24:56.000000 | mysql.fabric.command | Started command (manage, ping). | 1 | 0 |
| manage.ping | 2015-06-29 08:24:56.000000 | mysql.fabric.command | Finished command (manage, ping). | 1 | 1 |
| manage.ping | 2015-06-29 08:38:36.000000 | mysql.fabric.command | Started command (manage, ping). | 1 | 0 |
| manage.ping | 2015-06-29 08:38:36.000000 | mysql.fabric.command | Finished command (manage, ping). | 1 | 1 |
| group.create | 2015-06-29 09:16:05.000000 | mysql.fabric.command | Started command (group, create). | 1 | 0 |
| group.create | 2015-06-29 09:16:05.000000 | mysql.fabric.command | Finished command (group, create). | 1 | 1 |
| group.add | 2015-06-29 09:17:00.000000 | mysql.fabric.command | Started command (group, add). | 1 | 0 |
| group.add | 2015-06-29 09:17:01.000000 | mysql.fabric.command | Finished command (group, add). | 1 | 1 |
| group.lookup_groups | 2015-06-29 09:17:23.000000 | mysql.fabric.command | Started command (group, lookup_groups). | 1 | 0 |
| group.lookup_groups | 2015-06-29 09:17:23.000000 | mysql.fabric.command | Finished command (group, lookup_groups). | 1 | 1 |
| group.add | 2015-06-29 09:17:36.000000 | mysql.fabric.command | Started command (group, add). | 1 | 0 |
| group.add | 2015-06-29 09:17:37.000000 | mysql.fabric.command | Finished command (group, add). | 1 | 1 |
| group.add | 2015-06-29 09:17:53.000000 | mysql.fabric.command | Started command (group, add). | 1 | 0 |
| group.add | 2015-06-29 09:17:54.000000 | mysql.fabric.command | Finished command (group, add). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:18:56.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:18:56.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.promote | 2015-06-29 09:23:42.000000 | mysql.fabric.command | Started command (group, promote). | 1 | 0 |
| mysql-ha-group | 2015-06-29 09:23:47.000000 | mysql.fabric.server | Master has changed from None to 42aba790-1e32-11e5-926e-001635823fd4. | 3 | 3 |
| group.promote | 2015-06-29 09:23:50.000000 | mysql.fabric.command | Finished command (group, promote). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:23:55.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:23:55.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.promote | 2015-06-29 09:26:11.000000 | mysql.fabric.command | Started command (group, promote). | 1 | 0 |
| mysql-ha-group | 2015-06-29 09:26:12.000000 | mysql.fabric.server | Master has changed from 42aba790-1e32-11e5-926e-001635823fd4 to None. | 3 | 4 |
| mysql-ha-group | 2015-06-29 09:26:15.000000 | mysql.fabric.server | Master has changed from None to df9f3814-1e2d-11e5-9251-001a4bbee66e. | 3 | 3 |
| group.promote | 2015-06-29 09:26:18.000000 | mysql.fabric.command | Finished command (group, promote). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:26:24.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:26:24.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.activate | 2015-06-29 09:33:38.000000 | mysql.fabric.command | Started command (group, activate). | 1 | 0 |
| group.activate | 2015-06-29 09:33:38.000000 | mysql.fabric.command | Finished command (group, activate). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:35:09.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:35:09.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:35:25.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:35:25.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:35:27.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:35:27.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| mysql-ha-group | 2015-06-29 09:35:56.000000 | mysql.fabric.server | Master has changed from df9f3814-1e2d-11e5-9251-001a4bbee66e to ffe5315f-1e3e-11e5-92c1-005056b84054. | 3 | 3 |
| group.lookup_servers | 2015-06-29 09:36:09.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:36:09.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| mysql-ha-group | 2015-06-29 09:38:03.000000 | mysql.fabric.server | Master has changed from ffe5315f-1e3e-11e5-92c1-005056b84054 to 42aba790-1e32-11e5-926e-001635823fd4. | 3 | 3 |
| group.lookup_servers | 2015-06-29 09:38:27.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:38:27.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:39:41.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:39:41.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:41:20.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:41:20.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:41:23.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:41:23.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:41:38.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:41:38.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:41:53.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:41:53.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:41:55.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:41:55.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:42:22.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:42:22.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:42:25.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:42:25.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:42:45.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:42:45.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:43:15.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:43:15.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:43:17.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:43:17.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:43:47.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:43:47.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.activate | 2015-06-29 09:43:53.000000 | mysql.fabric.command | Started command (group, activate). | 1 | 0 |
| group.activate | 2015-06-29 09:43:53.000000 | mysql.fabric.command | Finished command (group, activate). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:43:56.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:43:56.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:43:58.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:43:58.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:45:17.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:45:17.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 09:45:18.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 09:45:18.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.health | 2015-06-29 09:46:07.000000 | mysql.fabric.command | Started command (group, health). | 1 | 0 |
| group.health | 2015-06-29 09:46:09.000000 | mysql.fabric.command | Finished command (group, health). | 1 | 1 |
| group.health | 2015-06-29 09:56:30.000000 | mysql.fabric.command | Started command (group, health). | 1 | 0 |
| group.health | 2015-06-29 09:56:33.000000 | mysql.fabric.command | Finished command (group, health). | 1 | 1 |
| group.health | 2015-06-29 09:58:30.000000 | mysql.fabric.command | Started command (group, health). | 1 | 0 |
| group.health | 2015-06-29 09:58:31.000000 | mysql.fabric.command | Finished command (group, health). | 1 | 1 |
| group.health | 2015-06-29 10:04:38.000000 | mysql.fabric.command | Started command (group, health). | 1 | 0 |
| group.health | 2015-06-29 10:04:39.000000 | mysql.fabric.command | Finished command (group, health). | 1 | 1 |
| group.lookup_servers | 2015-06-29 10:04:46.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 10:04:46.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.activate | 2015-06-29 10:04:53.000000 | mysql.fabric.command | Started command (group, activate). | 1 | 0 |
| group.activate | 2015-06-29 10:04:53.000000 | mysql.fabric.command | Finished command (group, activate). | 1 | 1 |
| group.lookup_servers | 2015-06-29 10:04:56.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 10:04:56.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 10:27:49.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 10:27:49.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 10:29:45.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 10:29:45.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.activate | 2015-06-29 10:29:50.000000 | mysql.fabric.command | Started command (group, activate). | 1 | 0 |
| group.activate | 2015-06-29 10:29:50.000000 | mysql.fabric.command | Finished command (group, activate). | 1 | 1 |
| group.lookup_servers | 2015-06-29 10:29:54.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 10:29:54.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.lookup_servers | 2015-06-29 10:31:05.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 10:31:05.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.deactivate | 2015-06-29 10:34:27.000000 | mysql.fabric.command | Started command (group, deactivate). | 1 | 0 |
| group.deactivate | 2015-06-29 10:34:27.000000 | mysql.fabric.command | Finished command (group, deactivate). | 1 | 1 |
| group.lookup_servers | 2015-06-29 10:34:36.000000 | mysql.fabric.command | Started command (group, lookup_servers). | 1 | 0 |
| group.lookup_servers | 2015-06-29 10:34:36.000000 | mysql.fabric.command | Finished command (group, lookup_servers). | 1 | 1 |
| group.health | 2015-06-29 10:34:48.000000 | mysql.fabric.command | Started command (group, health). | 1 | 0 |
| group.health | 2015-06-29 10:34:48.000000 | mysql.fabric.command | Finished command (group, health). | 1 | 2 |
| group.health | 2015-06-29 10:34:56.000000 | mysql.fabric.command | Started command (group, health). | 1 | 0 |
| group.health | 2015-06-29 10:34:56.000000 | mysql.fabric.command | Finished command (group, health). | 1 | 2 |
| group.create | 2015-06-29 10:38:09.000000 | mysql.fabric.command | Started command (group, create). | 1 | 0 |
| group.create | 2015-06-29 10:38:09.000000 | mysql.fabric.command | Finished command (group, create). | 1 | 1 |
| group.add | 2015-06-29 10:38:43.000000 | mysql.fabric.command | Started command (group, add). | 1 | 0 |
| group.add | 2015-06-29 10:38:44.000000 | mysql.fabric.command | Finished command (group, add). | 1 | 2 |
| manage.stop | 2015-06-29 10:39:30.000000 | mysql.fabric.command | Started command (manage, stop). | 1 | 0 |
| Node | 2015-06-29 10:39:30.000000 | mysql.fabric.services.manage | Fabric node stopped. | 0 | 1 |
| manage.stop | 2015-06-29 10:39:30.000000 | mysql.fabric.command | Finished command (manage, stop). | 1 | 1 |
+———————-+—————————-+——————————+——————————————————————————————————-+———-+——+
117 rows in set (0.00 sec)

mysql>
mysql> select * from permissions ;
+—————+———–+———–+———-+———————————————-+
| permission_id | subsystem | component | function | description |
+—————+———–+———–+———-+———————————————-+
| 1 | core | NULL | NULL | Full access to all core Fabric functionality |
| 2 | core | dump | NULL | Access to dump commands |
| 3 | core | user | NULL | User administration |
| 4 | core | role | NULL | Role administration |
| 5 | core | threat | NULL | Reporting to Fabric |
+—————+———–+———–+———-+———————————————-+
5 rows in set (0.00 sec)

mysql> select * from users;
+———+———-+———-+——————————————+
| user_id | username | protocol | password |
+———+———-+———-+——————————————+
| 1 | admin | xmlrpc | ac5fbef09cf1ae6d43184cece612d6ef |
| 2 | admin | mysql | E0FFFF6FC6CAD86606B36A91BF00BEB14CF06EA6 |
+———+———-+———-+——————————————+
2 rows in set (0.00 sec)

mysql>
MySQL Fabric-aware connectors (initially PHP, Python and Java) then that failover can be transparent to the application
If using one of the MySQL Fabric-aware connectors (initially PHP, Python and Java) then that failover can be transparent to the application.
How does MySQL Fabric Compare with MySQL Cluster?

MySQL Cluster is a mature, well proven solution for providing very high levels of availability and scaling out of both reads and writes. Some of the main extra capabilities that MySQL Cluster has over MySQL Fabric are:

Synchronous replication
Faster (automated) fail-over (resulting in higher availability)
Transparent sharding
Cross-shard joins and Foreign Keys
In-memory, real-time performance
MySQL Fabric on the other hand, allows the application to stick with the InnoDB storage engine which is better suited to many applications.

  • Ask Question