MySQL Cluster：APIノードが正常に起動しない
私はテスト環境でMySQL NDBクラスターをセットアップする作業をしています。管理ノードが1つ、SQLノードが2つ、NDBノードが4つあります。管理ノードとNDBノードはすべて正常に起動します。 MySQLドキュメント のように、手動で呼び出されます。 mysqld_safe手動で、MySQLがロードされ、管理ノードに接続して、正しく実行されます。ただし、service mysql start、MySQLをロードして管理ノードに接続しますが、NDBノードがないと判断して待機します。
これは本当に私を混乱させます。 /etc/init.d/mysql start、サービスの実行と同じ問題があります。ただし、cp /etc/init.d/mysql ~; ./mysql startサービスは、サービスの実行にかかる時間のほんの一部で開始され、正しくロードされます。
MySQL Clusterのバージョンは7.4.10です。 CentOS 7で実行します。
MySQL as Service
[root@MySQL1 mysql]# cat MySQL1.Cluster.err
160329 12:38:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2016-03-29 12:38:56 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-03-29 12:38:56 0 [Note] /usr/sbin/mysqld (mysqld 5.6.28-ndb-7.4.10-cluster-gpl) starting as process 12269 ...
2016-03-29 12:38:56 12269 [Note] Plugin 'FEDERATED' is disabled.
2016-03-29 12:38:56 12269 [Note] InnoDB: Using atomics to ref count buffer pool pages
2016-03-29 12:38:56 12269 [Note] InnoDB: The InnoDB memory heap is disabled
2016-03-29 12:38:56 12269 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-03-29 12:38:56 12269 [Note] InnoDB: Memory barrier is not used
2016-03-29 12:38:56 12269 [Note] InnoDB: Compressed tables use zlib 1.2.3
2016-03-29 12:38:56 12269 [Note] InnoDB: Using Linux native AIO
2016-03-29 12:38:56 12269 [Note] InnoDB: Not using CPU crc32 instructions
2016-03-29 12:38:56 12269 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-03-29 12:38:56 12269 [Note] InnoDB: Completed initialization of buffer pool
2016-03-29 12:38:57 12269 [Note] InnoDB: Highest supported file format is Barracuda.
2016-03-29 12:38:57 12269 [Note] InnoDB: 128 rollback segment(s) are active.
2016-03-29 12:38:57 12269 [Note] InnoDB: Waiting for purge to start
2016-03-29 12:38:57 12269 [Note] InnoDB: 5.6.28 started; log sequence number 1626423
2016-03-29 12:38:57 12269 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
2016-03-29 12:38:57 12269 [Note] NDB: NodeID is 50, management server '10.0.0.200:1186'
2016-03-29 12:39:27 12269 [Note] NDB[0]: NodeID: 50, no storage nodes connected (timed out)
2016-03-29 12:39:27 12269 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Starting...
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Started
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Setting up
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Created schema Ndb object, reference: 0x80040032, name: 'Ndb Binlog schema change monitoring'
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Created injector Ndb object, reference: 0x80050032, name: 'Ndb Binlog data change monitoring'
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Setup completed
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Wait for server start completed
2016-03-29 12:39:27 12269 [Note] NDB Util: Starting...
2016-03-29 12:39:27 12269 [Note] NDB Util: Wait for server start completed
2016-03-29 12:39:27 12269 [Note] NDB Index Stat: Starting...
2016-03-29 12:39:27 12269 [Note] NDB Index Stat: Wait for server start completed
2016-03-29 12:39:27 12269 [Note] Server hostname (bind-address): '*'; port: 3306
2016-03-29 12:39:27 12269 [Note] IPv6 is available.
2016-03-29 12:39:27 12269 [Note] - '::' resolves to '::';
2016-03-29 12:39:27 12269 [Note] Server socket created on IP: '::'.
2016-03-29 12:39:27 12269 [Note] Event Scheduler: Loaded 0 events
2016-03-29 12:39:27 12269 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.28-ndb-7.4.10-cluster-gpl' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Cluster Community Server (GPL)
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Check for incidents
2016-03-29 12:39:27 12269 [Note] NDB Binlog: Wait for cluster to start
2016-03-29 12:39:27 12269 [Note] NDB Util: Wait for cluster to start
2016-03-29 12:39:27 12269 [Note] NDB Util: Started
2016-03-29 12:39:27 12269 [Note] NDB Index Stat: Wait for cluster to start
2016-03-29 12:39:27 12269 [Note] NDB Index Stat: Started
2016-03-29 12:39:27 12269 [Note] created index stats Ndb object: reference 0x80070032, name: 'Ndb Index Statistics monitoring'
2016-03-29 12:39:57 12269 [Warning] NDB : Tables not available after 30 seconds. Consider increasing --ndb-wait-setup value
上記で最も興味深い行は、次のとおりです。
2016-03-29 12:38:57 12269 [Note] NDB: NodeID is 50, management server '10.0.0.200:1186'
2016-03-29 12:39:27 12269 [Note] NDB[0]: NodeID: 50, no storage nodes connected (timed out)
2016-03-29 12:39:27 12269 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
mysqld_safeの手動呼び出し
[root@MySQL1 mysql]# cat MySQL1.Cluster.err
160329 12:44:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2016-03-29 12:44:16 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-03-29 12:44:16 0 [Note] /usr/sbin/mysqld (mysqld 5.6.28-ndb-7.4.10-cluster-gpl) starting as process 12684 ...
2016-03-29 12:44:16 12684 [Note] Plugin 'FEDERATED' is disabled.
2016-03-29 12:44:16 12684 [Note] InnoDB: Using atomics to ref count buffer pool pages
2016-03-29 12:44:16 12684 [Note] InnoDB: The InnoDB memory heap is disabled
2016-03-29 12:44:16 12684 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-03-29 12:44:16 12684 [Note] InnoDB: Memory barrier is not used
2016-03-29 12:44:16 12684 [Note] InnoDB: Compressed tables use zlib 1.2.3
2016-03-29 12:44:16 12684 [Note] InnoDB: Using Linux native AIO
2016-03-29 12:44:16 12684 [Note] InnoDB: Not using CPU crc32 instructions
2016-03-29 12:44:16 12684 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-03-29 12:44:16 12684 [Note] InnoDB: Completed initialization of buffer pool
2016-03-29 12:44:16 12684 [Note] InnoDB: Highest supported file format is Barracuda.
2016-03-29 12:44:16 12684 [Note] InnoDB: 128 rollback segment(s) are active.
2016-03-29 12:44:16 12684 [Note] InnoDB: Waiting for purge to start
2016-03-29 12:44:16 12684 [Note] InnoDB: 5.6.28 started; log sequence number 1626433
2016-03-29 12:44:16 12684 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
2016-03-29 12:44:16 12684 [Note] NDB: NodeID is 50, management server '10.0.0.200:1186'
2016-03-29 12:44:17 12684 [Note] NDB[0]: NodeID: 50, all storage nodes connected
2016-03-29 12:44:17 12684 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Starting...
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Started
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Setting up
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Created schema Ndb object, reference: 0x80040032, name: 'Ndb Binlog schema change monitoring'
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Created injector Ndb object, reference: 0x80050032, name: 'Ndb Binlog data change monitoring'
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Setup completed
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Wait for server start completed
2016-03-29 12:44:17 12684 [Note] NDB Util: Starting...
2016-03-29 12:44:17 12684 [Note] NDB Util: Wait for server start completed
2016-03-29 12:44:17 12684 [Note] NDB Index Stat: Starting...
2016-03-29 12:44:17 12684 [Note] NDB Index Stat: Wait for server start completed
2016-03-29 12:44:17 12684 [Note] Server hostname (bind-address): '*'; port: 3306
2016-03-29 12:44:17 12684 [Note] IPv6 is available.
2016-03-29 12:44:17 12684 [Note] - '::' resolves to '::';
2016-03-29 12:44:17 12684 [Note] Server socket created on IP: '::'.
2016-03-29 12:44:17 12684 [Note] Event Scheduler: Loaded 0 events
2016-03-29 12:44:17 12684 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.28-ndb-7.4.10-cluster-gpl' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Cluster Community Server (GPL)
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Check for incidents
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Wait for cluster to start
2016-03-29 12:44:17 12684 [Note] NDB Util: Wait for cluster to start
2016-03-29 12:44:17 12684 [Note] NDB Util: Started
2016-03-29 12:44:17 12684 [Note] NDB Index Stat: Wait for cluster to start
2016-03-29 12:44:17 12684 [Note] NDB Index Stat: Started
2016-03-29 12:44:17 12684 [Note] created index stats Ndb object: reference 0x80070032, name: 'Ndb Index Statistics monitoring'
2016-03-29 12:44:17 12684 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema
2016-03-29 12:44:17 12684 [Note] NDB Binlog: logging ./mysql/ndb_schema (UPDATED,USE_WRITE)
2016-03-29 12:44:17 12684 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_apply_status
2016-03-29 12:44:17 12684 [Note] NDB Binlog: logging ./mysql/ndb_apply_status (UPDATED,USE_WRITE)
2016-03-29 12:44:17 12684 [Note] NDB: Cleaning stray tables from database 'ndbinfo'
2016-03-29 12:44:17 12684 [Note] NDB: Cleaning stray tables from database 'performance_schema'
2016-03-29 12:44:17 12684 [Note] NDB: Cleaning stray tables from database 'test'
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Wait for first event
2016-03-29 12:44:17 [NdbApi] INFO -- Flushing incomplete GCI:s < 212100/0
2016-03-29 12:44:17 [NdbApi] INFO -- Flushing incomplete GCI:s < 212100/0
2016-03-29 12:44:17 12684 [Note] NDB Binlog: starting log at Epoch 212100/0
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Got first event
2016-03-29 12:44:17 12684 [Note] NDB Binlog: ndb tables writable
2016-03-29 12:44:17 12684 [Note] NDB Binlog: Startup and setup completed
2016-03-29 12:44:17 12684 [Note] NDB Schema dist: Data node: 12 reports subscribe from node 51, subscriber bitmask 800000
2016-03-29 12:44:17 12684 [Note] NDB Schema dist: Data node: 11 reports subscribe from node 51, subscriber bitmask 800000
2016-03-29 12:44:17 12684 [Note] NDB Schema dist: Data node: 13 reports subscribe from node 51, subscriber bitmask 800000
2016-03-29 12:44:17 12684 [Note] NDB Schema dist: Data node: 14 reports subscribe from node 51, subscriber bitmask 800000
これらのログの同じセクションを見ると、それは異なります。
2016-03-29 12:44:16 12684 [Note] NDB: NodeID is 50, management server '10.0.0.200:1186'
2016-03-29 12:44:17 12684 [Note] NDB[0]: NodeID: 50, all storage nodes connected
2016-03-29 12:44:17 12684 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
後者の場合、APIノードが管理ノードに接続していることもわかります。前者では、管理ノードが接続を待機しています。
OK。この問題は、SELinuxに関連しています。 setenforce 0を使用してSELinuxをパーミッシブモードにし、MySQLを再起動しましたが、正常に接続されました。これにより、すぐに問題が解決しました。だから私はaudit2whyをロードしてそれからそれを理解しました。次のコマンドを実行してsetenforce 1を実行し、SELinuxを再度有効にし、MySQLを再起動すると、すべて期待どおりに機能します。
魔法のコマンド：setsebool -P mysql_connect_any 1
MySQLは管理ノードに接続してストレージノードを読み取りますが、ストレージノードに接続しようとすると（ランダムポートではなくポート50501で構成されています）SELinuxがステップインしてリクエストをブロックしているようです。
Ndb_mgmからの最終出力：
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)] 4 node(s)
id=11 @10.0.0.220 (mysql-5.6.28 ndb-7.4.10, Nodegroup: 0, *)
id=12 @10.0.0.221 (mysql-5.6.28 ndb-7.4.10, Nodegroup: 0)
id=13 @10.0.0.222 (mysql-5.6.28 ndb-7.4.10, Nodegroup: 1)
id=14 @10.0.0.223 (mysql-5.6.28 ndb-7.4.10, Nodegroup: 1)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @10.0.0.200 (mysql-5.6.28 ndb-7.4.10)
[mysqld(API)] 2 node(s)
id=50 @10.0.0.210 (mysql-5.6.28 ndb-7.4.10)
id=51 @10.0.0.211 (mysql-5.6.28 ndb-7.4.10)