web-dev-qa-db-ja.com

レプリケーションが実行されている場合のみ、percona / mariadbサーバーでの書き込みが遅い

こんにちはWindowsのmysql 5.6からperconaサーバーに移行しようとしています。私の開発マシンでは、perconaはvmware 5.5球にインストールされたcentos 6.5を実行しています。

私たちのビジネスアプリケーションはサーバー上で非常に優れたパフォーマンスを発揮しますが、循環レプリケーションを有効にすると、ビジネスアプリの両端が遅れます。書き込み(更新、削除など)が完了するのに数ミリ秒ではなく数秒かかることに気付きました(レプリケーションがオフの場合)

Windowsではこの問題は発生しません。パーコナのみ。その主な原因は書き込みとレプリケーションであるため、バイナリログとハードディスクのI/Oに関係している可能性があります。ダブルライトをオフにする、ログキャッシュファイルのサイズを試すなど、さまざまな設定を試しましたが、うまくいきませんでした。

更新:サーバーをプロファイリングしようとしても、MySQLチューニングガイドに従っています。レプリケーションがオフのときにアプリが適切に実行されるのはおかしいです。開発者に遅いクエリログを与え、それらを見るように指示しました。ただし、レプリケーションがオフの場合、これらのクエリは遅くありません。レプリケーションがオンの場合の16%に比べて、クエリログの遅い%は0%です

**​​ update 2:まったく同じインフラストラクチャの同じマシン上で、2つの新しいVMが追加され、在庫のMysql5.6 comエディションを使用してレプリケーションをテストしました。すべてが問題なく正常に動作しています。単なるperconaの問題です。Mariadb10にも同じ問題があります

**​​ update 3:perconaとmariadbでスローダウンが発生している場所を特定しようとしました。私のcnf.fileにはすべてのレプリケーション機能がオンになっています。私のビジネスアプリは正常に動作します。すべてのログファイルは更新されません問題。パフォーマンスは良好です。スレーブを起動したときにのみ、マスターサーバーのパフォーマンスが低下します。mysqlコマンドラインからスレーブを停止すると、パフォーマンスが回復します。

サーバーが循環レプリケーションを行っている間、ビジネスアプリのパフォーマンスは低下し、この状態では次のテストも行いました

sysbenchの使用

sysbench --db-driver=mysql --num-threads=16 --max-requests=10000 --test=oltp --oltp-table-size=500000 --mysql-Host="192.168.1.251" --oltp-test-mode=complex --mysql-user=admin --mysql-password=adm run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16

Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.

OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           50000
        other:                           20000
        total:                           210000
    transactions:                        10000  (1098.37 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190000 (20869.10 per sec.)
    other operations:                    20000  (2196.75 per sec.)

Test execution summary:
    total time:                          9.1044s
    total number of events:              10000
    total time taken by event execution: 145.5482
    per-request statistics:
         min:                                  4.87ms
         avg:                                 14.55ms
         max:                                197.28ms
         approx.  95 percentile:              19.67ms

Threads fairness:
    events (avg/stddev):           625.0000/12.79
    execution time (avg/stddev):   9.0968/0.00

Mysqltunerプロファイルも含めました。

[OK] Currently running supported MySQL version 5.6.17-65.0-56-log
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in InnoDB tables: 588M (Tables: 219)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 52)
[!!] Total fragmented tables: 11

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 11m 44s (1K q [1.781 qps], 368 conn, TX: 1M, RX: 274K)
[--] Reads / Writes: 87% / 13%
[--] Total buffers: 5.1G global + 1.1M per thread (200 max threads)
[OK] Maximum possible memory usage: 5.3G (68% of installed RAM)
[!!] Slow queries: 6% (82/1K)
[OK] Highest usage of available connections: 1% (3/200)
[OK] Key buffer size / total MyISAM indexes: 32.0M/97.0K
[OK] Query cache efficiency: 33.2% (130 cached / 392 selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 3% (2 temp sorts / 61 sorts)
[!!] Joins performed without indexes: 9
[OK] Temporary tables created on disk: 1% (4 on disk / 222 total)
[OK] Thread cache hit rate: 97% (9 created / 368 connections)
[OK] Table cache hit rate: 97% (379 open / 390 opened)
[OK] Open file limit used: 0% (54/65K)
[OK] Table locks acquired immediately: 100% (645 immediate / 645 locks)
[OK] InnoDB buffer pool / data size: 5.0G/588.9M
[OK] InnoDB log waits: 0
-------- Recommendations -----------------------------------------------------


# Percona Toolkit MySQL Summary Report #######################
              System time | 2014-07-11 19:43:05 UTC (local TZ: SAST +0200)
# Instances ##################################################
  Port  Data Directory             Nice OOM Socket
  ===== ========================== ==== === ======
        /var/lib/mysql/            0    0   /var/lib/mysql/mysql.sock
# MySQL Executable ###########################################
       Path to executable | /usr/sbin/mysqld
              Has symbols | No
# Report On Port 3306 ########################################
                     User | admin@localhost
                     Time | 2014-07-11 21:43:04 (SAST)
                 Hostname | mobeni-standalone
                  Version | 5.6.17-65.0-56-log Percona Server (GPL), Release 65.0, Revision 587
                 Built On | Linux x86_64
                  Started | 2014-07-11 21:26 (up 0+00:16:59)
                Databases | 6
                  Datadir | /var/lib/mysql/
                Processes | 2 connected, 2 running
              Replication | Is a slave, has 1 slaves connected
                  Pidfile | /var/lib/mysql/mysql.pid (exists)
# Processlist ################################################

  Command                        COUNT(*) Working SUM(Time) MAX(Time)
  ------------------------------ -------- ------- --------- ---------
  Binlog Dump GTID                      1       1      1000      1000
  Connect                               2       2      2000      1000
  Query                                 1       1         0         0

  User                           COUNT(*) Working SUM(Time) MAX(Time)
  ------------------------------ -------- ------- --------- ---------
  admin                                 2       2      1000      1000
  system user                           2       2      2000      1000

  Host                           COUNT(*) Working SUM(Time) MAX(Time)
  ------------------------------ -------- ------- --------- ---------
  10.2.0.2                              1       1      1000      1000
                                        2       2      2000      1000
  localhost                             1       1         0         0

  db                             COUNT(*) Working SUM(Time) MAX(Time)
  ------------------------------ -------- ------- --------- ---------
  NULL                                  4       4      3000      1000

  State                          COUNT(*) Working SUM(Time) MAX(Time)
  ------------------------------ -------- ------- --------- ---------
  init                                  1       1         0         0
  Master has sent all binlog to         1       1      1000      1000
  Slave has read all relay log;         1       1      1000      1000
  Waiting for master to send eve        1       1      1000      1000

# Status Counters (Wait 10 Seconds) ##########################
Variable                                Per day  Per second     16 secs
Binlog_cache_use                           1000
Bytes_received                         30000000         350         225
Bytes_sent                            100000000        1250        2000
Com_admin_commands                          600
Com_commit                                  600
Com_delete                                 1250
Com_insert                                 1500
Com_select                                30000                       1
Com_set_option                            60000
Com_show_status                             600
Com_show_storage_engines                    500
Com_show_variables                          800
Com_stmt_close                            10000
Com_stmt_execute                          10000
Com_stmt_prepare                          10000
Com_update                                 1500
Connections                               40000
Created_tmp_disk_tables                     350
Created_tmp_files                           800
Created_tmp_tables                        22500                       4
Flush_commands                               80
Handler_commit                            25000
Handler_delete                              800
Handler_external_lock                    125000           1
Handler_prepare                            7000
Handler_read_first                         7000
Handler_read_key                      225000000        2500
Handler_read_last                           500
Handler_read_next                     100000000        1250
Handler_read_prev                           800
Handler_read_rnd                        8000000         100
Handler_read_rnd_next                 150000000        1750          30
Handler_update                         10000000         125
Handler_write                          20000000         250          30
Innodb_background_log_sync                90000                       1
Innodb_buffer_pool_bytes_data       35000000000      400000
Innodb_buffer_pool_pages_flushed          45000
Innodb_buffer_pool_pages_made_young        1500
Innodb_buffer_pool_pages_old             800000           8
Innodb_buffer_pool_read_ahead            800000           8
Innodb_buffer_pool_read_requests      900000000       10000
Innodb_buffer_pool_reads                1250000          15
Innodb_buffer_pool_write_requests        100000           1
Innodb_checkpoint_max_age           35000000000      450000
Innodb_data_fsyncs                        12500
Innodb_data_read                    35000000000      400000
Innodb_data_reads                       2000000          25
Innodb_data_writes                        50000
Innodb_data_written                  1500000000       17500
Innodb_dblwr_pages_written                45000
Innodb_dblwr_writes                        1250
Innodb_history_list_length                50000
Innodb_ibuf_free_list                       800
Innodb_ibuf_merged_delete_marks            1250
Innodb_ibuf_merged_inserts                  800
Innodb_ibuf_merges                         2000
Innodb_ibuf_segment_size                    900
Innodb_ibuf_size                             80
Innodb_log_write_requests                 80000
Innodb_log_writes                          3500
Innodb_lsn_current                  1750000000000    20000000
Innodb_lsn_flushed                  1750000000000    20000000
Innodb_lsn_last_checkpoint          1750000000000    20000000
Innodb_master_thread_active_loops         12500
Innodb_master_thread_idle_loops           70000                       1
Innodb_max_trx_id                     700000000        8000
Innodb_mem_adaptive_hash            10000000000      125000
Innodb_mem_dictionary                2000000000       22500
Innodb_mem_total                    450000000000     5000000
Innodb_mutex_os_waits                       900
Innodb_mutex_spin_rounds                  50000
Innodb_mutex_spin_waits                   25000
Innodb_os_log_fsyncs                       4500
Innodb_os_log_written                  22500000         250
Innodb_pages_created                       2000
Innodb_pages_read                       2000000          25
Innodb_pages_written                      45000
Innodb_purge_trx_id                   700000000        8000
Innodb_rows_deleted                         800
Innodb_rows_inserted                       1500
Innodb_rows_read                      400000000        5000
Innodb_rows_updated                        1000
Innodb_num_open_files                     20000
Innodb_read_views_memory                  15000
Innodb_descriptors_memory                700000           7
Innodb_s_lock_os_waits                     9000
Innodb_s_lock_spin_rounds                250000           3
Innodb_s_lock_spin_waits                   9000
Innodb_available_undo_logs                10000
Innodb_x_lock_os_waits                       80
Innodb_x_lock_spin_rounds                  4000
Innodb_x_lock_spin_waits                     80
Open_table_definitions                    25000
Opened_files                              40000                       3
Opened_table_definitions                  25000
Opened_tables                             35000
Qcache_hits                               12500
Qcache_inserts                            15000
Qcache_not_cached                         15000                       1
Queries                                  175000           2           2
Questions                                150000           1           2
Select_full_join                            800
Select_range                                600
Select_scan                               10000
Slave_received_heartbeats                    80
Slow_queries                               8000
Sort_merge_passes                           175
Sort_range                                  500
Sort_rows                               8000000         100
Sort_scan                                  5000
Table_locks_immediate                     60000
Table_open_cache_hits                    500000           6
Table_open_cache_misses                   35000
Threadpool_idle_threads                     250
Threadpool_threads                          400
Threads_created                             900
Uptime                                    90000           1           1
# Table cache ################################################
                     Size | 4096
                    Usage | 9%
# Key Percona Server features ################################
      Table & Index Stats | Disabled
     Multiple I/O Threads | Enabled
     Corruption Resilient | Enabled
      Durable Replication | Not Supported
     Import InnoDB Tables | Not Supported
     Fast Server Restarts | Not Supported
         Enhanced Logging | Disabled
     Replica Perf Logging | Disabled
      Response Time Hist. | Not Supported
          Smooth Flushing | Not Supported
      HandlerSocket NoSQL | Not Supported
           Fast Hash UDFs | Unknown
# Percona XtraDB Cluster #####################################
# Plugins ####################################################
       InnoDB compression | ACTIVE
# Query cache ################################################
         query_cache_type | ON
                     Size | 1.0M
                    Usage | 40%
         HitToInsertRatio | 100%
# Schema #####################################################
Specify --databases or --all-databases to dump and summarize schemas
# Noteworthy Technologies ####################################
                      SSL | No
     Explicit LOCK TABLES | No
           Delayed Insert | No
          XA Transactions | No
              NDB Cluster | No
      Prepared Statements | Yes
 Prepared statement count | 0
# InnoDB #####################################################
                  Version | 5.6.17-65.0
         Buffer Pool Size | 5.0G
         Buffer Pool Fill | 8%
        Buffer Pool Dirty | 0%
           File Per Table | ON
                Page Size | 16k
            Log File Size | 2 * 256.0M = 512.0M
          Log Buffer Size | 8M
             Flush Method | O_DIRECT
      Flush Log At Commit | 1
               XA Support | ON
                Checksums | ON
              Doublewrite | ON
          R/W I/O Threads | 4 4
             I/O Capacity | 200
       Thread Concurrency | 0
      Concurrency Tickets | 5000
       Commit Concurrency | 0
      Txn Isolation Level | REPEATABLE-READ
        Adaptive Flushing | ON
      Adaptive Checkpoint |
           Checkpoint Age | 0
             InnoDB Queue | 0 queries inside InnoDB, 0 queries in queue
       Oldest Transaction | 0 Seconds
         History List Len | 609
609
               Read Views | 0
         Undo Log Entries | 0 transactions, 0 total undo, 0 max undo
        Pending I/O Reads | 0 buf pool reads, 0 normal AIO, 0 ibuf AIO, 0 preads
       Pending I/O Writes | 0 buf pool (0 LRU, 0 flush list, 0 page); 0 AIO, 0 sync, 0 log IO (0 log, 0 chkp); 0 pwrites
      Pending I/O Flushes | 0 buf pool, 0 log
       Transaction States | 2xnot started
# MyISAM #####################################################
                Key Cache | 32.0M
                 Pct Used | 20%
                Unflushed | 0%
# Security ###################################################
                    Users | 8 users, 0 anon, 0 w/o pw, 0 old pw
            Old Passwords | 0
# Binary Logging #############################################
                  Binlogs | 9
               Zero-Sized | 0
               Total Size | 71.2M
            binlog_format | ROW
         expire_logs_days | 7
              sync_binlog | 1
                server_id | 1
             binlog_do_db |
         binlog_ignore_db |
# Noteworthy Variables #######################################
     Auto-Inc Incr/Offset | 10/1
   default_storage_engine | InnoDB
               flush_time | 0
             init_connect |
                init_file |
                 sql_mode | NO_AUTO_VALUE_ON_ZERO,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
         join_buffer_size | 256k
         sort_buffer_size | 256k
         read_buffer_size | 128k
     read_rnd_buffer_size | 256k
       bulk_insert_buffer | 0.00
      max_heap_table_size | 32M
           tmp_table_size | 32M
       max_allowed_packet | 16M
             thread_stack | 256k
                      log |
                log_error | /var/log/mysqld.log
             log_warnings | 1
         log_slow_queries |
log_queries_not_using_indexes | ON
        log_slave_updates | ON
# Configuration File #########################################
              Config File | /etc/my.cnf

[mysql]
port                                = 3306
socket                              = /var/lib/mysql/mysql.sock

[mysqld]
user                                = mysql
default-storage-engine              = InnoDB
socket                              = /var/lib/mysql/mysql.sock
pid-file                            = /var/lib/mysql/mysql.pid
lower-case-table-names              = 1
key-buffer-size                     = 32M
myisam-recover                      = FORCE,BACKUP
max-allowed-packet                  = 16M
max-connect-errors                  = 1000000
skip-name-resolve
sql-mode                            = STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE
innodb                              = FORCE
innodb-strict-mode                  = 1
sysdate-is-now                      = 1
thread_handling                     = pool-of-threads
thread_pool_size                    = 2
datadir                             = /var/lib/mysql/
log-bin                             = /var/lib/mysql/mysql-bin
expire-logs-days                    = 14
sync-binlog                         = 1
tmp-table-size                      = 32M
max-heap-table-size                 = 32M
query-cache-type                    = 1
query-cache-size                    = 1M
max-connections                     = 200
thread-cache-size                   = 30
open-files-limit                    = 65535
table-definition-cache              = 4096
table-open-cache                    = 4096
innodb-flush-method                 = O_DIRECT
innodb-log-files-in-group           = 2
innodb-log-file-size                = 256M
innodb-flush-log-at-trx-commit      = 1
innodb-file-per-table               = 1
innodb-buffer-pool-size             = 5G
log-error                           = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes       = 1
slow-query-log                      = 1
slow-query-log-file                 = /var/lib/mysql/mysql-slow.log
binlog_format                       = ROW
log-bin                             = binlogG09
relay-log                           = relay-binG09
expire_logs_days                    = 7
log-slave-updates                   = 1
gtid_mode                           = ON
enforce-gtid-consistency            = 1
server-id                           = 1
auto_increment_increment            = 10
auto_increment_offset               = 1

[client]
user                                = admin
password                            = "adm"

[mysqldump]
max-allowed-packet                  = 512M

[MYSQLD_SAFE]
pid-file                            = mysqld.pid
log-error                           = /var/log/mysqld.log
basedir                             = /usr/
datadir                             = /var/lib/mysql
# The End ####################################################
2
user43526

Perconaディスカッショングループで、犯人がスレッドプールであることがわかったと指摘しました。その後、Laurynas Biveinisが返信しました:7月16(7日前)

ああ。 my.cnfのthread_pool_size = 2は低すぎるようで、スレッドプールのパフォーマンスの低下を説明している可能性があります。使用する場合は、設定を調整してください http://www.percona.com/doc/percona-server/5.6/performance/threadpool.html

1
Justin Swanhart