How to Installing Percona XtraDB Cluster from Binary Tarball ver 5.7

因為需要重建手冊的關係,版本又已經從 5.7.18 的時期長大到 5.7.35 了

懶得處理 yum / apt 套件管理更新,所以就直接從 Binary Tarball 處理了

以順序論,建議 Percona XtraBackup 2.4 -> Percona XtraDB Cluster 5.7

具體你會先得到原因知道為什麼上面順序需要這樣安排

Error: Package: Percona-XtraDB-Cluster-server-57-5.7.18-29.20.1.el7.x86_64 (/Percona-XtraDB-Cluster-server-57-5.7.18-29.20.1.el7.x86_64)
           Requires: percona-xtrabackup-24 >= 2.4.4
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

所以從 here 把相關套件帶回來並安裝起來

# wget https://downloads.percona.com/downloads/Percona-XtraBackup-2.4/Percona-XtraBackup-2.4.4/binary/redhat/7/x86_64/Percona-XtraBackup-2.4.4-rdf58cf2-el7-x86_64-bundle.tar
# tar -xvf Percona-XtraBackup-2.4.4-rdf58cf2-el7-x86_64-bundle.tar
percona-xtrabackup-24-2.4.4-1.el7.x86_64.rpm
percona-xtrabackup-24-debuginfo-2.4.4-1.el7.x86_64.rpm
percona-xtrabackup-test-24-2.4.4-1.el7.x86_64.rpm

# yum install percona-xtrabackup-24-2.4.4-1.el7.x86_64.rpm

Installed:
  percona-xtrabackup-24.x86_64 0:2.4.4-1.el7

Dependency Installed:
  libev.x86_64 0:4.15-7.el7               mariadb-libs.x86_64 1:5.5.68-1.el7     perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7     perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7
  perl-DBD-MySQL.x86_64 0:4.023-6.el7     perl-DBI.x86_64 0:1.627-4.el7          perl-IO-Compress.noarch 0:2.061-2.el7            perl-Net-Daemon.noarch 0:0.48-5.el7
  perl-PlRPC.noarch 0:0.2020-14.el7

Complete!

接著把 bundle 從官網帶回來,打開安裝起來便是

# wget https://downloads.percona.com/downloads/Percona-XtraDB-Cluster-57/Percona-XtraDB-Cluster-5.7.18-29.20/binary/redhat/7/x86_64/Percona-XtraDB-Cluster-5.7.18-29.20-r346-el7-x86_64-bundle.tar

# tar -xvf Percona-XtraDB-Cluster-5.7.18-29.20-r346-el7-x86_64-bundle.tar
Percona-XtraDB-Cluster-57-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-57-debuginfo-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-client-57-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-devel-57-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-full-57-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-garbd-57-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-server-57-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-shared-57-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-shared-compat-57-5.7.18-29.20.1.el7.x86_64.rpm
Percona-XtraDB-Cluster-test-57-5.7.18-29.20.1.el7.x86_64.rpm

# yum install *.rpm

接下來就是 database initial 的作業就先不贅述了

作業程序官網都有說了,我也覺得很有幫助就是了

How Big Can Your Galera with MariaDB Transactions Be

先說結論 Ref: wsrep_max_ws_size 官方文件寫得死死了 ; 看到 permitted the maximum to be set beyond 2GB, which was rejected by Galera.
就又再更心死了一次

2147483647 (2GB, >= MariaDB Galera 10.0.27, MariaDB 10.1.17)
1073741824 (1GB, <= MariaDB Galera 10.0.26, MariaDB 10.1.16)

為什麼這次要研究這個問題 ; 因為我們需要討論的是一大包 Transactions 的合理性,

改不改得動是商務邏輯的事,但從架構面能不能解決 ERROR 1180 看起來還是有點辦法的

MariaDB [test]> COMMIT;
ERROR 1180 (HY000): Got error 5 "Input/output error" during COMMIT

基於 Percona 對 MySQL 做的測試在這可以參考,我們這次的測試標的基本資訊如下

[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
[root@localhost ~]# mysql -e "\s" | grep -i "Server version"
Server version:         10.4.15-MariaDB-log MariaDB Server
[root@localhost ~]# mysql -e "show status like '%wsrep%'" | grep 'wsrep_local_state_comment\|wsrep_cluster_status\|wsrep_ready'
wsrep_local_state_comment       Synced
wsrep_cluster_status    Primary
wsrep_ready     ON
[root@localhost ~]# mysql -e "select @@wsrep_max_ws_size" # 因為需要容易觸發的條件,所以就把設定最小值帶上
+---------------------+
| @@wsrep_max_ws_size |
+---------------------+
|                1024 |
+---------------------+
[root@localhost ~]# mysql -e "select @@binlog_row_image"
+--------------------+
| @@binlog_row_image |
+--------------------+
| FULL               |
+--------------------+

測試經過語法 & schema 如下

CREATE TABLE t (
    a bigint primary key DEFAULT UUID_SHORT(),
    b text,
    d text default "123",
    e text default "123",
    f text default "123",
    g text default "123"
);

SET autocommit=0;
START TRANSACTION;
-- binlog_row_image = FULL 的情況下 b 僅能指定 300 個字元,再上去就 ERROR 1180
UPDATE t SET b='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
COMMIT;

-- 錯誤訊息就不另贅述了
MariaDB [test]> COMMIT;
ERROR 1180 (HY000): Got error 5 "Input/output error" during COMMIT

此時 COMMIT 成功的 binlog 長這樣,不相關的欄位也被一起帶進來了,所以合理推測 Transactions 內容也會把其他 column 帶入,如果是 blob / text 欄位的話

那踩到 wsrep_max_ws_size 上限也就合理了

#220220 20:19:55 server id 1  end_log_pos 4434 CRC32 0x68307979         Update_rows: table id 25 flags: STMT_END_F

BINLOG '
6zESYhMBAAAANgAAANAOAAAAABkAAAAAAAEABHRlc3QAAXQABgj8/Pz8/AUCAgICAj7xstL3
6zESYhgBAAAAggIAAFIRAAAAABkAAAAAAAEABv//wAEAANAREmIB9gB4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHgDADEyMwMAMTIzAwAxMjMDADEyM8ABAADQERJiASwBeHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4AwAxMjMDADEy
MwMAMTIzAwAxMjN5eTBo
'/*!*/;
### UPDATE `test`.`t`
### WHERE
###   @1=99662009468977153
###   @2='123'
###   @3='123'
###   @4='123'
###   @5='123'
###   @6='123'
### SET
###   @1=99662009468977153
###   @2='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
###   @3='123'
###   @4='123'
###   @5='123'
###   @6='123'
# Number of rows: 1
# at 4434
#220220 20:19:55 server id 1  end_log_pos 4465 CRC32 0x6dd7ab45         Xid = 11
COMMIT/*!*/;
# at 4465
#220220 20:22:56 server id 1  end_log_pos 4488 CRC32 0x8b4370c7         Stop
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

所以應該用 binlog_row_image=minimal 解

[root@localhost ~]# mysql -e "select @@wsrep_max_ws_size"
+---------------------+
| @@wsrep_max_ws_size |
+---------------------+
|                1024 |
+---------------------+
[root@localhost ~]# mysql -e "select @@binlog_row_image"
+--------------------+
| @@binlog_row_image |
+--------------------+
| MINIMAL            |
+--------------------+

[root@localhost ~]# mysql test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 22
Server version: 10.4.15-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [test]> SET autocommit=0;
Query OK, 0 rows affected (0.001 sec)

MariaDB [test]> START TRANSACTION;
Query OK, 0 rows affected (0.000 sec)

-- binlog_row_image = MINIMAL 的情況下 b 僅能指定 448 個字元,再上去就 ERROR 1180

MariaDB [test]> UPDATE t SET b='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
Query OK, 0 rows affected (0.001 sec)
Rows matched: 1  Changed: 0  Warnings: 0

MariaDB [test]> COMMIT;
Query OK, 0 rows affected (0.000 sec)

此時 binlog 狀態長這樣

#220220 20:49:28 server id 1  end_log_pos 2106 CRC32 0x118f42b9         Update_rows: table id 25 flags: STMT_END_F

BINLOG '
2DgSYhMBAAAANgAAAEwGAAAAABkAAAAAAAEABHRlc3QAAXQABgj8/Pz8/AUCAgICAj5x9VP7
2DgSYhgBAAAA7gEAADoIAAAAABkAAAAAAAEABgEC/gAAAMM3EmIB/sABeHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4
eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eLlCjxE=
'/*!*/;
### UPDATE `test`.`t`
### WHERE
###   @1=99662172459630592
### SET
###   @2='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
# Number of rows: 1
# at 2106
#220220 20:49:28 server id 1  end_log_pos 2137 CRC32 0x3ffe0ba5         Xid = 27
COMMIT/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

Transactions 內容是多了點沒錯,但原則上還是應急用考量

畢竟 binlog_row_image 不在 FULL 條件下的 Point-in-time recovery (PITR) 也不好實作

畢竟一個 2GB 的 Transactions 種種考量下要 roll back 也不容易。

所以還是建議拆細 Transactions 粒度才是最佳解。

[DBA] How to fix the read lock generated by the restored database of mariabackup?

維運案例分享

OS: CentOS 7u9
DB: MariaDB 10.4

這世界上太多教學告訴你資料庫怎麼還原,Replication 怎麼建立 EX: here

但從來不會有人知道為什麼 mariabackup 打包後的資料庫還原建完 replication 後,
Slave IO Thread 正常,但 Slave SQL Thread 永遠沒辦法正常消化,
放著等也就 ERROR 1205 (HY000): Lock wait timeout exceeded 了

這次遇到的狀況是如此,程序都正常執行著

- mariabackup ->
- move backup ->
- mariabackup --prepare ->
- mariabackup --copy-back ->
- service started ->
- change master for config replication ->
- start slave ->
- replication delay

在說明檢查程序前,我們先了解一下 XA Transactions

在 ERROR 1205 的提醒下,我們可以找到 trx_mysql_thread_id = 0 的 dead lock

MariaDB [information_schema]> select * from INNODB_trx\G
*************************** 1. row ***************************
trx_id: 154142686
trx_state: RUNNING
trx_started: 2022-01-05 13:52:26
trx_requested_lock_id: NULL
trx_wait_started: NULL
trx_weight: 2
trx_mysql_thread_id: 0
trx_query: NULL
trx_operation_state: NULL
trx_tables_in_use: 0
trx_tables_locked: 1
trx_lock_structs: 1
trx_lock_memory_bytes: 1128
trx_rows_locked: 0
trx_rows_modified: 1
trx_concurrency_tickets: 0
trx_isolation_level: REPEATABLE READ

接著把資料找出來

- mariabackup ->
MariaDB [information_schema]> xa recover format='SQL';
+----------+--------------+--------------+---------------------------------------------------------------------+
| formatID | gtrid_length | bqual_length | data                                                                |
+----------+--------------+--------------+---------------------------------------------------------------------+
|        1 |           32 |            0 | X'5753524550586965bddf8a8f51ef11eca1507e79643313a4e9db950400000000' |
+----------+--------------+--------------+---------------------------------------------------------------------+
1 row in set (0.000 sec)

Rollback 處理後,再觀察一次 Replication 的消化狀況

- mariabackup ->
MariaDB [information_schema]> xa rollback X'5753524550586965bddf8a8f51ef11eca1507e79643313a4e9db950400000000';
Query OK, 0 rows affected (0.058 sec)

總結:

我認為這狀況不常見,也不大合理

因為 mariabackup 原則上在 –prepare 下會把該做的 redo / undo 處理告一段落才是,不應該有還沒處理好的 transaction 遺留

所以也只能推測 mariabackup 的鍋

但我沒有證據

Ref: https://www.cnblogs.com/gjc592/p/11240811.html

[DBA] What happens when the server-id is repeated?

維運案例分享

OS: CentOS 7u9
DB: MariaDB 10.2

架構需求:

倚賴 MariaDB 10.2 的 multi channel replication 收容 MariaDB 5.X 的舊版資料 ; 進行後續 Migrate 處理

開始正題:

multi channel replication 很好用,但另一層面,當 server-id 變成多對一的時候,要面對的問題層次也高了些

1. 如果 MariaDB 10.2 (Master) -> MariaDB 10.2 (Slave)

如果 gtid-domain-id 從 Master 端就開始相同(重複) 時,Slave 在 gtid_executed 就會有點為難了

不同 channel 帶進來的 gtid_executed 會往前往後跑,導致 Slave Thread 在資料處理層面辨識錯誤
有不小概率會碰上 ERROR 1950: ER_GTID_STRICT_OUT_OF_ORDER

那就關閉 set gtid_strict_mode = off; 試試看吧

2. 如果 MariaDB 5.X (Master) -> MariaDB 10.2 (Slave)

這個時候我們不用擔心 gtid-domain-id 重複,因為還不支援的情況下 replication 只能辨認 server-id

但在這個 issue 下,我們可以先在 mysqld.log 看到提示的 Log

The slave I/O thread stops because m
aster and slave have equal MySQL server ids; these ids must be different for rep
lication to work (or the --replicate-same-server-id option must be used on slave but this does not always make sense; please check the manual before using it).

麻煩的事 Slave IO Thread 沒打算停,頻繁的 retry 只會造成 relay log 大量產生到無法正常消化的結果 ; EX:

-rw-rw----   1 mysql  mysql      1580 Jan 30 23:31 mysql-relay-bin.586501
-rw-rw----   1 mysql  mysql      2551 Jan 30 23:31 mysql-relay-bin.586502
-rw-rw----   1 mysql  mysql      5154 Jan 30 23:31 mysql-relay-bin.586503
-rw-rw----   1 mysql  mysql      1548 Jan 30 23:31 mysql-relay-bin.586504
-rw-rw----   1 mysql  mysql      8699 Jan 30 23:31 mysql-relay-bin.586505
-rw-rw----   1 mysql  mysql       710 Jan 30 23:31 mysql-relay-bin.586506
-rw-rw----   1 mysql  mysql      3665 Jan 30 23:31 mysql-relay-bin.586507
-rw-rw----   1 mysql  mysql      1846 Jan 30 23:31 mysql-relay-bin.586508
-rw-rw----   1 mysql  mysql      7486 Jan 30 23:31 mysql-relay-bin.586509
-rw-rw----   1 mysql  mysql      2624 Jan 30 23:31 mysql-relay-bin.586510
-rw-rw----   1 mysql  mysql      1195 Jan 30 23:31 mysql-relay-bin.586511
-rw-rw----   1 mysql  mysql     12683 Jan 30 23:31 mysql-relay-bin.586512
-rw-rw----   1 mysql  mysql      6765 Jan 30 23:31 mysql-relay-bin.586513
-rw-rw----   1 mysql  mysql      5477 Jan 30 23:32 mysql-relay-bin.586514
-rw-rw----   1 mysql  mysql      1613 Jan 30 23:32 mysql-relay-bin.586515
-rw-rw----   1 mysql  mysql      1373 Jan 30 23:32 mysql-relay-bin.586516
-rw-rw----   1 mysql  mysql     10729 Jan 30 23:32 mysql-relay-bin.586517
-rw-rw----   1 mysql  mysql       473 Jan 30 23:32 mysql-relay-bin.586518
-rw-rw----   1 mysql  mysql      1159 Jan 30 23:32 mysql-relay-bin.586519
-rw-rw----   1 mysql  mysql      5469 Jan 30 23:32 mysql-relay-bin.586520
-rw-rw----   1 mysql  mysql     12258 Jan 30 23:32 mysql-relay-bin.586521

維運紀錄上 relay log 長到 mysql-relay-bin.999999 後,這條 replication 也就沒救了

所以建立 replication 後,或修正 server-id 後,還是適度關懷一下 binlog / relay log 的成長消化狀況也合適些