2023-02-03 10:13:37 0 [Note] WSREP: (030c9e6c-992a, 'ssl://172.18.9.41:4567') connection established to 68e01be8-bed3 ssl://172.18.9.42:4567 2023-02-03 10:13:37 0 [Note] WSREP: (030c9e6c-992a, 'ssl://172.18.9.41:4567') turning message relay requesting on, nonlive peers: 2023-02-03 10:13:38 0 [Note] WSREP: declaring 68e01be8-bed3 at ssl://172.18.9.42:4567 stable 2023-02-03 10:13:38 0 [Note] WSREP: declaring abacacf9-8276 at ssl://10.130.5.65:4567 stable 2023-02-03 10:13:38 0 [Note] WSREP: declaring ac5379a6-8a17 at ssl://10.130.5.66:4567 stable 2023-02-03 10:13:38 0 [Note] WSREP: Node 030c9e6c-992a state prim 2023-02-03 10:13:38 0 [Note] WSREP: view(view_id(PRIM,030c9e6c-992a,2784) memb { 030c9e6c-992a,0 68e01be8-bed3,0 abacacf9-8276,0 ac5379a6-8a17,0 } joined { } left { } partitioned { }) 2023-02-03 10:13:38 0 [Note] WSREP: save pc into disk 2023-02-03 10:13:38 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 4 2023-02-03 10:13:38 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0b60af7e-a3a3-11ed-82b7-73037c3089a1 2023-02-03 10:13:38 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0b60af7e-a3a3-11ed-82b7-73037c3089a1 2023-02-03 10:13:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0b60af7e-a3a3-11ed-82b7-73037c3089a1 from 0 (maria2021-ham-1) 2023-02-03 10:13:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0b60af7e-a3a3-11ed-82b7-73037c3089a1 from 1 (maria2021-ham-2) 2023-02-03 10:13:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0b60af7e-a3a3-11ed-82b7-73037c3089a1 from 2 (maria2021-muc-1) 2023-02-03 10:13:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0b60af7e-a3a3-11ed-82b7-73037c3089a1 from 3 (maria2021-muc-2) 2023-02-03 10:13:38 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 84, members = 3/4 (joined/total), act_id = 7107702, last_appl. = 7107663, protocols = 2/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 0f364b85-ba40-11eb-b030-031c949c7f83 2023-02-03 10:13:38 0 [Note] WSREP: Flow-control interval: [32, 32] 2023-02-03 10:13:38 2 [Note] WSREP: ####### processing CC 7107703, local, ordered 2023-02-03 10:13:38 2 [Note] WSREP: ####### My UUID: 030c9e6c-a303-11ed-992a-87217ec36a24 2023-02-03 10:13:38 2 [Note] WSREP: Skipping cert index reset 2023-02-03 10:13:38 2 [Note] WSREP: REPL Protocols: 10 (5) 2023-02-03 10:13:38 2 [Note] WSREP: ####### Adjusting cert position: 7107702 -> 7107703 2023-02-03 10:13:38 0 [Note] WSREP: Service thread queue flushed. 2023-02-03 10:13:38 2 [Note] WSREP: ================================================ View: id: 0f364b85-ba40-11eb-b030-031c949c7f83:7107703 status: primary protocol_version: 4 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO final: no own_index: 0 members(4): 0: 030c9e6c-a303-11ed-992a-87217ec36a24, maria2021-ham-1 1: 68e01be8-97f8-11ed-bed3-fbadea276b27, maria2021-ham-2 2: abacacf9-a2f2-11ed-8276-93037b207101, maria2021-muc-1 3: ac5379a6-a2f2-11ed-8a17-6a541283a18e, maria2021-muc-2 ================================================= 2023-02-03 10:13:38 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-02-03 10:13:38 2 [Note] WSREP: Lowest cert index boundary for CC from group: 7107664 2023-02-03 10:13:38 2 [Note] WSREP: Min available from gcache for CC from group: 7097225 2023-02-03 10:13:38 0 [Note] WSREP: Member 1.0 (maria2021-ham-2) requested state transfer from '*any*'. Selected 0.0 (maria2021-ham-1)(SYNCED) as donor. 2023-02-03 10:13:38 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 7107703) 2023-02-03 10:13:38 2 [Note] WSREP: Detected STR version: 1, req_len: 132, req: STRv1 2023-02-03 10:13:38 2 [Note] WSREP: Cert index preload: 7107664 -> 7107703 2023-02-03 10:13:38 2 [Note] WSREP: Server status change synced -> donor 2023-02-03 10:13:38 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-02-03 10:13:38 0 [Note] WSREP: async IST sender starting to serve ssl://172.18.9.42:4568 sending 7107664-7107703, preload starts from 7107664 2023-02-03 10:13:38 0 [Note] WSREP: Donor monitor thread started to monitor 2023-02-03 10:13:38 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '172.18.9.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/run/mysqld/mysqld.sock' --progress 0 --datadir '/GALERANFS_HAM_DATA/data/mariadb/' --gtid '0f364b85-ba40-11eb-b030-031c949c7f83:7107703' --gtid-domain-id 0 --mysqld-args --wsrep_start_position=0f364b85-ba40-11eb-b030-031c949c7f83:7091599' 2023-02-03 10:13:38 0 [Note] WSREP: IST sender 7107664 -> 7107703 2023-02-03 10:13:38 2 [Note] WSREP: sst_donor_thread signaled with 0 WSREP_SST: [INFO] mariabackup SST started on donor (20230203 10:13:39.002) WSREP_SST: [INFO] SSL configuration: CA='/etc/mysql/cert/ca-cert.pem', CAPATH='', CERT='/etc/mysql/cert/maria2021-ham-1.rz.hsu-hh-cert.pem', KEY='/etc/mysql/priv/maria2021-ham-1.rz.hsu-hh-key.pem', MODE='DISABLED', encrypt='4' (20230203 10:13:39.137) 2023-02-03 10:13:39 0 [Note] WSREP: async IST sender served WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230203 10:13:39.342) WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230203 10:13:39.346) WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20230203 10:13:39.368) Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735263]: Streaming with mbstream Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735264]: Using socat as streamer Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735273]: Using openssl based encryption with socat: with key and crt Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735281]: Encrypting with cert='/etc/mysql/cert/maria2021-ham-1.rz.hsu-hh-cert.pem', key='/etc/mysql/priv/maria2021-ham-1.rz.hsu-hh-key.pem', ca='/etc/mysql/cert/ca-cert.pem', capath='' Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735288]: Using '/dbtmp/tmp.zbEfzwsUgV' as mariabackup temporary directory Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735290]: Using '/tmp/tmp.75RolhK1sv' as mariabackup working directory Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735296]: SST footprint estimate: data: 13390180352, undo: 0 Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735297]: To transfer: 13390180352 Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735298]: Streaming GTID file before SST Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735299]: Evaluating '/usr//bin/mbstream' -c 'xtrabackup_galera_info' | socat stdio openssl-connect:172.18.9.42:4444,linger=10,cert='/etc/mysql/cert/maria2021-ham-1.rz.hsu-hh-cert.pem',key='/etc/mysql/priv/maria2021-ham-1.rz.hsu-hh-key.pem',cafile='/etc/mysql/cert/ca-cert.pem',commonname=''; RC=( ${PIPESTATUS[@]} ) Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735261]: 2023/02/03 10:13:39 socat[3735301] E connect(7, AF=2 172.18.9.42:4444, 16): Connection refused Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735302]: Error while sending data to joiner node: exit codes: 0 1 Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735303]: Cleanup after exit with status: 32 Feb 03 10:13:39 maria2021-ham-1 -wsrep-sst-donor[3735305]: Removing /GALERANFS_HAM_DATA/data/mariadb/xtrabackup_galera_info file due to signal 2023-02-03 10:13:39 0 [ERROR] WSREP: Failed to read from: wsrep_sst_mariabackup --role 'donor' --address '172.18.9.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/run/mysqld/mysqld.sock' --progress 0 --datadir '/GALERANFS_HAM_DATA/data/mariadb/' --gtid '0f364b85-ba40-11eb-b030-031c949c7f83:7107703' --gtid-domain-id 0 --mysqld-args --wsrep_start_position=0f364b85-ba40-11eb-b030-031c949c7f83:7091599 2023-02-03 10:13:39 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'donor' --address '172.18.9.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/run/mysqld/mysqld.sock' --progress 0 --datadir '/GALERANFS_HAM_DATA/data/mariadb/' --gtid '0f364b85-ba40-11eb-b030-031c949c7f83:7107703' --gtid-domain-id 0 --mysqld-args --wsrep_start_position=0f364b85-ba40-11eb-b030-031c949c7f83:7091599: 32 (Broken pipe) 2023-02-03 10:13:39 0 [Note] WSREP: SST sending failed: -32 2023-02-03 10:13:39 0 [Note] WSREP: Server status change donor -> joined 2023-02-03 10:13:39 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-02-03 10:13:39 0 [ERROR] WSREP: Command did not run: wsrep_sst_mariabackup --role 'donor' --address '172.18.9.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/run/mysqld/mysqld.sock' --progress 0 --datadir '/GALERANFS_HAM_DATA/data/mariadb/' --gtid '0f364b85-ba40-11eb-b030-031c949c7f83:7107703' --gtid-domain-id 0 --mysqld-args --wsrep_start_position=0f364b85-ba40-11eb-b030-031c949c7f83:7091599 2023-02-03 10:13:39 0 [Note] WSREP: Donor monitor thread ended with total time 1 sec 2023-02-03 10:13:40 0 [Note] WSREP: (030c9e6c-992a, 'ssl://172.18.9.41:4567') reconnecting to 68e01be8-bed3 (ssl://172.18.9.42:4567), attempt 0 2023-02-03 10:13:44 0 [Note] WSREP: evs::proto(030c9e6c-992a, OPERATIONAL, view_id(REG,030c9e6c-992a,2784)) suspecting node: 68e01be8-bed3 2023-02-03 10:13:44 0 [Note] WSREP: evs::proto(030c9e6c-992a, OPERATIONAL, view_id(REG,030c9e6c-992a,2784)) suspected node without join message, declaring inactive 2023-02-03 10:13:45 0 [Warning] WSREP: 0.0 (maria2021-ham-1): State transfer to 1.0 (maria2021-ham-2) failed: -32 (Broken pipe) 2023-02-03 10:13:45 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 7107703) 2023-02-03 10:13:45 0 [Note] WSREP: Processing event queue:... -nan% (0/0 events) complete. 2023-02-03 10:13:45 0 [Note] WSREP: declaring abacacf9-8276 at ssl://10.130.5.65:4567 stable 2023-02-03 10:13:45 0 [Note] WSREP: declaring ac5379a6-8a17 at ssl://10.130.5.66:4567 stable 2023-02-03 10:13:45 0 [Note] WSREP: Node 030c9e6c-992a state prim 2023-02-03 10:13:45 0 [Note] WSREP: view(view_id(PRIM,030c9e6c-992a,2785) memb { 030c9e6c-992a,0 abacacf9-8276,0 ac5379a6-8a17,0 } joined { } left { } partitioned { 68e01be8-bed3,0 }) 2023-02-03 10:13:45 0 [Note] WSREP: save pc into disk 2023-02-03 10:13:45 0 [Note] WSREP: forgetting 68e01be8-bed3 (ssl://172.18.9.42:4567) 2023-02-03 10:13:45 0 [Note] WSREP: (030c9e6c-992a, 'ssl://172.18.9.41:4567') turning message relay requesting off 2023-02-03 10:13:45 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2023-02-03 10:13:45 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0f8e6b6a-a3a3-11ed-9fa6-d715642bf4dc 2023-02-03 10:13:45 0 [Warning] WSREP: SYNC message from member 0 in non-primary configuration. Ignored. 2023-02-03 10:13:45 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0f8e6b6a-a3a3-11ed-9fa6-d715642bf4dc 2023-02-03 10:13:45 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f8e6b6a-a3a3-11ed-9fa6-d715642bf4dc from 0 (maria2021-ham-1) 2023-02-03 10:13:45 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f8e6b6a-a3a3-11ed-9fa6-d715642bf4dc from 1 (maria2021-muc-1) 2023-02-03 10:13:45 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f8e6b6a-a3a3-11ed-9fa6-d715642bf4dc from 2 (maria2021-muc-2) 2023-02-03 10:13:45 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 85, members = 3/3 (joined/total), act_id = 7107703, last_appl. = 7107663, protocols = 2/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 0f364b85-ba40-11eb-b030-031c949c7f83 2023-02-03 10:13:45 0 [Note] WSREP: Flow-control interval: [28, 28] 2023-02-03 10:13:45 2 [Note] WSREP: ####### processing CC 7107704, local, ordered 2023-02-03 10:13:45 2 [Note] WSREP: ####### My UUID: 030c9e6c-a303-11ed-992a-87217ec36a24 2023-02-03 10:13:45 2 [Note] WSREP: Skipping cert index reset 2023-02-03 10:13:45 2 [Note] WSREP: REPL Protocols: 10 (5) 2023-02-03 10:13:45 2 [Note] WSREP: ####### Adjusting cert position: 7107703 -> 7107704 2023-02-03 10:13:45 0 [Note] WSREP: Service thread queue flushed. 2023-02-03 10:13:45 0 [Note] WSREP: Member 0.0 (maria2021-ham-1) synced with group. 2023-02-03 10:13:45 0 [Note] WSREP: Processing event queue:...100.0% (2/2 events) complete. 2023-02-03 10:13:45 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 7107704) 2023-02-03 10:13:45 2 [Note] WSREP: ================================================ View: id: 0f364b85-ba40-11eb-b030-031c949c7f83:7107704 status: primary protocol_version: 4 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO final: no own_index: 0 members(3): 0: 030c9e6c-a303-11ed-992a-87217ec36a24, maria2021-ham-1 1: abacacf9-a2f2-11ed-8276-93037b207101, maria2021-muc-1 2: ac5379a6-a2f2-11ed-8a17-6a541283a18e, maria2021-muc-2 ================================================= 2023-02-03 10:13:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-02-03 10:13:45 2 [Note] WSREP: Lowest cert index boundary for CC from group: 7107664 2023-02-03 10:13:45 2 [Note] WSREP: Min available from gcache for CC from group: 7097225 2023-02-03 10:13:45 2 [Note] WSREP: Server maria2021-ham-1 synced with group 2023-02-03 10:13:45 2 [Note] WSREP: Server status change joined -> synced 2023-02-03 10:13:45 2 [Note] WSREP: Synchronized with group, ready for connections 2023-02-03 10:13:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-02-03 10:13:48 0 [Note] WSREP: cleaning up 68e01be8-bed3 (ssl://172.18.9.42:4567)