210531 13:53:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210531 13:53:21 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210531 13:53:24 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1 2021-05-31 13:53:24 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 247 ... 2021-05-31 13:53:24 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-05-31 13:53:24 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-05-31 13:53:24 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-05-31 13:53:24 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-05-31 13:53:24 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat' 2021-05-31 13:53:24 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1 2021-05-31 13:53:24 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-05-31 13:53:24 0 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 00000000-0000-0000-0000-000000000000:-1 2021-05-31 13:53:24 0 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 2021-05-31 13:53:24 0 [Note] WSREP: wsrep_sst_grab() 2021-05-31 13:53:24 0 [Note] WSREP: Start replication 2021-05-31 13:53:24 0 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 2021-05-31 13:53:24 0 [Note] WSREP: protonet asio version 0 2021-05-31 13:53:24 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-05-31 13:53:24 0 [Note] WSREP: backend: asio 2021-05-31 13:53:24 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-05-31 13:53:24 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) 2021-05-31 13:53:24 0 [Note] WSREP: restore pc from disk failed 2021-05-31 13:53:24 0 [Note] WSREP: GMCast version 0 2021-05-31 13:53:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-05-31 13:53:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-05-31 13:53:24 0 [Note] WSREP: EVS version 0 2021-05-31 13:53:24 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-05-31 13:53:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 88467a99 tcp://10.33.0.13:4567 2021-05-31 13:53:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: 2021-05-31 13:53:24 0 [Note] WSREP: declaring 88467a99 at tcp://10.33.0.13:4567 stable 2021-05-31 13:53:24 0 [Note] WSREP: Node 88467a99 state prim 2021-05-31 13:53:24 0 [Note] WSREP: view(view_id(PRIM,88467a99,2) memb { 88467a99,0 913b953a,0 } joined { } left { } partitioned { }) 2021-05-31 13:53:24 0 [Note] WSREP: save pc into disk 2021-05-31 13:53:24 0 [Note] WSREP: discarding pending addr without UUID: tcp://10.33.3.175:4567 2021-05-31 13:53:25 0 [Note] WSREP: gcomm: connected 2021-05-31 13:53:25 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-05-31 13:53:25 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-05-31 13:53:25 0 [Note] WSREP: Opened channel 'openstack' 2021-05-31 13:53:25 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 2021-05-31 13:53:25 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2021-05-31 13:53:25 0 [Note] WSREP: Waiting for SST to complete. 2021-05-31 13:53:25 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 918a9137-c217-11eb-8e20-2b970f31ba29 2021-05-31 13:53:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: 918a9137-c217-11eb-8e20-2b970f31ba29 from 0 (os-ctrl-0) 2021-05-31 13:53:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: 918a9137-c217-11eb-8e20-2b970f31ba29 from 1 (os-ctrl-1) 2021-05-31 13:53:25 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 1, members = 1/2 (joined/total), act_id = 2, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-05-31 13:53:25 0 [Note] WSREP: Flow-control interval: [23, 23] 2021-05-31 13:53:25 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 2) 2021-05-31 13:53:25 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:2 Local state: 00000000-0000-0000-0000-000000000000:-1 2021-05-31 13:53:25 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-05-31 13:53:25 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:2, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3 2021-05-31 13:53:25 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-05-31 13:53:25 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '247' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1' 2021-05-31 13:53:25 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 WSREP_SST: [INFO] Streaming with xbstream (20210531 13:53:25.267) WSREP_SST: [INFO] Using socat as streamer (20210531 13:53:25.269) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210531 13:53:25.287) 2021-05-31 13:53:25 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-05-31 13:53:25 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 1 --members 88467a99-c217-11eb-9205-53b344cffa5b/os-ctrl-0/10.33.0.13:3306,913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306: 1 (Operation not permitted) 2021-05-31 13:53:25 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 1 --members 88467a99-c217-11eb-9205-53b344cffa5b/os-ctrl-0/10.33.0.13:3306,913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306" 2021-05-31 13:53:25 2 [Note] WSREP: Assign initial position for certification: 2, protocol version: 4 2021-05-31 13:53:25 0 [Note] WSREP: Service thread queue flushed. 2021-05-31 13:53:25 2 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (8848e2e7-c217-11eb-adfb-4a913920f98c): 1 (Operation not permitted) at /home/buildbot/buildbot/build/galera/src/replicator_str.cpp:prepare_for_IST():467. IST will be unavailable. 2021-05-31 13:53:25 0 [Note] WSREP: Member 1.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 0.0 (os-ctrl-0)(SYNCED) as donor. 2021-05-31 13:53:25 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2) 2021-05-31 13:53:25 2 [Note] WSREP: Requesting state transfer: success, donor: 0 2021-05-31 13:53:25 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:2 WSREP_SST: [INFO] Proceeding with SST (20210531 13:53:25.675) WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210531 13:53:25.676) WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20210531 13:53:25.677) removed '/var/lib/mysql/mysql-bin.index' removed '/var/lib/mysql/mysql-bin.state' removed '/var/lib/mysql/ibdata1' removed '/var/lib/mysql/mysql-bin.000001' removed '/var/lib/mysql/aria_log.00000001' removed '/var/lib/mysql/aria_log_control' removed '/var/lib/mysql/ib_logfile1' WSREP_SST: [INFO] Waiting for SST streaming to complete! (20210531 13:53:25.686) 2021-05-31 13:53:25 0 [Note] WSREP: declaring 88467a99 at tcp://10.33.0.13:4567 stable 2021-05-31 13:53:25 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-05-31 13:53:25 0 [Note] WSREP: Node 88467a99 state prim 2021-05-31 13:53:25 0 [Note] WSREP: view(view_id(PRIM,88467a99,3) memb { 88467a99,0 913b953a,0 91e3db36,0 } joined { } left { } partitioned { }) 2021-05-31 13:53:25 0 [Note] WSREP: save pc into disk 2021-05-31 13:53:25 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3 2021-05-31 13:53:25 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2021-05-31 13:53:25 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 9232bacf-c217-11eb-9010-564b46ef1620 2021-05-31 13:53:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9232bacf-c217-11eb-9010-564b46ef1620 from 0 (os-ctrl-0) 2021-05-31 13:53:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9232bacf-c217-11eb-9010-564b46ef1620 from 1 (os-ctrl-1) 2021-05-31 13:53:26 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9232bacf-c217-11eb-9010-564b46ef1620 from 2 (os-ctrl-2) 2021-05-31 13:53:26 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 2, members = 1/3 (joined/total), act_id = 2, last_appl. = 0, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-05-31 13:53:26 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-05-31 13:53:26 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:27 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:28 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:28 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-05-31 13:53:29 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:30 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:31 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:32 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:33 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:34 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:35 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:36 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:37 0 [Warning] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable 2021-05-31 13:53:38 0 [Note] WSREP: 0.0 (os-ctrl-0): State transfer to 1.0 (os-ctrl-1) complete. 2021-05-31 13:53:38 0 [Note] WSREP: Member 0.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20210531 13:53:38.375) WSREP_SST: [INFO] Evaluating /usr//bin/mariabackup --prepare $INNOEXTRA --target-dir=${DATA} &> /var/lib/mysql//mariabackup.prepare.log (20210531 13:53:38.376) 2021-05-31 13:53:38 0 [Note] WSREP: Member 2.0 (os-ctrl-2) requested state transfer from '*any*'. Selected 0.0 (os-ctrl-0)(SYNCED) as donor. WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20210531 13:53:39.328) WSREP_SST: [INFO] Evaluating /usr//bin/mariabackup --move-back --force-non-empty-directories --target-dir=${DATA} &> /var/lib/mysql//mariabackup.move.log (20210531 13:53:39.330) WSREP_SST: [INFO] Move successful, removing /var/lib/mysql//.sst (20210531 13:53:39.338) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:2 (20210531 13:53:39.341) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210531 13:53:39.343) WSREP_SST: [INFO] Removing the sst_in_progress file (20210531 13:53:39.344) 2021-05-31 13:53:39 0 [Note] WSREP: SST complete, seqno: 2 2021-05-31 13:53:39 0 [Note] InnoDB: Using Linux native AIO 2021-05-31 13:53:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-05-31 13:53:39 0 [Note] InnoDB: Uses event mutexes 2021-05-31 13:53:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-05-31 13:53:39 0 [Note] InnoDB: Number of pools: 1 2021-05-31 13:53:39 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-05-31 13:53:39 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-05-31 13:53:39 0 [Note] InnoDB: Completed initialization of buffer pool 2021-05-31 13:53:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-05-31 13:53:39 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-05-31 13:53:39 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-05-31 13:53:39 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-05-31 13:53:39 0 [Note] InnoDB: New log files created, LSN=1628976 2021-05-31 13:53:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-05-31 13:53:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-05-31 13:53:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-05-31 13:53:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-05-31 13:53:39 0 [Note] InnoDB: 10.3.29 started; log sequence number 1629196; transaction id 20 2021-05-31 13:53:39 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-05-31 13:53:39 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-05-31 13:53:39 0 [Note] Recovering after a crash using mysql-bin 2021-05-31 13:53:39 0 [Note] Starting crash recovery... 2021-05-31 13:53:39 0 [Note] InnoDB: Buffer pool(s) load completed at 210531 13:53:39 2021-05-31 13:53:39 0 [Note] Crash recovery finished. 2021-05-31 13:53:39 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-05-31 13:53:39 0 [Note] WSREP: Signalling provider to continue. 2021-05-31 13:53:39 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:2 2021-05-31 13:53:39 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-05-31 13:53:39 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:2, view# 3: Primary, number of nodes: 3, my index: 1, protocol version 3 2021-05-31 13:53:39 0 [Note] Reading of all Master_info entries succeeded 2021-05-31 13:53:39 0 [Note] Added new Master_info '' to hash table 2021-05-31 13:53:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-05-31 13:53:39 0 [Note] WSREP: 1.0 (os-ctrl-1): State transfer from 0.0 (os-ctrl-0) complete. 2021-05-31 13:53:39 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 2) 2021-05-31 13:53:39 0 [Note] WSREP: Member 1.0 (os-ctrl-1) synced with group. 2021-05-31 13:53:39 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 2) 2021-05-31 13:53:39 2 [Note] WSREP: Assign initial position for certification: 2, protocol version: 4 2021-05-31 13:53:39 0 [Note] WSREP: Service thread queue flushed. 2021-05-31 13:53:39 2 [Note] WSREP: Synchronized with group, ready for connections 2021-05-31 13:53:39 0 [Warning] WSREP: Protocol violation. JOIN message sender 1.0 (os-ctrl-1) is not in state transfer (SYNCED). Message ignored. 2021-05-31 13:53:51 0 [Note] WSREP: 0.0 (os-ctrl-0): State transfer to 2.0 (os-ctrl-2) complete. 2021-05-31 13:53:51 0 [Note] WSREP: Member 0.0 (os-ctrl-0) synced with group. 2021-05-31 13:53:53 0 [Note] WSREP: 2.0 (os-ctrl-2): State transfer from 0.0 (os-ctrl-0) complete. 2021-05-31 13:53:53 0 [Note] WSREP: Member 2.0 (os-ctrl-2) synced with group. 2021-05-31 13:53:57 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-05-31 13:53:57 0 [Note] WSREP: forgetting 88467a99 (tcp://10.33.0.13:4567) 2021-05-31 13:53:57 0 [Note] WSREP: Node 913b953a state prim 2021-05-31 13:53:57 0 [Note] WSREP: view(view_id(PRIM,913b953a,4) memb { 913b953a,0 91e3db36,0 } joined { } left { } partitioned { 88467a99,0 }) 2021-05-31 13:53:57 0 [Note] WSREP: save pc into disk 2021-05-31 13:53:57 0 [Note] WSREP: forgetting 88467a99 (tcp://10.33.0.13:4567) 2021-05-31 13:53:57 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 2021-05-31 13:53:57 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: a50aa5d7-c217-11eb-a9ee-cab9231d1892 2021-05-31 13:53:57 0 [Note] WSREP: STATE EXCHANGE: sent state msg: a50aa5d7-c217-11eb-a9ee-cab9231d1892 2021-05-31 13:53:57 0 [Note] WSREP: STATE EXCHANGE: got state msg: a50aa5d7-c217-11eb-a9ee-cab9231d1892 from 0 (os-ctrl-1) 2021-05-31 13:53:57 0 [Note] WSREP: STATE EXCHANGE: got state msg: a50aa5d7-c217-11eb-a9ee-cab9231d1892 from 1 (os-ctrl-2) 2021-05-31 13:53:57 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 3, members = 2/2 (joined/total), act_id = 2, last_appl. = 0, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-05-31 13:53:57 0 [Note] WSREP: Flow-control interval: [23, 23] 2021-05-31 13:53:57 9 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-05-31 13:53:57 9 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:2, view# 4: Primary, number of nodes: 2, my index: 0, protocol version 3 2021-05-31 13:53:57 9 [Note] WSREP: Assign initial position for certification: 2, protocol version: 4 2021-05-31 13:53:57 0 [Note] WSREP: Service thread queue flushed. 2021-05-31 13:54:02 0 [Note] WSREP: cleaning up 88467a99 (tcp://10.33.0.13:4567) 2021-05-31 13:54:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-05-31 13:54:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: 2021-05-31 13:54:05 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-05-31 13:54:05 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-05-31 13:54:05 0 [Note] WSREP: Node 913b953a state prim 2021-05-31 13:54:05 0 [Note] WSREP: view(view_id(PRIM,913b953a,5) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-05-31 13:54:05 0 [Note] WSREP: save pc into disk 2021-05-31 13:54:05 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-05-31 13:54:05 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: aa0149b8-c217-11eb-bb3f-534d7fdee0c7 2021-05-31 13:54:05 0 [Note] WSREP: STATE EXCHANGE: sent state msg: aa0149b8-c217-11eb-bb3f-534d7fdee0c7 2021-05-31 13:54:05 0 [Note] WSREP: STATE EXCHANGE: got state msg: aa0149b8-c217-11eb-bb3f-534d7fdee0c7 from 0 (os-ctrl-1) 2021-05-31 13:54:05 0 [Note] WSREP: STATE EXCHANGE: got state msg: aa0149b8-c217-11eb-bb3f-534d7fdee0c7 from 1 (os-ctrl-2) 2021-05-31 13:54:06 0 [Note] WSREP: STATE EXCHANGE: got state msg: aa0149b8-c217-11eb-bb3f-534d7fdee0c7 from 2 (os-ctrl-0) 2021-05-31 13:54:06 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 4, members = 3/3 (joined/total), act_id = 2, last_appl. = 0, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-05-31 13:54:06 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-05-31 13:54:06 9 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-05-31 13:54:06 9 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:2, view# 5: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-05-31 13:54:06 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. 2021-05-31 13:54:06 9 [Note] WSREP: Assign initial position for certification: 2, protocol version: 4 2021-05-31 13:54:06 0 [Note] WSREP: Service thread queue flushed. 2021-05-31 13:54:08 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-05-31 14:21:32 0 [ERROR] WSREP: Trx 9305 tries to abort slave trx 9306. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210531 14:21:32 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=2 max_threads=10002 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561eac495b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x561eabf2a8c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f41e13e1980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f41e101cfb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f41e101e921] /usr/sbin/mysqld(+0x90a914)[0x561eac0b4914] /usr/sbin/mysqld(+0x90f83c)[0x561eac0b983c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x561eabd3ca83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f41e13d66db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f41e10ff71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E 210531 14:21:33 mysqld_safe Number of processes running now: 0 210531 14:21:33 mysqld_safe WSREP: not restarting wsrep node automatically 210531 14:21:33 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210531 14:21:34 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210531 14:21:34 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210531 14:21:38 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:9304 2021-05-31 14:21:38 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 248 ... 2021-05-31 14:21:38 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-05-31 14:21:38 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-05-31 14:21:38 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-05-31 14:21:38 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-05-31 14:21:38 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-05-31 14:21:38 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-05-31 14:21:38 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:9304 2021-05-31 14:21:38 0 [Note] WSREP: Assign initial position for certification: 9304, protocol version: -1 2021-05-31 14:21:38 0 [Note] WSREP: wsrep_sst_grab() 2021-05-31 14:21:38 0 [Note] WSREP: Start replication 2021-05-31 14:21:38 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:9304 2021-05-31 14:21:38 0 [Note] WSREP: protonet asio version 0 2021-05-31 14:21:38 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-05-31 14:21:38 0 [Note] WSREP: backend: asio 2021-05-31 14:21:38 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-05-31 14:21:38 0 [Note] WSREP: restore pc from disk successfully 2021-05-31 14:21:38 0 [Note] WSREP: GMCast version 0 2021-05-31 14:21:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-05-31 14:21:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-05-31 14:21:38 0 [Note] WSREP: EVS version 0 2021-05-31 14:21:38 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-05-31 14:21:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-05-31 14:21:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: 2021-05-31 14:21:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-05-31 14:21:38 0 [Note] WSREP: gcomm: connected 2021-05-31 14:21:38 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-05-31 14:21:38 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-05-31 14:21:38 0 [Note] WSREP: Opened channel 'openstack' 2021-05-31 14:21:38 0 [Note] WSREP: Waiting for SST to complete. 2021-05-31 14:21:39 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-05-31 14:21:39 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,7) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-05-31 14:21:39 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-05-31 14:21:39 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-05-31 14:21:39 0 [Note] WSREP: Received NON-PRIMARY. 2021-05-31 14:21:39 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:9304, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-05-31 14:21:39 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-05-31 14:21:39 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-05-31 14:21:39 0 [Note] WSREP: Node 91e3db36 state prim 2021-05-31 14:21:39 0 [Note] WSREP: view(view_id(PRIM,913b953a,8) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-05-31 14:21:39 0 [Note] WSREP: save pc into disk 2021-05-31 14:21:39 0 [Note] WSREP: clear restored view 2021-05-31 14:21:39 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-05-31 14:21:39 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 83a6ad73-c21b-11eb-b52a-ce4c4817d4cd 2021-05-31 14:21:39 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 83a6ad73-c21b-11eb-b52a-ce4c4817d4cd 2021-05-31 14:21:39 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-05-31 14:21:39 0 [Note] WSREP: STATE EXCHANGE: got state msg: 83a6ad73-c21b-11eb-b52a-ce4c4817d4cd from 0 (os-ctrl-1) 2021-05-31 14:21:39 0 [Note] WSREP: STATE EXCHANGE: got state msg: 83a6ad73-c21b-11eb-b52a-ce4c4817d4cd from 1 (os-ctrl-2) 2021-05-31 14:21:39 0 [Note] WSREP: STATE EXCHANGE: got state msg: 83a6ad73-c21b-11eb-b52a-ce4c4817d4cd from 2 (os-ctrl-0) 2021-05-31 14:21:39 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 5, members = 2/3 (joined/total), act_id = 9317, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-05-31 14:21:39 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-05-31 14:21:39 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 9317) 2021-05-31 14:21:39 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:9317 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:9304 2021-05-31 14:21:39 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-05-31 14:21:39 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:9317, view# 6: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-05-31 14:21:39 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-05-31 14:21:39 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '248' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:9304' WSREP_SST: [INFO] Moving /var/lib/mysql//mariabackup.prepare.log to /tmp/sst_log_archive/mariabackup.prepare.log.2021.05.31-14.21.39.492088827 (20210531 14:21:39.494) WSREP_SST: [INFO] Moving /var/lib/mysql//mariabackup.move.log to /tmp/sst_log_archive/mariabackup.move.log.2021.05.31-14.21.39.492088827 (20210531 14:21:39.498) WSREP_SST: [INFO] Streaming with xbstream (20210531 14:21:39.502) WSREP_SST: [INFO] Using socat as streamer (20210531 14:21:39.503) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210531 14:21:39.528) 2021-05-31 14:21:39 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-05-31 14:21:39 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-05-31 14:21:39 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-05-31 14:21:39 2 [Note] WSREP: Assign initial position for certification: 9317, protocol version: 4 2021-05-31 14:21:39 0 [Note] WSREP: Service thread queue flushed. 2021-05-31 14:21:39 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-05-31 14:21:39 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-05-31 14:21:39 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-05-31 14:21:39 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 9333) 2021-05-31 14:21:39 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-05-31 14:21:39 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:9317 2021-05-31 14:21:40 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-05-31 14:21:40 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210531 14:21:40.163) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:9304 0 (20210531 14:21:40.165) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210531 14:21:40.167) WSREP_SST: [INFO] Removing the sst_in_progress file (20210531 14:21:40.169) 2021-05-31 14:21:40 0 [Note] WSREP: SST complete, seqno: 9304 2021-05-31 14:21:40 0 [Note] InnoDB: Using Linux native AIO 2021-05-31 14:21:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-05-31 14:21:40 0 [Note] InnoDB: Uses event mutexes 2021-05-31 14:21:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-05-31 14:21:40 0 [Note] InnoDB: Number of pools: 1 2021-05-31 14:21:40 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-05-31 14:21:40 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-05-31 14:21:40 0 [Note] InnoDB: Completed initialization of buffer pool 2021-05-31 14:21:40 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-05-31 14:21:40 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-05-31 14:21:40 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-05-31 14:21:40 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-05-31 14:21:40 0 [Note] InnoDB: New log files created, LSN=32804760 2021-05-31 14:21:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-05-31 14:21:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-05-31 14:21:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-05-31 14:21:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-05-31 14:21:40 0 [Note] InnoDB: 10.3.29 started; log sequence number 32804876; transaction id 36478 2021-05-31 14:21:40 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-05-31 14:21:40 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-05-31 14:21:40 0 [Note] InnoDB: Buffer pool(s) load completed at 210531 14:21:40 2021-05-31 14:21:40 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-05-31 14:21:40 0 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired 2021-05-31 14:21:40 0 [Warning] Checking table: './mysql/user' 2021-05-31 14:21:40 0 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly 2021-05-31 14:21:40 0 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired 2021-05-31 14:21:40 0 [Warning] Checking table: './mysql/db' 2021-05-31 14:21:40 0 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly 2021-05-31 14:21:40 0 [Note] WSREP: Signalling provider to continue. 2021-05-31 14:21:40 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:9304 2021-05-31 14:21:40 0 [Note] Reading of all Master_info entries succeeded 2021-05-31 14:21:40 0 [Note] Added new Master_info '' to hash table 2021-05-31 14:21:40 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-05-31 14:21:40 2 [Note] WSREP: Receiving IST: 13 writesets, seqnos 9304-9317 2021-05-31 14:21:40 0 [Note] WSREP: Receiving IST... 0.0% ( 0/13 events) complete. 2021-05-31 14:21:40 0 [Note] WSREP: Receiving IST...100.0% (13/13 events) complete. 2021-05-31 14:21:40 2 [Note] WSREP: IST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:9317 2021-05-31 14:21:40 0 [Note] WSREP: 0.0 (os-ctrl-1): State transfer from 2.0 (os-ctrl-0) complete. 2021-05-31 14:21:40 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 9367) 2021-05-31 14:21:41 0 [Note] WSREP: Member 0.0 (os-ctrl-1) synced with group. 2021-05-31 14:21:41 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 9373) 2021-05-31 14:21:41 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-05-31 14:21:41 9 [Note] WSREP: Synchronized with group, ready for connections 2021-05-31 14:28:02 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.3.175:4567 2021-05-31 14:28:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-05-31 14:28:07 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-05-31 14:28:08 0 [Note] WSREP: evs::proto(913b953a, GATHER, view_id(REG,913b953a,8)) suspecting node: 91e3db36 2021-05-31 14:28:08 0 [Note] WSREP: evs::proto(913b953a, GATHER, view_id(REG,913b953a,8)) suspected node without join message, declaring inactive 2021-05-31 14:28:08 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-05-31 14:28:08 0 [Note] WSREP: Node 913b953a state prim 2021-05-31 14:28:08 0 [Warning] WSREP: 913b953a sending install message failed: Resource temporarily unavailable 2021-05-31 14:28:09 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-05-31 14:28:09 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-05-31 14:28:09 0 [Note] WSREP: Node 913b953a state prim 2021-05-31 14:28:09 0 [Note] WSREP: view(view_id(PRIM,913b953a,11) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-05-31 14:28:09 0 [Note] WSREP: save pc into disk 2021-05-31 14:28:09 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-05-31 14:28:09 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 6c707acc-c21c-11eb-846c-568544b04f28 2021-05-31 14:28:09 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-05-31 14:28:09 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 6c707acc-c21c-11eb-846c-568544b04f28 2021-05-31 14:28:09 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6c707acc-c21c-11eb-846c-568544b04f28 from 0 (os-ctrl-1) 2021-05-31 14:28:09 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6c707acc-c21c-11eb-846c-568544b04f28 from 1 (os-ctrl-2) 2021-05-31 14:28:09 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6c707acc-c21c-11eb-846c-568544b04f28 from 2 (os-ctrl-0) 2021-05-31 14:28:09 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 6, members = 2/3 (joined/total), act_id = 12748, last_appl. = 12653, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-05-31 14:28:09 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-05-31 14:28:09 10 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-05-31 14:28:09 10 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:12748, view# 7: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-05-31 14:28:09 10 [Note] WSREP: Assign initial position for certification: 12748, protocol version: 4 2021-05-31 14:28:09 0 [Note] WSREP: Service thread queue flushed. 2021-05-31 14:28:10 0 [Note] WSREP: Member 1.0 (os-ctrl-2) requested state transfer from '*any*'. Selected 0.0 (os-ctrl-1)(SYNCED) as donor. 2021-05-31 14:28:10 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 12754) 2021-05-31 14:28:10 11 [Note] WSREP: IST request: 8848e2e7-c217-11eb-adfb-4a913920f98c:12732-12748|tcp://10.33.3.175:4568 2021-05-31 14:28:10 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '10.33.3.175:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --gtid '8848e2e7-c217-11eb-adfb-4a913920f98c:12732' --gtid-domain-id '0' --binlog 'mysql-bin' --bypass --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:9304' 2021-05-31 14:28:10 11 [Note] WSREP: sst_donor_thread signaled with 0 2021-05-31 14:28:10 0 [Note] WSREP: async IST sender starting to serve tcp://10.33.3.175:4568 sending 12733-12748 WSREP_SST: [INFO] Streaming with xbstream (20210531 14:28:10.444) WSREP_SST: [INFO] Using socat as streamer (20210531 14:28:10.450) WSREP_SST: [INFO] Bypassing the SST for IST (20210531 14:28:10.453) WSREP_SST: [INFO] Evaluating mbstream -c ${INFO_FILE} ${IST_FILE} | socat -u stdio TCP:10.33.3.175:4444; RC=( ${PIPESTATUS[@]} ) (20210531 14:28:10.455) 2021-05-31 14:28:10 0 [Note] WSREP: 0.0 (os-ctrl-1): State transfer to 1.0 (os-ctrl-2) complete. 2021-05-31 14:28:10 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 12757) 2021-05-31 14:28:10 0 [Note] WSREP: Member 0.0 (os-ctrl-1) synced with group. 2021-05-31 14:28:10 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 12757) 2021-05-31 14:28:10 10 [Note] WSREP: Synchronized with group, ready for connections WSREP_SST: [INFO] Total time on donor: 0 seconds (20210531 14:28:10.459) WSREP_SST: [INFO] Cleaning up temporary directories (20210531 14:28:10.462) 2021-05-31 14:28:11 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-05-31 14:28:11 0 [Note] WSREP: async IST sender served 2021-05-31 14:28:11 0 [Note] WSREP: 1.0 (os-ctrl-2): State transfer from 0.0 (os-ctrl-1) complete. 2021-05-31 14:28:11 0 [Note] WSREP: Member 1.0 (os-ctrl-2) synced with group. 2021-06-01 9:01:39 0 [ERROR] WSREP: Trx 233063 tries to abort slave trx 233065. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:01:39 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=2 max_threads=10002 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5562178e7b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55621737c8c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fb71f5ad980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fb71f1e8fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fb71f1ea921] /usr/sbin/mysqld(+0x90a914)[0x556217506914] /usr/sbin/mysqld(+0x90f83c)[0x55621750b83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x55621718ea83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fb71f5a26db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fb71f2cb71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:01:42 mysqld_safe Number of processes running now: 0 210601 09:01:42 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:01:42 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:01:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:01:43 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:01:47 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233062 2021-06-01 9:01:47 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 250 ... 2021-06-01 9:01:47 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:01:47 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:01:47 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:01:47 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:01:47 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:01:47 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:01:47 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233062 2021-06-01 9:01:47 0 [Note] WSREP: Assign initial position for certification: 233062, protocol version: -1 2021-06-01 9:01:47 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:01:47 0 [Note] WSREP: Start replication 2021-06-01 9:01:47 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233062 2021-06-01 9:01:47 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:01:47 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:01:47 0 [Note] WSREP: backend: asio 2021-06-01 9:01:47 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:01:47 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:01:47 0 [Note] WSREP: GMCast version 0 2021-06-01 9:01:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:01:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:01:47 0 [Note] WSREP: EVS version 0 2021-06-01 9:01:47 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:01:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:01:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:01:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:01:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:01:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:01:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:01:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:01:48 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:01:48 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:01:48 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:01:48 0 [Note] WSREP: view(view_id(PRIM,913b953a,13) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:01:48 0 [Note] WSREP: save pc into disk 2021-06-01 9:01:48 0 [Note] WSREP: clear restored view 2021-06-01 9:01:49 0 [Note] WSREP: gcomm: connected 2021-06-01 9:01:49 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:01:49 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:01:49 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:01:49 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:01:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:01:49 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: fffb5eb9-c2b7-11eb-83ab-42c34b1c80d6 2021-06-01 9:01:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: fffb5eb9-c2b7-11eb-83ab-42c34b1c80d6 2021-06-01 9:01:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: fffb5eb9-c2b7-11eb-83ab-42c34b1c80d6 from 0 (os-ctrl-1) 2021-06-01 9:01:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: fffb5eb9-c2b7-11eb-83ab-42c34b1c80d6 from 1 (os-ctrl-2) 2021-06-01 9:01:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: fffb5eb9-c2b7-11eb-83ab-42c34b1c80d6 from 2 (os-ctrl-0) 2021-06-01 9:01:49 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 8, members = 2/3 (joined/total), act_id = 233189, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:01:49 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:01:49 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 233189) 2021-06-01 9:01:49 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233189 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233062 2021-06-01 9:01:49 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:01:49 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233189, view# 9: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:01:49 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:01:49 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '250' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233062' 2021-06-01 9:01:49 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.04558S), skipping check WSREP_SST: [INFO] Streaming with xbstream (20210601 09:01:49.594) WSREP_SST: [INFO] Using socat as streamer (20210601 09:01:49.597) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:01:49.630) 2021-06-01 9:01:49 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:01:49 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:01:49 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:01:49 2 [Note] WSREP: Assign initial position for certification: 233189, protocol version: 4 2021-06-01 9:01:49 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:01:49 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:01:49 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:01:49 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:01:49 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 233206) 2021-06-01 9:01:49 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:01:49 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233189 2021-06-01 9:01:50 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:01:50 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:01:50.107) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233062 0 (20210601 09:01:50.109) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:01:50.112) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:01:50.116) 2021-06-01 9:01:50 0 [Note] WSREP: SST complete, seqno: 233062 2021-06-01 9:01:50 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:01:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:01:50 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:01:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:01:50 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:01:50 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:01:50 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:01:50 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:01:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:01:51 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:01:51 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:01:51 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:01:51 0 [Note] InnoDB: New log files created, LSN=257166106 2021-06-01 9:01:51 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:01:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:01:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:01:51 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:01:51 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:01:51 0 [Note] InnoDB: 10.3.29 started; log sequence number 257166348; transaction id 510709 2021-06-01 9:01:51 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:01:51 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:01:51 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:01:51 2021-06-01 9:01:51 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:01:51 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:01:51 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233062 2021-06-01 9:01:51 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:01:51 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:01:51 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:01:51 2 [Note] WSREP: Receiving IST: 127 writesets, seqnos 233062-233189 2021-06-01 9:01:51 0 [Note] WSREP: Receiving IST... 0.0% ( 0/127 events) complete. 2021-06-01 9:01:52 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:01:54 0 [Note] WSREP: Receiving IST...100.0% (127/127 events) complete. 2021-06-01 9:01:54 2 [Note] WSREP: IST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233189 2021-06-01 9:01:54 0 [Note] WSREP: 0.0 (os-ctrl-1): State transfer from 2.0 (os-ctrl-0) complete. 2021-06-01 9:01:54 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 233327) 2021-06-01 9:01:57 0 [Note] WSREP: Member 0.0 (os-ctrl-1) synced with group. 2021-06-01 9:01:57 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 233389) 2021-06-01 9:01:57 0 [ERROR] WSREP: Trx 233370 tries to abort slave trx 233373. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:01:57 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=1 max_threads=10002 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5567fba74b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x5567fb5098c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fc0d3f45980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fc0d3b80fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fc0d3b82921] /usr/sbin/mysqld(+0x90a914)[0x5567fb693914] /usr/sbin/mysqld(+0x90f83c)[0x5567fb69883c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x5567fb31ba83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fc0d3f3a6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc0d3c6371f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:01:59 mysqld_safe Number of processes running now: 0 210601 09:01:59 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:01:59 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:02:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:02:00 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:02:04 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233369 2021-06-01 9:02:04 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:02:04 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:02:04 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 251 ... 2021-06-01 9:02:04 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:02:04 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:02:04 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:02:04 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:02:04 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233369 2021-06-01 9:02:04 0 [Note] WSREP: Assign initial position for certification: 233369, protocol version: -1 2021-06-01 9:02:04 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:02:04 0 [Note] WSREP: Start replication 2021-06-01 9:02:04 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233369 2021-06-01 9:02:04 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:02:04 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:02:04 0 [Note] WSREP: backend: asio 2021-06-01 9:02:04 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:02:04 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:02:04 0 [Note] WSREP: GMCast version 0 2021-06-01 9:02:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:02:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:02:04 0 [Note] WSREP: EVS version 0 2021-06-01 9:02:04 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:02:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:02:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:02:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:02:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:07 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:02:07 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:02:07 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:02:07 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,15) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:07 0 [Note] WSREP: gcomm: connected 2021-06-01 9:02:07 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:02:07 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:02:07 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:02:07 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:02:07 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:02:07 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:02:07 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:02:07 1 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233369, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:02:07 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:07 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:07 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:02:07 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:02:07 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:02:07 0 [Note] WSREP: view(view_id(PRIM,913b953a,16) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:07 0 [Note] WSREP: save pc into disk 2021-06-01 9:02:07 0 [Note] WSREP: clear restored view 2021-06-01 9:02:07 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:02:07 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0a9bf31c-c2b8-11eb-9dc4-cbb412a1c97e 2021-06-01 9:02:07 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:02:07 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0a9bf31c-c2b8-11eb-9dc4-cbb412a1c97e 2021-06-01 9:02:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0a9bf31c-c2b8-11eb-9dc4-cbb412a1c97e from 0 (os-ctrl-1) 2021-06-01 9:02:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0a9bf31c-c2b8-11eb-9dc4-cbb412a1c97e from 2 (os-ctrl-0) 2021-06-01 9:02:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0a9bf31c-c2b8-11eb-9dc4-cbb412a1c97e from 1 (os-ctrl-2) 2021-06-01 9:02:07 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 10, members = 2/3 (joined/total), act_id = 233541, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:02:07 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:02:07 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 233541) 2021-06-01 9:02:07 1 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233541 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233369 2021-06-01 9:02:07 1 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:02:07 1 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233541, view# 11: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:02:07 1 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:02:07 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '251' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233369' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:02:07.507) WSREP_SST: [INFO] Using socat as streamer (20210601 09:02:07.509) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:02:07.549) 2021-06-01 9:02:07 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.52709S), skipping check 2021-06-01 9:02:07 1 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:02:07 1 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:02:07 1 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:02:07 1 [Note] WSREP: Assign initial position for certification: 233541, protocol version: 4 2021-06-01 9:02:07 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:02:07 1 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:02:07 1 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:02:07 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:02:07 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 233560) 2021-06-01 9:02:07 1 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:02:07 1 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233541 2021-06-01 9:02:08 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:02:08.101) 2021-06-01 9:02:08 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233369 0 (20210601 09:02:08.104) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:02:08.106) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:02:08.108) 2021-06-01 9:02:08 0 [Note] WSREP: SST complete, seqno: 233369 2021-06-01 9:02:08 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:02:08 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:02:08 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:02:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:02:08 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:02:08 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:02:08 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:02:08 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:02:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:02:08 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:02:08 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:02:08 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:02:08 0 [Note] InnoDB: New log files created, LSN=257716045 2021-06-01 9:02:08 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:02:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:02:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:02:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:02:08 0 [Note] InnoDB: 10.3.29 started; log sequence number 257716236; transaction id 511521 2021-06-01 9:02:08 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:02:08 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:02:08 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:02:08 2021-06-01 9:02:08 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:02:08 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:02:08 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233369 2021-06-01 9:02:08 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:02:08 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:02:08 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:02:08 1 [Note] WSREP: Receiving IST: 172 writesets, seqnos 233369-233541 2021-06-01 9:02:08 0 [Note] WSREP: Receiving IST... 0.0% ( 0/172 events) complete. 2021-06-01 9:02:08 0 [ERROR] WSREP: Trx 233372 tries to abort slave trx 233373. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:02:08 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55cd7f32cb8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55cd7edc18c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fc0b187e980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fc0b14b9fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fc0b14bb921] /usr/sbin/mysqld(+0x90a914)[0x55cd7ef4b914] /usr/sbin/mysqld(+0x90f83c)[0x55cd7ef5083c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x55cd7ebd3a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fc0b18736db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc0b159c71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:02:10 mysqld_safe Number of processes running now: 0 210601 09:02:10 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:02:10 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:02:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:02:11 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:02:14 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233371 2021-06-01 9:02:14 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 251 ... 2021-06-01 9:02:14 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:02:14 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:02:14 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:02:14 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:02:14 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:02:14 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:02:14 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233371 2021-06-01 9:02:14 0 [Note] WSREP: Assign initial position for certification: 233371, protocol version: -1 2021-06-01 9:02:14 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:02:14 0 [Note] WSREP: Start replication 2021-06-01 9:02:14 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233371 2021-06-01 9:02:14 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:02:14 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:02:14 0 [Note] WSREP: backend: asio 2021-06-01 9:02:14 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:02:14 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:02:14 0 [Note] WSREP: GMCast version 0 2021-06-01 9:02:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:02:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:02:14 0 [Note] WSREP: EVS version 0 2021-06-01 9:02:14 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:02:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: 2021-06-01 9:02:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:15 0 [Note] WSREP: gcomm: connected 2021-06-01 9:02:15 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:02:15 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:02:15 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:02:15 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:02:16 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:02:16 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:02:16 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:16 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:17 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:02:17 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:02:17 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:17 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:21 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:02:22 0 [Warning] WSREP: evs::proto(913b953a, GATHER, view_id(TRANS,913b953a,17)) install timer expired evs::proto(evs::proto(913b953a, GATHER, view_id(TRANS,913b953a,17)), GATHER) { current_view=view(view_id(TRANS,913b953a,17) memb { 913b953a,0 } joined { } left { } partitioned { }), input_map=evs::input_map: {aru_seq=-1,safe_seq=-1,node_index=node: {idx=0,range=[0,-1],safe_seq=-1} }, fifo_seq=12, last_sent=-1, known: 913b953a at {o=1,s=0,i=0,fs=-1,jm= {v=0,t=4,ut=255,o=1,s=-1,sr=-1,as=-1,f=0,src=913b953a,srcvid=view_id(TRANS,913b953a,17),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=12,nl=( 913b953a, {o=1,s=0,e=0,ls=-1,vid=view_id(TRANS,913b953a,17),ss=-1,ir=[0,-1],} 91e3db36, {o=0,s=1,e=0,ls=-1,vid=view_id(REG,00000000,0),ss=-1,ir=[-1,-1],} a9b132d9, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,17),ss=91,ir=[92,91],} ) }, } 91e3db36 at tcp://10.33.3.175:4567 {o=0,s=1,i=0,fs=836259,} a9b132d9 at tcp://10.33.0.13:4567 {o=1,s=0,i=0,fs=874496,jm= {v=0,t=4,ut=255,o=1,s=91,sr=-1,as=91,f=4,src=a9b132d9,srcvid=view_id(REG,91e3db36,17),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=874496,nl=( 913b953a, {o=1,s=0,e=0,ls=-1,vid=view_id(TRANS,913b953a,17),ss=-1,ir=[0,-1],} 91e3db36, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,17),ss=91,ir=[92,91],} a9b132d9, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,17),ss=91,ir=[92,91],} ) }, } } 2021-06-01 9:02:22 0 [Note] WSREP: no install message received 2021-06-01 9:02:22 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:02:22 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,18) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:22 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:02:22 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:02:22 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:02:22 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233371, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:02:22 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:02:22 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:02:22 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:02:22 0 [Note] WSREP: view(view_id(PRIM,913b953a,19) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:22 0 [Note] WSREP: save pc into disk 2021-06-01 9:02:22 0 [Note] WSREP: clear restored view 2021-06-01 9:02:22 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:02:22 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 13d8f749-c2b8-11eb-b911-061db10afa39 2021-06-01 9:02:22 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:02:22 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 13d8f749-c2b8-11eb-b911-061db10afa39 2021-06-01 9:02:22 0 [Note] WSREP: STATE EXCHANGE: got state msg: 13d8f749-c2b8-11eb-b911-061db10afa39 from 0 (os-ctrl-1) 2021-06-01 9:02:22 0 [Note] WSREP: STATE EXCHANGE: got state msg: 13d8f749-c2b8-11eb-b911-061db10afa39 from 1 (os-ctrl-2) 2021-06-01 9:02:22 0 [Note] WSREP: STATE EXCHANGE: got state msg: 13d8f749-c2b8-11eb-b911-061db10afa39 from 2 (os-ctrl-0) 2021-06-01 9:02:22 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 12, members = 2/3 (joined/total), act_id = 233677, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:02:22 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:02:22 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 233677) 2021-06-01 9:02:22 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233677 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233371 2021-06-01 9:02:22 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:02:22 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233677, view# 13: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:02:22 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:02:22 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '251' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233371' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:02:22.922) WSREP_SST: [INFO] Using socat as streamer (20210601 09:02:22.924) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:02:22.959) 2021-06-01 9:02:23 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:02:23 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:02:23 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:02:23 2 [Note] WSREP: Assign initial position for certification: 233677, protocol version: 4 2021-06-01 9:02:23 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:02:23 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:02:23 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:02:23 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:02:23 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 233706) 2021-06-01 9:02:23 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:02:23 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233677 2021-06-01 9:02:23 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.50162S), skipping check 2021-06-01 9:02:23 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:02:23 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:02:23.578) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233371 0 (20210601 09:02:23.580) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:02:23.582) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:02:23.584) 2021-06-01 9:02:23 0 [Note] WSREP: SST complete, seqno: 233371 2021-06-01 9:02:23 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:02:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:02:23 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:02:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:02:23 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:02:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:02:23 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:02:23 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:02:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:02:23 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:02:23 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:02:23 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:02:23 0 [Note] InnoDB: New log files created, LSN=257719531 2021-06-01 9:02:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:02:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:02:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:02:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:02:23 0 [Note] InnoDB: 10.3.29 started; log sequence number 257719820; transaction id 511535 2021-06-01 9:02:23 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:02:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:02:23 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:02:23 2021-06-01 9:02:24 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:02:24 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:02:24 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233371 2021-06-01 9:02:24 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:02:24 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:02:24 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:02:24 2 [Note] WSREP: Receiving IST: 306 writesets, seqnos 233371-233677 2021-06-01 9:02:24 0 [Note] WSREP: Receiving IST... 0.0% ( 0/306 events) complete. 2021-06-01 9:02:24 0 [ERROR] WSREP: Trx 233386 tries to abort slave trx 233387. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:02:24 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ad73ecab8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55ad7395f8c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f560b3f8980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f560b033fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f560b035921] /usr/sbin/mysqld(+0x90a914)[0x55ad73ae9914] /usr/sbin/mysqld(+0x90f83c)[0x55ad73aee83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x55ad73771a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f560b3ed6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f560b11671f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:02:26 mysqld_safe Number of processes running now: 0 210601 09:02:26 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:02:26 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:02:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:02:27 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:02:30 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:30 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:02:30 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 250 ... 2021-06-01 9:02:30 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:02:30 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:02:30 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:02:30 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:02:30 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:02:30 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:30 0 [Note] WSREP: Assign initial position for certification: 233385, protocol version: -1 2021-06-01 9:02:30 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:02:30 0 [Note] WSREP: Start replication 2021-06-01 9:02:30 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:30 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:02:30 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:02:30 0 [Note] WSREP: backend: asio 2021-06-01 9:02:30 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:02:30 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:02:30 0 [Note] WSREP: GMCast version 0 2021-06-01 9:02:30 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:02:30 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:02:30 0 [Note] WSREP: EVS version 0 2021-06-01 9:02:30 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:02:30 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:30 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:31 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:02:32 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:02:32 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:02:32 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:32 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:33 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:02:33 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:02:33 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:33 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:02:33 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,21) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:33 0 [Note] WSREP: gcomm: connected 2021-06-01 9:02:33 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:02:33 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:02:33 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:02:33 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:02:33 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:02:33 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:33 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:02:33 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:02:33 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233385, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:02:34 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50597S), skipping check 2021-06-01 9:02:34 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:02:34 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:02:34 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:02:34 0 [Note] WSREP: view(view_id(PRIM,913b953a,22) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:34 0 [Note] WSREP: save pc into disk 2021-06-01 9:02:34 0 [Note] WSREP: clear restored view 2021-06-01 9:02:34 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:02:34 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 1b0649db-c2b8-11eb-9977-dbda8522123b 2021-06-01 9:02:34 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 1b0649db-c2b8-11eb-9977-dbda8522123b 2021-06-01 9:02:34 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:02:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: 1b0649db-c2b8-11eb-9977-dbda8522123b from 0 (os-ctrl-1) 2021-06-01 9:02:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: 1b0649db-c2b8-11eb-9977-dbda8522123b from 1 (os-ctrl-2) 2021-06-01 9:02:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: 1b0649db-c2b8-11eb-9977-dbda8522123b from 2 (os-ctrl-0) 2021-06-01 9:02:34 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 14, members = 2/3 (joined/total), act_id = 233815, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:02:34 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:02:34 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 233815) 2021-06-01 9:02:34 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233815 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:34 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:02:34 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233815, view# 15: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:02:34 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:02:34 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '250' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233385' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:02:34.990) WSREP_SST: [INFO] Using socat as streamer (20210601 09:02:34.994) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:02:35.034) 2021-06-01 9:02:35 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:02:35 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:02:35 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:02:35 2 [Note] WSREP: Assign initial position for certification: 233815, protocol version: 4 2021-06-01 9:02:35 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:02:35 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:02:35 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:02:35 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:02:35 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 233823) 2021-06-01 9:02:35 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:02:35 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233815 2021-06-01 9:02:35 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:02:35 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:02:35.466) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 0 (20210601 09:02:35.469) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:02:35.471) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:02:35.473) 2021-06-01 9:02:35 0 [Note] WSREP: SST complete, seqno: 233385 2021-06-01 9:02:35 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:02:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:02:35 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:02:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:02:35 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:02:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:02:35 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:02:35 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:02:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:02:36 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:02:36 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:02:36 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:02:36 0 [Note] InnoDB: New log files created, LSN=257764037 2021-06-01 9:02:36 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:02:36 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:02:36 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:02:36 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:02:36 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:02:36 0 [Note] InnoDB: 10.3.29 started; log sequence number 257764364; transaction id 511693 2021-06-01 9:02:36 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:02:36 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:02:36 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:02:36 2021-06-01 9:02:36 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:02:36 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:02:36 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:36 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:02:36 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:02:36 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:02:36 2 [Note] WSREP: Receiving IST: 430 writesets, seqnos 233385-233815 2021-06-01 9:02:36 0 [Note] WSREP: Receiving IST... 0.0% ( 0/430 events) complete. 2021-06-01 9:02:37 0 [ERROR] WSREP: Trx 233386 tries to abort slave trx 233387. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:02:37 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562be0c90b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x562be07258c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f15b630d980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f15b5f48fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f15b5f4a921] /usr/sbin/mysqld(+0x90a914)[0x562be08af914] /usr/sbin/mysqld(+0x90f83c)[0x562be08b483c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x562be0537a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f15b63026db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f15b602b71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:02:38 mysqld_safe Number of processes running now: 0 210601 09:02:38 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:02:38 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:02:39 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:02:39 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:02:42 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:42 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 251 ... 2021-06-01 9:02:42 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:02:42 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:02:42 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:02:42 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:02:42 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:02:42 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:02:42 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:42 0 [Note] WSREP: Assign initial position for certification: 233385, protocol version: -1 2021-06-01 9:02:42 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:02:42 0 [Note] WSREP: Start replication 2021-06-01 9:02:42 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:42 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:02:42 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:02:42 0 [Note] WSREP: backend: asio 2021-06-01 9:02:42 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:02:42 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:02:42 0 [Note] WSREP: GMCast version 0 2021-06-01 9:02:42 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:02:42 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:02:42 0 [Note] WSREP: EVS version 0 2021-06-01 9:02:42 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:02:42 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:42 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: 2021-06-01 9:02:42 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:42 0 [Note] WSREP: gcomm: connected 2021-06-01 9:02:42 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:02:42 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:02:42 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:02:42 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:02:43 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:02:43 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,24) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:43 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:02:43 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:02:43 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:02:43 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233385, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:02:43 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:02:43 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:02:43 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:02:43 0 [Note] WSREP: view(view_id(PRIM,913b953a,25) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:43 0 [Note] WSREP: save pc into disk 2021-06-01 9:02:43 0 [Note] WSREP: clear restored view 2021-06-01 9:02:43 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:02:43 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 2018d05c-c2b8-11eb-a83f-d70829f0a7a2 2021-06-01 9:02:43 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:02:43 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 2018d05c-c2b8-11eb-a83f-d70829f0a7a2 2021-06-01 9:02:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2018d05c-c2b8-11eb-a83f-d70829f0a7a2 from 0 (os-ctrl-1) 2021-06-01 9:02:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2018d05c-c2b8-11eb-a83f-d70829f0a7a2 from 1 (os-ctrl-2) 2021-06-01 9:02:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2018d05c-c2b8-11eb-a83f-d70829f0a7a2 from 2 (os-ctrl-0) 2021-06-01 9:02:43 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 15, members = 2/3 (joined/total), act_id = 233890, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:02:43 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:02:43 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 233890) 2021-06-01 9:02:43 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233890 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:43 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:02:43 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233890, view# 16: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:02:43 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:02:43 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '251' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233385' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:02:43.495) WSREP_SST: [INFO] Using socat as streamer (20210601 09:02:43.497) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:02:43.521) 2021-06-01 9:02:43 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:02:43 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:02:43 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:02:43 2 [Note] WSREP: Assign initial position for certification: 233890, protocol version: 4 2021-06-01 9:02:43 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:02:43 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:02:43 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:02:43 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:02:43 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 233907) 2021-06-01 9:02:43 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:02:43 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233890 2021-06-01 9:02:44 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:02:44.119) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 0 (20210601 09:02:44.122) 2021-06-01 9:02:44 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:02:44.124) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:02:44.126) 2021-06-01 9:02:44 0 [Note] WSREP: SST complete, seqno: 233385 2021-06-01 9:02:44 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:02:44 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:02:44 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:02:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:02:44 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:02:44 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:02:44 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:02:44 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:02:44 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:02:44 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:02:44 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:02:44 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:02:44 0 [Note] InnoDB: New log files created, LSN=257764373 2021-06-01 9:02:44 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:02:44 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:02:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:02:44 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:02:44 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:02:44 0 [Note] InnoDB: 10.3.29 started; log sequence number 257764876; transaction id 511693 2021-06-01 9:02:44 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:02:44 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:02:44 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:02:44 2021-06-01 9:02:44 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:02:44 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:02:44 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233385 2021-06-01 9:02:44 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:02:44 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:02:44 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:02:44 2 [Note] WSREP: Receiving IST: 505 writesets, seqnos 233385-233890 2021-06-01 9:02:44 0 [Note] WSREP: Receiving IST... 0.0% ( 0/505 events) complete. 2021-06-01 9:02:45 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:02:46 0 [ERROR] WSREP: Trx 233461 tries to abort slave trx 233462. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:02:46 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55719a446b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x557199edb8c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f1e3841a980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f1e38055fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f1e38057921] /usr/sbin/mysqld(+0x90a914)[0x55719a065914] /usr/sbin/mysqld(+0x90f83c)[0x55719a06a83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x557199ceda83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f1e3840f6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f1e3813871f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:02:48 mysqld_safe Number of processes running now: 0 210601 09:02:48 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:02:48 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:02:49 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:02:49 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:02:53 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233460 2021-06-01 9:02:53 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 253 ... 2021-06-01 9:02:53 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:02:53 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:02:53 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:02:53 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:02:53 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:02:53 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:02:53 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233460 2021-06-01 9:02:54 0 [Note] WSREP: Assign initial position for certification: 233460, protocol version: -1 2021-06-01 9:02:54 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:02:54 0 [Note] WSREP: Start replication 2021-06-01 9:02:54 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233460 2021-06-01 9:02:54 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:02:54 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:02:54 0 [Note] WSREP: backend: asio 2021-06-01 9:02:54 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:02:54 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:02:54 0 [Note] WSREP: GMCast version 0 2021-06-01 9:02:54 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:02:54 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:02:54 0 [Note] WSREP: EVS version 0 2021-06-01 9:02:54 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:02:54 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:54 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:54 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:02:55 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:02:55 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:02:55 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:02:55 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:02:55 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:02:55 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:02:55 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:02:55 0 [Note] WSREP: view(view_id(PRIM,913b953a,27) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:02:55 0 [Note] WSREP: save pc into disk 2021-06-01 9:02:55 0 [Note] WSREP: clear restored view 2021-06-01 9:02:56 0 [Note] WSREP: gcomm: connected 2021-06-01 9:02:56 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:02:56 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:02:56 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:02:56 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:02:56 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 27a8adbd-c2b8-11eb-a238-a706ca620a12 2021-06-01 9:02:56 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:02:56 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 27a8adbd-c2b8-11eb-a238-a706ca620a12 2021-06-01 9:02:56 0 [Note] WSREP: STATE EXCHANGE: got state msg: 27a8adbd-c2b8-11eb-a238-a706ca620a12 from 0 (os-ctrl-1) 2021-06-01 9:02:56 0 [Note] WSREP: STATE EXCHANGE: got state msg: 27a8adbd-c2b8-11eb-a238-a706ca620a12 from 1 (os-ctrl-2) 2021-06-01 9:02:56 0 [Note] WSREP: STATE EXCHANGE: got state msg: 27a8adbd-c2b8-11eb-a238-a706ca620a12 from 2 (os-ctrl-0) 2021-06-01 9:02:56 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 17, members = 2/3 (joined/total), act_id = 234046, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:02:56 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:02:56 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 234046) 2021-06-01 9:02:56 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234046 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233460 2021-06-01 9:02:56 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:02:56 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234046, view# 18: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:02:56 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:02:56 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '253' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233460' 2021-06-01 9:02:56 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.09562S), skipping check WSREP_SST: [INFO] Streaming with xbstream (20210601 09:02:56.149) WSREP_SST: [INFO] Using socat as streamer (20210601 09:02:56.151) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:02:56.177) 2021-06-01 9:02:56 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:02:56 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:02:56 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:02:56 2 [Note] WSREP: Assign initial position for certification: 234046, protocol version: 4 2021-06-01 9:02:56 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:02:56 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:02:56 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:02:56 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:02:56 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 234054) 2021-06-01 9:02:56 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:02:56 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234046 2021-06-01 9:02:56 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:02:56 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:02:56.619) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233460 0 (20210601 09:02:56.621) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:02:56.624) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:02:56.626) 2021-06-01 9:02:56 0 [Note] WSREP: SST complete, seqno: 233460 2021-06-01 9:02:56 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:02:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:02:56 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:02:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:02:56 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:02:56 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:02:56 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:02:56 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:02:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:02:56 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:02:56 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:02:56 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:02:56 0 [Note] InnoDB: New log files created, LSN=257920627 2021-06-01 9:02:57 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:02:57 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:02:57 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:02:57 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:02:57 0 [Note] InnoDB: 10.3.29 started; log sequence number 257921036; transaction id 511983 2021-06-01 9:02:57 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:02:57 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:02:57 2021-06-01 9:02:57 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:02:57 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:02:57 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:02:57 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233460 2021-06-01 9:02:57 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:02:57 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:02:57 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:02:57 2 [Note] WSREP: Receiving IST: 586 writesets, seqnos 233460-234046 2021-06-01 9:02:57 0 [Note] WSREP: Receiving IST... 0.0% ( 0/586 events) complete. 2021-06-01 9:02:59 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:03:02 0 [ERROR] WSREP: Trx 233723 tries to abort slave trx 233724. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:03:02 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=2 max_threads=10002 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 2021-06-01 9:03:02 2 [Warning] WSREP: conflict state after RBR event applying: 1, 233724 2021-06-01 9:03:02 2 [Warning] WSREP: RBR event apply failed, rolling back: 233724 stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ebc363bb8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55ebc30d08c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f4ccc132980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f4ccbd6dfb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f4ccbd6f921] /usr/sbin/mysqld(+0x90a914)[0x55ebc325a914] /usr/sbin/mysqld(+0x90f83c)[0x55ebc325f83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x55ebc2ee2a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f4ccc1276db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f4ccbe5071f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:03:04 mysqld_safe Number of processes running now: 0 210601 09:03:04 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:03:04 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:03:05 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:03:05 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:03:09 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233722 2021-06-01 9:03:09 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:03:09 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 251 ... 2021-06-01 9:03:09 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:03:09 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:03:09 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:03:09 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:03:09 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:03:09 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233722 2021-06-01 9:03:09 0 [Note] WSREP: Assign initial position for certification: 233722, protocol version: -1 2021-06-01 9:03:09 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:03:09 0 [Note] WSREP: Start replication 2021-06-01 9:03:09 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233722 2021-06-01 9:03:09 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:03:09 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:03:09 0 [Note] WSREP: backend: asio 2021-06-01 9:03:09 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:03:09 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:03:09 0 [Note] WSREP: GMCast version 0 2021-06-01 9:03:09 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:03:09 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:03:09 0 [Note] WSREP: EVS version 0 2021-06-01 9:03:09 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:03:09 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:09 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:09 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:03:10 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:03:10 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:03:10 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:10 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:12 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:03:12 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:03:12 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:03:12 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,29) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:03:12 0 [Note] WSREP: gcomm: connected 2021-06-01 9:03:12 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:03:12 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:03:12 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:03:12 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:03:12 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:03:12 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:03:12 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:03:12 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233722, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:03:12 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:12 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:12 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:03:12 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:03:12 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:03:12 0 [Note] WSREP: view(view_id(PRIM,913b953a,30) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:03:12 0 [Note] WSREP: save pc into disk 2021-06-01 9:03:12 0 [Note] WSREP: clear restored view 2021-06-01 9:03:12 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:03:12 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 31513f3c-c2b8-11eb-9611-838c86979b6b 2021-06-01 9:03:12 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 31513f3c-c2b8-11eb-9611-838c86979b6b 2021-06-01 9:03:12 0 [Note] WSREP: STATE EXCHANGE: got state msg: 31513f3c-c2b8-11eb-9611-838c86979b6b from 0 (os-ctrl-1) 2021-06-01 9:03:12 0 [Note] WSREP: STATE EXCHANGE: got state msg: 31513f3c-c2b8-11eb-9611-838c86979b6b from 1 (os-ctrl-2) 2021-06-01 9:03:12 0 [Note] WSREP: STATE EXCHANGE: got state msg: 31513f3c-c2b8-11eb-9611-838c86979b6b from 2 (os-ctrl-0) 2021-06-01 9:03:12 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 19, members = 2/3 (joined/total), act_id = 234287, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:03:12 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:03:12 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 234287) 2021-06-01 9:03:12 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234287 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233722 2021-06-01 9:03:12 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:03:12 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234287, view# 20: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:03:12 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:03:12 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '251' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233722' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:03:12.368) WSREP_SST: [INFO] Using socat as streamer (20210601 09:03:12.369) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:03:12.393) 2021-06-01 9:03:12 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:03:12 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:03:12 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:03:12 2 [Note] WSREP: Assign initial position for certification: 234287, protocol version: 4 2021-06-01 9:03:12 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:03:12 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:03:12 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:03:12 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:03:12 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 234295) 2021-06-01 9:03:12 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:03:12 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234287 2021-06-01 9:03:12 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.55389S), skipping check 2021-06-01 9:03:12 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:03:12 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:03:12.824) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233722 0 (20210601 09:03:12.828) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:03:12.830) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:03:12.832) 2021-06-01 9:03:12 0 [Note] WSREP: SST complete, seqno: 233722 2021-06-01 9:03:12 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:03:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:03:12 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:03:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:03:12 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:03:12 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:03:12 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:03:13 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:03:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:03:13 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:03:13 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:03:13 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:03:13 0 [Note] InnoDB: New log files created, LSN=258361649 2021-06-01 9:03:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:03:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:03:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:03:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:03:13 0 [Note] InnoDB: 10.3.29 started; log sequence number 258361868; transaction id 512716 2021-06-01 9:03:13 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:03:13 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:03:13 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:03:13 2021-06-01 9:03:13 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:03:13 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:03:13 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233722 2021-06-01 9:03:13 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:03:13 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:03:13 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:03:13 2 [Note] WSREP: Receiving IST: 565 writesets, seqnos 233722-234287 2021-06-01 9:03:13 0 [Note] WSREP: Receiving IST... 0.0% ( 0/565 events) complete. 2021-06-01 9:03:15 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:03:17 0 [ERROR] WSREP: Trx 233935 tries to abort slave trx 233936. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:03:17 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=2 max_threads=10002 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55cc67c56b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55cc676eb8c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f9662a66980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f96626a1fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f96626a3921] /usr/sbin/mysqld(+0x90a914)[0x55cc67875914] /usr/sbin/mysqld(+0x90f83c)[0x55cc6787a83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x55cc674fda83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f9662a5b6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f966278471f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:03:19 mysqld_safe Number of processes running now: 0 210601 09:03:19 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:03:19 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:03:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:03:20 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:03:24 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233934 2021-06-01 9:03:24 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 250 ... 2021-06-01 9:03:24 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:03:24 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:03:24 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:03:24 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:03:24 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:03:24 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:03:24 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233934 2021-06-01 9:03:24 0 [Note] WSREP: Assign initial position for certification: 233934, protocol version: -1 2021-06-01 9:03:24 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:03:24 0 [Note] WSREP: Start replication 2021-06-01 9:03:24 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233934 2021-06-01 9:03:24 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:03:24 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:03:24 0 [Note] WSREP: backend: asio 2021-06-01 9:03:24 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:03:24 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:03:24 0 [Note] WSREP: GMCast version 0 2021-06-01 9:03:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:03:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:03:24 0 [Note] WSREP: EVS version 0 2021-06-01 9:03:24 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:03:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:03:25 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:03:25 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:03:25 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:25 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:03:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:03:27 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:03:27 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,32) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:03:27 0 [Note] WSREP: gcomm: connected 2021-06-01 9:03:27 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:03:27 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:03:27 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:03:27 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:03:27 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:03:27 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:03:27 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:03:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:27 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233934, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:03:27 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:03:27 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:03:27 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:03:27 0 [Note] WSREP: view(view_id(PRIM,913b953a,33) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:03:27 0 [Note] WSREP: save pc into disk 2021-06-01 9:03:27 0 [Note] WSREP: clear restored view 2021-06-01 9:03:27 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:03:27 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 3a3cddf1-c2b8-11eb-91f6-6a73949806cb 2021-06-01 9:03:27 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 3a3cddf1-c2b8-11eb-91f6-6a73949806cb 2021-06-01 9:03:27 0 [Note] WSREP: STATE EXCHANGE: got state msg: 3a3cddf1-c2b8-11eb-91f6-6a73949806cb from 0 (os-ctrl-1) 2021-06-01 9:03:27 0 [Note] WSREP: STATE EXCHANGE: got state msg: 3a3cddf1-c2b8-11eb-91f6-6a73949806cb from 2 (os-ctrl-0) 2021-06-01 9:03:27 0 [Note] WSREP: STATE EXCHANGE: got state msg: 3a3cddf1-c2b8-11eb-91f6-6a73949806cb from 1 (os-ctrl-2) 2021-06-01 9:03:27 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 21, members = 2/3 (joined/total), act_id = 234526, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:03:27 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:03:27 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 234526) 2021-06-01 9:03:27 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234526 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233934 2021-06-01 9:03:27 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:03:27 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234526, view# 22: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:03:27 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:03:27 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '250' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233934' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:03:27.427) WSREP_SST: [INFO] Using socat as streamer (20210601 09:03:27.430) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:03:27.465) 2021-06-01 9:03:27 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 2021-06-01 9:03:27 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.5829S), skipping check ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:03:27 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:03:27 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:03:27 2 [Note] WSREP: Assign initial position for certification: 234526, protocol version: 4 2021-06-01 9:03:27 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:03:27 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:03:27 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:03:27 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:03:27 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 234535) 2021-06-01 9:03:27 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:03:27 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234526 2021-06-01 9:03:28 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:03:28 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:03:28.141) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233934 0 (20210601 09:03:28.152) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:03:28.157) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:03:28.161) 2021-06-01 9:03:28 0 [Note] WSREP: SST complete, seqno: 233934 2021-06-01 9:03:28 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:03:28 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:03:28 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:03:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:03:28 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:03:28 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:03:28 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:03:28 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:03:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:03:28 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:03:28 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:03:28 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:03:28 0 [Note] InnoDB: New log files created, LSN=258707573 2021-06-01 9:03:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:03:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:03:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:03:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:03:28 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:03:28 0 [Note] InnoDB: 10.3.29 started; log sequence number 258707980; transaction id 513307 2021-06-01 9:03:28 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:03:28 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:03:28 2021-06-01 9:03:28 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:03:28 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:03:28 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:03:28 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233934 2021-06-01 9:03:28 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:03:28 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:03:28 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:03:28 2 [Note] WSREP: Receiving IST: 592 writesets, seqnos 233934-234526 2021-06-01 9:03:28 0 [Note] WSREP: Receiving IST... 0.0% ( 0/592 events) complete. 2021-06-01 9:03:29 0 [ERROR] WSREP: Trx 233950 tries to abort slave trx 233951. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:03:29 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x565021346b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x565020ddb8c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f25d9c6a980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f25d98a5fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f25d98a7921] /usr/sbin/mysqld(+0x90a914)[0x565020f65914] /usr/sbin/mysqld(+0x90f83c)[0x565020f6a83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x565020beda83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f25d9c5f6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f25d998871f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:03:30 mysqld_safe Number of processes running now: 0 210601 09:03:30 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:03:30 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:03:31 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:03:31 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:03:35 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233948 2021-06-01 9:03:35 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 252 ... 2021-06-01 9:03:35 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:03:35 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:03:35 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:03:35 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:03:35 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:03:35 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:03:35 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233948 2021-06-01 9:03:35 0 [Note] WSREP: Assign initial position for certification: 233948, protocol version: -1 2021-06-01 9:03:35 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:03:35 0 [Note] WSREP: Start replication 2021-06-01 9:03:35 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233948 2021-06-01 9:03:35 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:03:35 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:03:35 0 [Note] WSREP: backend: asio 2021-06-01 9:03:35 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:03:35 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:03:35 0 [Note] WSREP: GMCast version 0 2021-06-01 9:03:35 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:03:35 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:03:35 0 [Note] WSREP: EVS version 0 2021-06-01 9:03:35 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:03:35 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:35 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:36 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:03:37 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:03:37 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:03:37 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:37 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:03:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:03:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:38 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:03:38 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,35) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:03:38 0 [Note] WSREP: gcomm: connected 2021-06-01 9:03:38 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:03:38 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:03:38 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:03:38 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:03:38 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:03:38 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:03:38 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:03:38 1 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233948, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:03:39 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50388S), skipping check 2021-06-01 9:03:39 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:03:39 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:03:40 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:03:40 0 [Note] WSREP: view(view_id(PRIM,913b953a,36) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:03:40 0 [Note] WSREP: save pc into disk 2021-06-01 9:03:40 0 [Note] WSREP: clear restored view 2021-06-01 9:03:40 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:03:40 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 41e39f8b-c2b8-11eb-a43b-ef4c790b8efa 2021-06-01 9:03:40 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 41e39f8b-c2b8-11eb-a43b-ef4c790b8efa 2021-06-01 9:03:40 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:03:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 41e39f8b-c2b8-11eb-a43b-ef4c790b8efa from 0 (os-ctrl-1) 2021-06-01 9:03:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 41e39f8b-c2b8-11eb-a43b-ef4c790b8efa from 1 (os-ctrl-2) 2021-06-01 9:03:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 41e39f8b-c2b8-11eb-a43b-ef4c790b8efa from 2 (os-ctrl-0) 2021-06-01 9:03:40 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 23, members = 2/3 (joined/total), act_id = 234659, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:03:40 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:03:40 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 234659) 2021-06-01 9:03:40 1 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234659 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233948 2021-06-01 9:03:40 1 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:03:40 1 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234659, view# 24: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:03:40 1 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:03:40 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '252' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233948' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:03:40.273) WSREP_SST: [INFO] Using socat as streamer (20210601 09:03:40.280) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:03:40.310) 2021-06-01 9:03:40 1 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:03:40 1 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:03:40 1 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:03:40 1 [Note] WSREP: Assign initial position for certification: 234659, protocol version: 4 2021-06-01 9:03:40 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:03:40 1 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:03:40 1 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:03:40 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:03:40 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 234695) 2021-06-01 9:03:40 1 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:03:40 1 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234659 2021-06-01 9:03:40 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:03:40 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:03:40.860) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233948 0 (20210601 09:03:40.866) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:03:40.872) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:03:40.875) 2021-06-01 9:03:40 0 [Note] WSREP: SST complete, seqno: 233948 2021-06-01 9:03:40 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:03:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:03:40 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:03:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:03:40 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:03:40 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:03:40 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:03:41 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:03:41 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:03:41 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:03:41 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:03:41 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:03:41 0 [Note] InnoDB: New log files created, LSN=258737782 2021-06-01 9:03:41 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:03:41 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:03:41 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:03:41 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:03:41 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:03:41 0 [Note] InnoDB: 10.3.29 started; log sequence number 258738188; transaction id 513456 2021-06-01 9:03:41 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:03:41 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:03:41 2021-06-01 9:03:41 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:03:41 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:03:41 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:03:41 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233948 2021-06-01 9:03:41 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:03:41 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:03:41 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:03:41 1 [Note] WSREP: Receiving IST: 711 writesets, seqnos 233948-234659 2021-06-01 9:03:41 0 [Note] WSREP: Receiving IST... 0.0% ( 0/711 events) complete. 2021-06-01 9:03:41 0 [ERROR] WSREP: Trx 233966 tries to abort slave trx 233967. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:03:41 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562ffa281b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x562ff9d168c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fa3b215c980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fa3b1d97fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fa3b1d99921] /usr/sbin/mysqld(+0x90a914)[0x562ff9ea0914] /usr/sbin/mysqld(+0x90f83c)[0x562ff9ea583c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x562ff9b28a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fa3b21516db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fa3b1e7a71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:03:43 mysqld_safe Number of processes running now: 0 210601 09:03:43 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:03:43 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:03:44 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:03:44 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:03:48 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:233965 2021-06-01 9:03:48 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 251 ... 2021-06-01 9:03:48 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:03:48 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:03:48 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:03:48 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:03:48 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:03:48 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:03:48 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:233965 2021-06-01 9:03:48 0 [Note] WSREP: Assign initial position for certification: 233965, protocol version: -1 2021-06-01 9:03:48 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:03:48 0 [Note] WSREP: Start replication 2021-06-01 9:03:48 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:233965 2021-06-01 9:03:48 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:03:48 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:03:48 0 [Note] WSREP: backend: asio 2021-06-01 9:03:48 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:03:48 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:03:48 0 [Note] WSREP: GMCast version 0 2021-06-01 9:03:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:03:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:03:48 0 [Note] WSREP: EVS version 0 2021-06-01 9:03:48 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:03:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:48 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:03:49 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:03:49 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:03:49 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:49 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:03:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:03:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:03:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:03:51 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:03:51 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,38) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:03:51 0 [Note] WSREP: gcomm: connected 2021-06-01 9:03:51 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:03:51 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:03:51 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:03:51 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:03:51 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:03:51 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:03:51 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233965, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:03:51 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:03:51 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.5041S), skipping check 2021-06-01 9:03:52 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:03:52 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:03:52 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:03:52 0 [Note] WSREP: view(view_id(PRIM,913b953a,39) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:03:52 0 [Note] WSREP: save pc into disk 2021-06-01 9:03:52 0 [Note] WSREP: clear restored view 2021-06-01 9:03:52 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:03:52 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 492cb4ad-c2b8-11eb-bbb4-e2adc4d55ac5 2021-06-01 9:03:52 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:03:52 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 492cb4ad-c2b8-11eb-bbb4-e2adc4d55ac5 2021-06-01 9:03:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 492cb4ad-c2b8-11eb-bbb4-e2adc4d55ac5 from 0 (os-ctrl-1) 2021-06-01 9:03:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 492cb4ad-c2b8-11eb-bbb4-e2adc4d55ac5 from 1 (os-ctrl-2) 2021-06-01 9:03:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 492cb4ad-c2b8-11eb-bbb4-e2adc4d55ac5 from 2 (os-ctrl-0) 2021-06-01 9:03:52 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 25, members = 2/3 (joined/total), act_id = 234857, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:03:52 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:03:52 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 234857) 2021-06-01 9:03:52 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234857 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:233965 2021-06-01 9:03:52 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:03:52 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234857, view# 26: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:03:52 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:03:52 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '251' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:233965' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:03:52.424) WSREP_SST: [INFO] Using socat as streamer (20210601 09:03:52.428) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:03:52.492) 2021-06-01 9:03:52 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:03:52 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:03:52 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:03:52 2 [Note] WSREP: Assign initial position for certification: 234857, protocol version: 4 2021-06-01 9:03:52 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:03:52 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:03:52 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:03:52 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:03:52 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 234879) 2021-06-01 9:03:52 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:03:52 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234857 2021-06-01 9:03:53 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:03:53.020) 2021-06-01 9:03:53 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:233965 0 (20210601 09:03:53.024) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:03:53.031) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:03:53.034) 2021-06-01 9:03:53 0 [Note] WSREP: SST complete, seqno: 233965 2021-06-01 9:03:53 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:03:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:03:53 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:03:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:03:53 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:03:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:03:53 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:03:53 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:03:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:03:53 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:03:53 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:03:53 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:03:53 0 [Note] InnoDB: New log files created, LSN=258775626 2021-06-01 9:03:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:03:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:03:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:03:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:03:53 0 [Note] InnoDB: 10.3.29 started; log sequence number 258776076; transaction id 513619 2021-06-01 9:03:53 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:03:53 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:03:53 2021-06-01 9:03:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:03:53 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:03:53 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:03:53 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:233965 2021-06-01 9:03:53 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:03:53 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:03:53 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:03:53 2 [Note] WSREP: Receiving IST: 892 writesets, seqnos 233965-234857 2021-06-01 9:03:53 0 [Note] WSREP: Receiving IST... 0.0% ( 0/892 events) complete. 2021-06-01 9:03:54 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:03:55 0 [ERROR] WSREP: Trx 234057 tries to abort slave trx 234059. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:03:55 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=1 max_threads=10002 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5596bf99ab8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x5596bf42f8c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fdc8ff99980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fdc8fbd4fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fdc8fbd6921] /usr/sbin/mysqld(+0x90a914)[0x5596bf5b9914] /usr/sbin/mysqld(+0x90f83c)[0x5596bf5be83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x5596bf241a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fdc8ff8e6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fdc8fcb771f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:03:56 mysqld_safe Number of processes running now: 0 210601 09:03:56 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:03:56 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:03:57 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:03:57 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:04:01 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:234056 2021-06-01 9:04:01 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 250 ... 2021-06-01 9:04:01 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:04:01 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:04:01 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:04:01 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:04:01 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:04:01 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:04:01 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234056 2021-06-01 9:04:01 0 [Note] WSREP: Assign initial position for certification: 234056, protocol version: -1 2021-06-01 9:04:01 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:04:01 0 [Note] WSREP: Start replication 2021-06-01 9:04:01 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:234056 2021-06-01 9:04:01 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:04:01 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:04:01 0 [Note] WSREP: backend: asio 2021-06-01 9:04:01 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:04:01 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:04:01 0 [Note] WSREP: GMCast version 0 2021-06-01 9:04:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:04:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:04:01 0 [Note] WSREP: EVS version 0 2021-06-01 9:04:01 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:04:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:04:02 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:04:02 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:04:02 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:02 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:04:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:04:04 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:04:04 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,41) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:04:04 0 [Note] WSREP: gcomm: connected 2021-06-01 9:04:04 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:04:04 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:04:04 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:04:04 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:04:04 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:04:04 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:04:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:04 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:04:04 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:04 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234056, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:04:04 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:04:04 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:04:04 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:04:04 0 [Note] WSREP: view(view_id(PRIM,913b953a,42) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:04:04 0 [Note] WSREP: save pc into disk 2021-06-01 9:04:04 0 [Note] WSREP: clear restored view 2021-06-01 9:04:04 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:04:04 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 50939cdc-c2b8-11eb-8162-feac00e7edec 2021-06-01 9:04:04 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 50939cdc-c2b8-11eb-8162-feac00e7edec 2021-06-01 9:04:04 0 [Note] WSREP: STATE EXCHANGE: got state msg: 50939cdc-c2b8-11eb-8162-feac00e7edec from 0 (os-ctrl-1) 2021-06-01 9:04:04 0 [Note] WSREP: STATE EXCHANGE: got state msg: 50939cdc-c2b8-11eb-8162-feac00e7edec from 1 (os-ctrl-2) 2021-06-01 9:04:04 0 [Note] WSREP: STATE EXCHANGE: got state msg: 50939cdc-c2b8-11eb-8162-feac00e7edec from 2 (os-ctrl-0) 2021-06-01 9:04:04 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 27, members = 2/3 (joined/total), act_id = 234961, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:04:04 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:04:04 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 234961) 2021-06-01 9:04:04 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234961 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234056 2021-06-01 9:04:04 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:04:04 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234961, view# 28: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:04:04 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:04:04 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '250' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:234056' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:04:04.860) WSREP_SST: [INFO] Using socat as streamer (20210601 09:04:04.862) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:04:04.900) 2021-06-01 9:04:05 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:04:05 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:04:05 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:04:05 2 [Note] WSREP: Assign initial position for certification: 234961, protocol version: 4 2021-06-01 9:04:05 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:04:05 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:04:05 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:04:05 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:04:05 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 234965) 2021-06-01 9:04:05 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:04:05 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234961 2021-06-01 9:04:05 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.74264S), skipping check 2021-06-01 9:04:05 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:04:05 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:04:05.298) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:234056 0 (20210601 09:04:05.301) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:04:05.305) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:04:05.306) 2021-06-01 9:04:05 0 [Note] WSREP: SST complete, seqno: 234056 2021-06-01 9:04:05 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:04:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:04:05 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:04:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:04:05 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:04:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:04:05 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:04:05 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:04:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:04:05 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:04:05 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:04:05 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:04:05 0 [Note] InnoDB: New log files created, LSN=258955905 2021-06-01 9:04:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:04:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:04:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:04:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:04:05 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:04:05 0 [Note] InnoDB: 10.3.29 started; log sequence number 258956300; transaction id 513935 2021-06-01 9:04:05 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:04:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:04:05 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:04:05 2021-06-01 9:04:05 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:04:05 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:04:05 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:234056 2021-06-01 9:04:05 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:04:05 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:04:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:04:05 2 [Note] WSREP: Receiving IST: 905 writesets, seqnos 234056-234961 2021-06-01 9:04:05 0 [Note] WSREP: Receiving IST... 0.0% ( 0/905 events) complete. 2021-06-01 9:04:05 0 [ERROR] WSREP: Trx 234058 tries to abort slave trx 234059. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:04:05 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56089a990b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x56089a4258c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fb5cc99e980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fb5cc5d9fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fb5cc5db921] /usr/sbin/mysqld(+0x90a914)[0x56089a5af914] /usr/sbin/mysqld(+0x90f83c)[0x56089a5b483c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x56089a237a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fb5cc9936db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fb5cc6bc71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:04:07 mysqld_safe Number of processes running now: 0 210601 09:04:07 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:04:07 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:04:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:04:08 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:04:12 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:234057 2021-06-01 9:04:12 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 250 ... 2021-06-01 9:04:12 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:04:12 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:04:12 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:04:12 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:04:12 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:04:12 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:04:12 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234057 2021-06-01 9:04:12 0 [Note] WSREP: Assign initial position for certification: 234057, protocol version: -1 2021-06-01 9:04:12 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:04:12 0 [Note] WSREP: Start replication 2021-06-01 9:04:12 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:234057 2021-06-01 9:04:12 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:04:12 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:04:12 0 [Note] WSREP: backend: asio 2021-06-01 9:04:12 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:04:12 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:04:12 0 [Note] WSREP: GMCast version 0 2021-06-01 9:04:12 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:04:12 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:04:12 0 [Note] WSREP: EVS version 0 2021-06-01 9:04:12 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:04:12 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:12 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:13 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:04:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:04:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:04:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:14 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:15 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:04:15 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:04:15 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:04:15 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,44) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:04:15 0 [Note] WSREP: gcomm: connected 2021-06-01 9:04:15 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:04:15 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:04:15 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:04:15 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:04:15 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:04:15 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:04:15 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:15 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:04:15 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234057, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:04:15 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:16 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.51675S), skipping check 2021-06-01 9:04:16 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:04:16 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:04:16 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:04:16 0 [Note] WSREP: view(view_id(PRIM,913b953a,45) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:04:16 0 [Note] WSREP: save pc into disk 2021-06-01 9:04:16 0 [Note] WSREP: clear restored view 2021-06-01 9:04:16 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:04:16 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 57cc48b7-c2b8-11eb-97ba-1f8c97cef254 2021-06-01 9:04:16 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 57cc48b7-c2b8-11eb-97ba-1f8c97cef254 2021-06-01 9:04:16 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:04:16 0 [Note] WSREP: STATE EXCHANGE: got state msg: 57cc48b7-c2b8-11eb-97ba-1f8c97cef254 from 0 (os-ctrl-1) 2021-06-01 9:04:16 0 [Note] WSREP: STATE EXCHANGE: got state msg: 57cc48b7-c2b8-11eb-97ba-1f8c97cef254 from 1 (os-ctrl-2) 2021-06-01 9:04:16 0 [Note] WSREP: STATE EXCHANGE: got state msg: 57cc48b7-c2b8-11eb-97ba-1f8c97cef254 from 2 (os-ctrl-0) 2021-06-01 9:04:16 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 29, members = 2/3 (joined/total), act_id = 235055, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:04:16 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:04:16 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 235055) 2021-06-01 9:04:16 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235055 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234057 2021-06-01 9:04:16 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:04:16 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235055, view# 30: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:04:16 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:04:16 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '250' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:234057' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:04:16.901) WSREP_SST: [INFO] Using socat as streamer (20210601 09:04:16.902) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:04:16.933) 2021-06-01 9:04:17 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:04:17 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:04:17 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:04:17 2 [Note] WSREP: Assign initial position for certification: 235055, protocol version: 4 2021-06-01 9:04:17 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:04:17 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:04:17 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:04:17 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:04:17 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 235065) 2021-06-01 9:04:17 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:04:17 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:235055 2021-06-01 9:04:17 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:04:17 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:04:17.335) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:234057 0 (20210601 09:04:17.337) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:04:17.340) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:04:17.343) 2021-06-01 9:04:17 0 [Note] WSREP: SST complete, seqno: 234057 2021-06-01 9:04:17 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:04:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:04:17 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:04:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:04:17 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:04:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:04:17 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:04:17 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:04:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:04:17 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:04:17 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:04:17 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:04:17 0 [Note] InnoDB: New log files created, LSN=258957655 2021-06-01 9:04:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:04:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:04:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:04:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:04:17 0 [Note] InnoDB: 10.3.29 started; log sequence number 258957836; transaction id 513945 2021-06-01 9:04:17 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:04:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:04:17 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:04:17 2021-06-01 9:04:17 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:04:17 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:04:17 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:234057 2021-06-01 9:04:17 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:04:17 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:04:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:04:17 2 [Note] WSREP: Receiving IST: 998 writesets, seqnos 234057-235055 2021-06-01 9:04:17 0 [Note] WSREP: Receiving IST... 0.0% ( 0/998 events) complete. 2021-06-01 9:04:19 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:04:21 0 [ERROR] WSREP: Trx 234243 tries to abort slave trx 234246. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:04:21 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=2 max_threads=10002 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d3e6550b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55d3e5fe58c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fcfb7d42980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcfb797dfb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcfb797f921] /usr/sbin/mysqld(+0x90a914)[0x55d3e616f914] /usr/sbin/mysqld(+0x90f83c)[0x55d3e617483c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x55d3e5df7a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fcfb7d376db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fcfb7a6071f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:04:23 mysqld_safe Number of processes running now: 0 210601 09:04:23 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:04:23 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:04:23 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:04:23 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:04:27 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:234242 2021-06-01 9:04:27 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 250 ... 2021-06-01 9:04:27 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:04:27 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:04:27 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:04:27 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:04:27 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:04:27 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:04:27 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234242 2021-06-01 9:04:27 0 [Note] WSREP: Assign initial position for certification: 234242, protocol version: -1 2021-06-01 9:04:27 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:04:27 0 [Note] WSREP: Start replication 2021-06-01 9:04:27 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:234242 2021-06-01 9:04:27 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:04:27 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:04:27 0 [Note] WSREP: backend: asio 2021-06-01 9:04:27 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:04:27 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:04:27 0 [Note] WSREP: GMCast version 0 2021-06-01 9:04:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:04:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:04:27 0 [Note] WSREP: EVS version 0 2021-06-01 9:04:27 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:04:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: 2021-06-01 9:04:27 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:28 0 [Note] WSREP: gcomm: connected 2021-06-01 9:04:28 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:04:28 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:04:28 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:04:28 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:04:29 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:04:29 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:04:29 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:29 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:30 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:04:30 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:04:30 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:30 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:34 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:04:35 0 [Warning] WSREP: evs::proto(913b953a, GATHER, view_id(TRANS,913b953a,46)) install timer expired evs::proto(evs::proto(913b953a, GATHER, view_id(TRANS,913b953a,46)), GATHER) { current_view=view(view_id(TRANS,913b953a,46) memb { 913b953a,0 } joined { } left { } partitioned { }), input_map=evs::input_map: {aru_seq=-1,safe_seq=-1,node_index=node: {idx=0,range=[0,-1],safe_seq=-1} }, fifo_seq=12, last_sent=-1, known: 913b953a at {o=1,s=0,i=0,fs=-1,jm= {v=0,t=4,ut=255,o=1,s=-1,sr=-1,as=-1,f=0,src=913b953a,srcvid=view_id(TRANS,913b953a,46),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=12,nl=( 913b953a, {o=1,s=0,e=0,ls=-1,vid=view_id(TRANS,913b953a,46),ss=-1,ir=[0,-1],} 91e3db36, {o=0,s=1,e=0,ls=-1,vid=view_id(REG,00000000,0),ss=-1,ir=[-1,-1],} a9b132d9, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,46),ss=64,ir=[65,64],} ) }, } 91e3db36 at tcp://10.33.3.175:4567 {o=0,s=1,i=0,fs=839019,} a9b132d9 at tcp://10.33.0.13:4567 {o=1,s=0,i=0,fs=877876,jm= {v=0,t=4,ut=255,o=1,s=64,sr=-1,as=64,f=4,src=a9b132d9,srcvid=view_id(REG,91e3db36,46),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=877876,nl=( 913b953a, {o=1,s=0,e=0,ls=-1,vid=view_id(TRANS,913b953a,46),ss=-1,ir=[0,-1],} 91e3db36, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,46),ss=64,ir=[65,64],} a9b132d9, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,46),ss=64,ir=[65,64],} ) }, } } 2021-06-01 9:04:35 0 [Note] WSREP: no install message received 2021-06-01 9:04:35 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:04:35 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,47) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:04:35 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:04:35 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:04:35 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:04:35 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234242, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:04:35 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:04:35 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:04:35 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:04:35 0 [Note] WSREP: view(view_id(PRIM,913b953a,48) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:04:35 0 [Note] WSREP: save pc into disk 2021-06-01 9:04:35 0 [Note] WSREP: clear restored view 2021-06-01 9:04:35 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:04:35 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 6321244b-c2b8-11eb-9238-c2e71fc8cdb5 2021-06-01 9:04:35 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 6321244b-c2b8-11eb-9238-c2e71fc8cdb5 2021-06-01 9:04:35 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:04:35 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6321244b-c2b8-11eb-9238-c2e71fc8cdb5 from 0 (os-ctrl-1) 2021-06-01 9:04:35 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6321244b-c2b8-11eb-9238-c2e71fc8cdb5 from 1 (os-ctrl-2) 2021-06-01 9:04:35 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6321244b-c2b8-11eb-9238-c2e71fc8cdb5 from 2 (os-ctrl-0) 2021-06-01 9:04:35 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 31, members = 2/3 (joined/total), act_id = 235244, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:04:35 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:04:35 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 235244) 2021-06-01 9:04:35 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235244 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234242 2021-06-01 9:04:35 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:04:35 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235244, view# 32: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:04:35 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:04:35 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '250' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:234242' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:04:35.913) WSREP_SST: [INFO] Using socat as streamer (20210601 09:04:35.915) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:04:35.937) 2021-06-01 9:04:36 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:04:36 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:04:36 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:04:36 2 [Note] WSREP: Assign initial position for certification: 235244, protocol version: 4 2021-06-01 9:04:36 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:04:36 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:04:36 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:04:36 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:04:36 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 235274) 2021-06-01 9:04:36 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:04:36 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:235244 2021-06-01 9:04:36 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.50443S), skipping check 2021-06-01 9:04:36 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:04:36 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:04:36.484) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:234242 0 (20210601 09:04:36.487) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:04:36.493) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:04:36.496) 2021-06-01 9:04:36 0 [Note] WSREP: SST complete, seqno: 234242 2021-06-01 9:04:36 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:04:36 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:04:36 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:04:36 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:04:36 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:04:36 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:04:36 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:04:36 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:04:36 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:04:36 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:04:36 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:04:36 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:04:36 0 [Note] InnoDB: New log files created, LSN=259287793 2021-06-01 9:04:36 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:04:36 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:04:36 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:04:36 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:04:36 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:04:36 0 [Note] InnoDB: 10.3.29 started; log sequence number 259288076; transaction id 514475 2021-06-01 9:04:36 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:04:36 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:04:36 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:04:36 2021-06-01 9:04:36 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:04:36 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:04:36 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:234242 2021-06-01 9:04:36 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:04:36 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:04:36 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:04:36 2 [Note] WSREP: Receiving IST: 1002 writesets, seqnos 234242-235244 2021-06-01 9:04:36 0 [Note] WSREP: Receiving IST... 0.0% ( 0/1002 events) complete. 2021-06-01 9:04:38 0 [ERROR] WSREP: Trx 234293 tries to abort slave trx 234295. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:04:38 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=1 max_threads=10002 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5555b541db8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x5555b4eb28c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fbc30d66980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fbc309a1fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fbc309a3921] /usr/sbin/mysqld(+0x90a914)[0x5555b503c914] /usr/sbin/mysqld(+0x90f83c)[0x5555b504183c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x5555b4cc4a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fbc30d5b6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fbc30a8471f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:04:40 mysqld_safe Number of processes running now: 0 210601 09:04:40 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:04:40 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:04:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:04:40 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:04:44 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:234292 2021-06-01 9:04:44 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 257 ... 2021-06-01 9:04:44 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:04:44 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:04:44 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:04:44 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:04:44 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:04:44 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:04:44 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234292 2021-06-01 9:04:44 0 [Note] WSREP: Assign initial position for certification: 234292, protocol version: -1 2021-06-01 9:04:44 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:04:44 0 [Note] WSREP: Start replication 2021-06-01 9:04:44 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:234292 2021-06-01 9:04:44 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:04:44 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:04:44 0 [Note] WSREP: backend: asio 2021-06-01 9:04:44 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:04:44 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:04:44 0 [Note] WSREP: GMCast version 0 2021-06-01 9:04:44 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:04:44 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:04:44 0 [Note] WSREP: EVS version 0 2021-06-01 9:04:44 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:04:44 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:44 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: 2021-06-01 9:04:44 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:45 0 [Note] WSREP: gcomm: connected 2021-06-01 9:04:45 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:04:45 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:04:45 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:04:45 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:04:46 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:04:46 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:04:46 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:46 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:04:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:04:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:04:47 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:04:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:04:52 0 [Warning] WSREP: evs::proto(913b953a, GATHER, view_id(TRANS,913b953a,49)) install timer expired evs::proto(evs::proto(913b953a, GATHER, view_id(TRANS,913b953a,49)), GATHER) { current_view=view(view_id(TRANS,913b953a,49) memb { 913b953a,0 } joined { } left { } partitioned { }), input_map=evs::input_map: {aru_seq=-1,safe_seq=-1,node_index=node: {idx=0,range=[0,-1],safe_seq=-1} }, fifo_seq=13, last_sent=-1, known: 913b953a at {o=1,s=0,i=0,fs=-1,jm= {v=0,t=4,ut=255,o=1,s=-1,sr=-1,as=-1,f=0,src=913b953a,srcvid=view_id(TRANS,913b953a,49),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=13,nl=( 913b953a, {o=1,s=0,e=0,ls=-1,vid=view_id(TRANS,913b953a,49),ss=-1,ir=[0,-1],} 91e3db36, {o=0,s=1,e=0,ls=-1,vid=view_id(REG,00000000,0),ss=-1,ir=[-1,-1],} a9b132d9, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,49),ss=65,ir=[66,65],} ) }, } 91e3db36 at tcp://10.33.3.175:4567 {o=0,s=1,i=0,fs=839299,} a9b132d9 at tcp://10.33.0.13:4567 {o=1,s=0,i=0,fs=878216,jm= {v=0,t=4,ut=255,o=1,s=65,sr=-1,as=65,f=4,src=a9b132d9,srcvid=view_id(REG,91e3db36,49),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=878216,nl=( 913b953a, {o=1,s=0,e=0,ls=-1,vid=view_id(TRANS,913b953a,49),ss=-1,ir=[0,-1],} 91e3db36, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,49),ss=65,ir=[66,65],} a9b132d9, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,49),ss=65,ir=[66,65],} ) }, } } 2021-06-01 9:04:52 0 [Note] WSREP: no install message received 2021-06-01 9:04:52 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:04:52 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,50) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:04:52 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:04:52 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:04:52 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:04:52 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234292, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:04:52 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:04:52 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:04:52 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:04:52 0 [Note] WSREP: view(view_id(PRIM,913b953a,51) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:04:52 0 [Note] WSREP: save pc into disk 2021-06-01 9:04:52 0 [Note] WSREP: clear restored view 2021-06-01 9:04:52 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:04:52 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 6d441b49-c2b8-11eb-a577-975bcc23485d 2021-06-01 9:04:52 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:04:52 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 6d441b49-c2b8-11eb-a577-975bcc23485d 2021-06-01 9:04:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6d441b49-c2b8-11eb-a577-975bcc23485d from 0 (os-ctrl-1) 2021-06-01 9:04:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6d441b49-c2b8-11eb-a577-975bcc23485d from 1 (os-ctrl-2) 2021-06-01 9:04:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6d441b49-c2b8-11eb-a577-975bcc23485d from 2 (os-ctrl-0) 2021-06-01 9:04:52 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 33, members = 2/3 (joined/total), act_id = 235407, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:04:52 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:04:52 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 235407) 2021-06-01 9:04:52 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235407 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234292 2021-06-01 9:04:52 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:04:52 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235407, view# 34: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:04:52 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:04:52 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '257' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:234292' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:04:52.933) WSREP_SST: [INFO] Using socat as streamer (20210601 09:04:52.936) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:04:52.963) 2021-06-01 9:04:53 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:04:53 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:04:53 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:04:53 2 [Note] WSREP: Assign initial position for certification: 235407, protocol version: 4 2021-06-01 9:04:53 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:04:53 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:04:53 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:04:53 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:04:53 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 235434) 2021-06-01 9:04:53 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:04:53 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:235407 2021-06-01 9:04:53 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:04:53 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:04:53.469) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:234292 0 (20210601 09:04:53.472) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:04:53.474) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:04:53.477) 2021-06-01 9:04:53 0 [Note] WSREP: SST complete, seqno: 234292 2021-06-01 9:04:53 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:04:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:04:53 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:04:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:04:53 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:04:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:04:53 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:04:53 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:04:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:04:53 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:04:53 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:04:53 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:04:53 0 [Note] InnoDB: New log files created, LSN=259429685 2021-06-01 9:04:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:04:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:04:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:04:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:04:53 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:04:53 0 [Note] InnoDB: 10.3.29 started; log sequence number 259429900; transaction id 514746 2021-06-01 9:04:53 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:04:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:04:53 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:04:53 2021-06-01 9:04:53 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:04:53 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:04:53 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:234292 2021-06-01 9:04:53 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:04:53 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:04:53 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:04:53 2 [Note] WSREP: Receiving IST: 1115 writesets, seqnos 234292-235407 2021-06-01 9:04:53 0 [Note] WSREP: Receiving IST... 0.0% ( 0/1115 events) complete. 2021-06-01 9:04:53 0 [ERROR] WSREP: Trx 234294 tries to abort slave trx 234295. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:04:53 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5632933ddb8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x563292e728c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f2fa2337980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f2fa1f72fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f2fa1f74921] /usr/sbin/mysqld(+0x90a914)[0x563292ffc914] /usr/sbin/mysqld(+0x90f83c)[0x56329300183c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x563292c84a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f2fa232c6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f2fa205571f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:04:56 mysqld_safe Number of processes running now: 0 210601 09:04:56 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:04:56 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:04:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:04:56 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:05:00 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:234293 2021-06-01 9:05:00 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 249 ... 2021-06-01 9:05:00 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:05:00 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:05:00 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:05:00 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:05:00 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:05:00 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:05:00 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:234293 2021-06-01 9:05:00 0 [Note] WSREP: Assign initial position for certification: 234293, protocol version: -1 2021-06-01 9:05:00 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:05:00 0 [Note] WSREP: Start replication 2021-06-01 9:05:00 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:234293 2021-06-01 9:05:00 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:05:00 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:05:00 0 [Note] WSREP: backend: asio 2021-06-01 9:05:00 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:05:00 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:05:00 0 [Note] WSREP: GMCast version 0 2021-06-01 9:05:00 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:05:00 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:05:00 0 [Note] WSREP: EVS version 0 2021-06-01 9:05:00 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:05:00 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:05:00 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: 2021-06-01 9:05:00 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:05:00 0 [Note] WSREP: gcomm: connected 2021-06-01 9:05:00 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:05:00 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:05:00 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:05:00 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:05:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:05:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:05:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:05:01 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:05:03 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:05:03 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:05:03 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:05:03 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:05:06 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:05:07 0 [Warning] WSREP: evs::proto(913b953a, GATHER, view_id(TRANS,913b953a,52)) install timer expired evs::proto(evs::proto(913b953a, GATHER, view_id(TRANS,913b953a,52)), GATHER) { current_view=view(view_id(TRANS,913b953a,52) memb { 913b953a,0 } joined { } left { } partitioned { }), input_map=evs::input_map: {aru_seq=-1,safe_seq=-1,node_index=node: {idx=0,range=[0,-1],safe_seq=-1} }, fifo_seq=12, last_sent=-1, known: 913b953a at {o=1,s=0,i=0,fs=-1,jm= {v=0,t=4,ut=255,o=1,s=-1,sr=-1,as=-1,f=0,src=913b953a,srcvid=view_id(TRANS,913b953a,52),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=12,nl=( 913b953a, {o=1,s=0,e=0,ls=-1,vid=view_id(TRANS,913b953a,52),ss=-1,ir=[0,-1],} 91e3db36, {o=0,s=1,e=0,ls=-1,vid=view_id(REG,00000000,0),ss=-1,ir=[-1,-1],} a9b132d9, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,52),ss=56,ir=[57,56],} ) }, } 91e3db36 at tcp://10.33.3.175:4567 {o=0,s=1,i=0,fs=839527,} a9b132d9 at tcp://10.33.0.13:4567 {o=1,s=0,i=0,fs=878475,jm= {v=0,t=4,ut=255,o=1,s=56,sr=-1,as=56,f=4,src=a9b132d9,srcvid=view_id(REG,91e3db36,52),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=878475,nl=( 913b953a, {o=1,s=0,e=0,ls=-1,vid=view_id(TRANS,913b953a,52),ss=-1,ir=[0,-1],} 91e3db36, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,52),ss=56,ir=[57,56],} a9b132d9, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,91e3db36,52),ss=56,ir=[57,56],} ) }, } } 2021-06-01 9:05:07 0 [Note] WSREP: no install message received 2021-06-01 9:05:07 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:05:07 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,53) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:05:07 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:05:07 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:05:07 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:05:07 1 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234293, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:05:09 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:05:09 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:05:09 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:05:09 0 [Note] WSREP: view(view_id(PRIM,913b953a,54) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:05:09 0 [Note] WSREP: save pc into disk 2021-06-01 9:05:09 0 [Note] WSREP: clear restored view 2021-06-01 9:05:09 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:05:09 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 771a9620-c2b8-11eb-acf0-7329115341ec 2021-06-01 9:05:09 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:05:09 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 771a9620-c2b8-11eb-acf0-7329115341ec 2021-06-01 9:05:09 0 [Note] WSREP: STATE EXCHANGE: got state msg: 771a9620-c2b8-11eb-acf0-7329115341ec from 0 (os-ctrl-1) 2021-06-01 9:05:09 0 [Note] WSREP: STATE EXCHANGE: got state msg: 771a9620-c2b8-11eb-acf0-7329115341ec from 1 (os-ctrl-2) 2021-06-01 9:05:09 0 [Note] WSREP: STATE EXCHANGE: got state msg: 771a9620-c2b8-11eb-acf0-7329115341ec from 2 (os-ctrl-0) 2021-06-01 9:05:09 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 35, members = 2/3 (joined/total), act_id = 235524, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:05:09 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:05:09 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 235524) 2021-06-01 9:05:09 1 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235524 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:234293 2021-06-01 9:05:09 1 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:05:09 1 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235524, view# 36: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:05:09 1 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:05:09 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '249' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:234293' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:05:09.418) WSREP_SST: [INFO] Using socat as streamer (20210601 09:05:09.420) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:05:09.442) 2021-06-01 9:05:09 1 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:05:09 1 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:05:09 1 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:05:09 1 [Note] WSREP: Assign initial position for certification: 235524, protocol version: 4 2021-06-01 9:05:09 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:05:09 1 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:05:09 1 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:05:09 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:05:09 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 235534) 2021-06-01 9:05:09 1 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:05:09 1 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:235524 2021-06-01 9:05:09 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:05:09 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:05:09.814) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:234293 0 (20210601 09:05:09.816) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:05:09.818) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:05:09.820) 2021-06-01 9:05:09 0 [Note] WSREP: SST complete, seqno: 234293 2021-06-01 9:05:09 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:05:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:05:09 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:05:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:05:09 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:05:09 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:05:09 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:05:10 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:05:10 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:05:10 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:05:10 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:05:10 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:05:10 0 [Note] InnoDB: New log files created, LSN=259430857 2021-06-01 9:05:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:05:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:05:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:05:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:05:10 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:05:10 0 [Note] InnoDB: 10.3.29 started; log sequence number 259430924; transaction id 514754 2021-06-01 9:05:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:05:10 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:05:10 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:05:10 2021-06-01 9:05:10 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:05:10 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:05:10 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:234293 2021-06-01 9:05:10 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:05:10 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:05:10 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:05:10 1 [Note] WSREP: Receiving IST: 1231 writesets, seqnos 234293-235524 2021-06-01 9:05:10 0 [Note] WSREP: Receiving IST... 0.0% ( 0/1231 events) complete. 2021-06-01 9:05:20 0 [Note] WSREP: Receiving IST... 50.7% ( 624/1231 events) complete. 2021-06-01 9:05:29 0 [Note] WSREP: Receiving IST...100.0% (1231/1231 events) complete. 2021-06-01 9:05:29 1 [Note] WSREP: IST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:235524 2021-06-01 9:05:29 0 [Note] WSREP: 0.0 (os-ctrl-1): State transfer from 2.0 (os-ctrl-0) complete. 2021-06-01 9:05:29 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 235887) 2021-06-01 9:05:29 0 [ERROR] WSREP: Trx 235546 tries to abort slave trx 235548. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:05:29 2021-06-01 9:05:29 9 [Warning] WSREP: conflict state after RBR event applying: 1, 235548 [ERROR] mysqld got signal 6 ; 2021-06-01 9:05:29 9 [Warning] WSREP: RBR event apply failed, rolling back: 235548 This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=1 max_threads=10002 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55757396fb8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x5575734048c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fdc63d4f980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fdc6398afb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fdc6398c921] /usr/sbin/mysqld(+0x90a914)[0x55757358e914] /usr/sbin/mysqld(+0x90f83c)[0x55757359383c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x557573216a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fdc63d446db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fdc63a6d71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:05:33 mysqld_safe Number of processes running now: 0 210601 09:05:33 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:05:33 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:05:33 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:05:33 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:05:37 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:235545 2021-06-01 9:05:37 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 249 ... 2021-06-01 9:05:37 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:05:37 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:05:37 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:05:37 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:05:37 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:05:37 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:05:37 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:235545 2021-06-01 9:05:37 0 [Note] WSREP: Assign initial position for certification: 235545, protocol version: -1 2021-06-01 9:05:37 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:05:37 0 [Note] WSREP: Start replication 2021-06-01 9:05:37 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:235545 2021-06-01 9:05:37 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:05:37 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:05:37 0 [Note] WSREP: backend: asio 2021-06-01 9:05:37 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:05:37 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:05:37 0 [Note] WSREP: GMCast version 0 2021-06-01 9:05:37 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:05:37 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:05:37 0 [Note] WSREP: EVS version 0 2021-06-01 9:05:37 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:05:37 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:05:37 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:05:38 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:05:39 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:05:39 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:05:39 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:05:39 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:05:39 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:05:39 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:05:39 0 [Note] WSREP: gcomm: connected 2021-06-01 9:05:39 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:05:39 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:05:39 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:05:39 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:05:40 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.4286S), skipping check 2021-06-01 9:05:40 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:05:40 0 [Note] WSREP: view(view_id(PRIM,913b953a,56) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:05:40 0 [Note] WSREP: save pc into disk 2021-06-01 9:05:40 0 [Note] WSREP: clear restored view 2021-06-01 9:05:40 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:05:40 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 89f0948b-c2b8-11eb-9d75-b7f8fd03d22a 2021-06-01 9:05:40 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 89f0948b-c2b8-11eb-9d75-b7f8fd03d22a 2021-06-01 9:05:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 89f0948b-c2b8-11eb-9d75-b7f8fd03d22a from 0 (os-ctrl-1) 2021-06-01 9:05:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 89f0948b-c2b8-11eb-9d75-b7f8fd03d22a from 1 (os-ctrl-2) 2021-06-01 9:05:40 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:05:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 89f0948b-c2b8-11eb-9d75-b7f8fd03d22a from 2 (os-ctrl-0) 2021-06-01 9:05:40 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 37, members = 2/3 (joined/total), act_id = 235981, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:05:40 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:05:40 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 235981) 2021-06-01 9:05:40 1 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235981 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235545 2021-06-01 9:05:40 1 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:05:40 1 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235981, view# 38: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:05:40 1 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:05:40 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '249' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:235545' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:05:41.067) WSREP_SST: [INFO] Using socat as streamer (20210601 09:05:41.070) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:05:41.098) 2021-06-01 9:05:41 1 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:05:41 1 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:05:41 1 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:05:41 1 [Note] WSREP: Assign initial position for certification: 235981, protocol version: 4 2021-06-01 9:05:41 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:05:41 1 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:05:41 1 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:05:41 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:05:41 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 235987) 2021-06-01 9:05:41 1 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:05:41 1 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:235981 2021-06-01 9:05:41 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:05:41 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:05:41.532) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:235545 0 (20210601 09:05:41.536) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:05:41.538) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:05:41.540) 2021-06-01 9:05:41 0 [Note] WSREP: SST complete, seqno: 235545 2021-06-01 9:05:41 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:05:41 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:05:41 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:05:41 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:05:41 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:05:41 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:05:41 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:05:41 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:05:41 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:05:41 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:05:41 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:05:41 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:05:41 0 [Note] InnoDB: New log files created, LSN=261174545 2021-06-01 9:05:41 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:05:41 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:05:41 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:05:41 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:05:41 0 [Note] InnoDB: Waiting for purge to start 2021-06-01 9:05:41 0 [Note] InnoDB: 10.3.29 started; log sequence number 261174796; transaction id 517457 2021-06-01 9:05:42 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:05:42 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:05:42 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:05:41 2021-06-01 9:05:42 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:05:42 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:05:42 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:235545 2021-06-01 9:05:42 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:05:42 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:05:42 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:05:42 1 [Note] WSREP: Receiving IST: 436 writesets, seqnos 235545-235981 2021-06-01 9:05:42 0 [Note] WSREP: Receiving IST... 0.0% ( 0/436 events) complete. 2021-06-01 9:05:42 0 [ERROR] WSREP: Trx 235579 tries to abort slave trx 235581. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:05:42 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=0 max_threads=10002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 2021-06-01 9:05:42 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561a28f7bb8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x561a28a108c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f8d23200980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f8d22e3bfb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f8d22e3d921] /usr/sbin/mysqld(+0x90a914)[0x561a28b9a914] /usr/sbin/mysqld(+0x90f83c)[0x561a28b9f83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x561a28822a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f8d231f56db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f8d22f1e71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:05:45 mysqld_safe Number of processes running now: 0 210601 09:05:45 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:05:45 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:05:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:05:46 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:05:49 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:235578 2021-06-01 9:05:49 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 250 ... 2021-06-01 9:05:49 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:05:49 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:05:49 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:05:49 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:05:49 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:05:49 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:05:49 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:235578 2021-06-01 9:05:49 0 [Note] WSREP: Assign initial position for certification: 235578, protocol version: -1 2021-06-01 9:05:49 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:05:49 0 [Note] WSREP: Start replication 2021-06-01 9:05:49 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:235578 2021-06-01 9:05:49 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:05:49 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:05:49 0 [Note] WSREP: backend: asio 2021-06-01 9:05:49 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:05:49 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:05:49 0 [Note] WSREP: GMCast version 0 2021-06-01 9:05:49 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:05:49 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:05:49 0 [Note] WSREP: EVS version 0 2021-06-01 9:05:49 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:05:49 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:05:49 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:05:50 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:05:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:05:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:05:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:05:51 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:05:52 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:05:52 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:05:52 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2021-06-01 9:05:52 0 [Note] WSREP: view(view_id(NON_PRIM,913b953a,58) memb { 913b953a,0 } joined { } left { } partitioned { }) 2021-06-01 9:05:52 0 [Note] WSREP: gcomm: connected 2021-06-01 9:05:52 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:05:52 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:05:52 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:05:52 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:05:52 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2021-06-01 9:05:52 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:05:52 0 [Note] WSREP: Flow-control interval: [16, 16] 2021-06-01 9:05:52 0 [Note] WSREP: Received NON-PRIMARY. 2021-06-01 9:05:52 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:05:52 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235578, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1 2021-06-01 9:05:52 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:05:52 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:05:52 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:05:52 0 [Note] WSREP: view(view_id(PRIM,913b953a,59) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:05:52 0 [Note] WSREP: save pc into disk 2021-06-01 9:05:52 0 [Note] WSREP: clear restored view 2021-06-01 9:05:52 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:05:52 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 911763b8-c2b8-11eb-aad9-434a9fe90e9b 2021-06-01 9:05:52 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 911763b8-c2b8-11eb-aad9-434a9fe90e9b 2021-06-01 9:05:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 911763b8-c2b8-11eb-aad9-434a9fe90e9b from 0 (os-ctrl-1) 2021-06-01 9:05:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 911763b8-c2b8-11eb-aad9-434a9fe90e9b from 2 (os-ctrl-0) 2021-06-01 9:05:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 911763b8-c2b8-11eb-aad9-434a9fe90e9b from 1 (os-ctrl-2) 2021-06-01 9:05:52 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 39, members = 2/3 (joined/total), act_id = 236073, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:05:52 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:05:52 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 236073) 2021-06-01 9:05:52 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:236073 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235578 2021-06-01 9:05:52 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:05:52 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:236073, view# 40: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:05:52 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:05:52 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '250' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:235578' WSREP_SST: [INFO] Streaming with xbstream (20210601 09:05:53.090) WSREP_SST: [INFO] Using socat as streamer (20210601 09:05:53.094) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:05:53.134) 2021-06-01 9:05:53 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:05:53 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:05:53 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:05:53 2 [Note] WSREP: Assign initial position for certification: 236073, protocol version: 4 2021-06-01 9:05:53 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:05:53 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:05:53 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:05:53 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:05:53 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 236080) 2021-06-01 9:05:53 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:05:53 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:236073 2021-06-01 9:05:53 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.57569S), skipping check 2021-06-01 9:05:53 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:05:53 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:05:53.602) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:235578 0 (20210601 09:05:53.605) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:05:53.608) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:05:53.610) 2021-06-01 9:05:53 0 [Note] WSREP: SST complete, seqno: 235578 2021-06-01 9:05:53 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:05:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:05:53 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:05:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:05:53 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:05:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:05:53 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:05:53 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:05:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:05:54 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:05:54 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:05:54 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:05:54 0 [Note] InnoDB: New log files created, LSN=261274500 2021-06-01 9:05:54 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:05:54 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:05:54 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:05:54 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:05:54 0 [Note] InnoDB: 10.3.29 started; log sequence number 261274636; transaction id 517678 2021-06-01 9:05:54 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:05:54 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:05:54 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:05:54 2021-06-01 9:05:55 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:05:55 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:05:55 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:235578 2021-06-01 9:05:55 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:05:55 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:05:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:05:55 2 [Note] WSREP: Receiving IST: 495 writesets, seqnos 235578-236073 2021-06-01 9:05:55 0 [Note] WSREP: Receiving IST... 0.0% ( 0/495 events) complete. 2021-06-01 9:05:56 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:05:57 0 [ERROR] WSREP: Trx 235725 tries to abort slave trx 235726. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:05:57 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=1 max_threads=10002 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556c4ec25b8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x556c4e6ba8c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fd2010d7980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fd200d12fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fd200d14921] /usr/sbin/mysqld(+0x90a914)[0x556c4e844914] /usr/sbin/mysqld(+0x90f83c)[0x556c4e84983c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x556c4e4cca83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fd2010cc6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fd200df571f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:06:00 mysqld_safe Number of processes running now: 0 210601 09:06:00 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:06:00 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:06:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:06:01 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:06:05 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:235724 2021-06-01 9:06:05 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 251 ... 2021-06-01 9:06:05 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:06:05 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:06:05 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:06:05 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:06:05 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:06:05 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:06:05 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:235724 2021-06-01 9:06:05 0 [Note] WSREP: Assign initial position for certification: 235724, protocol version: -1 2021-06-01 9:06:05 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:06:05 0 [Note] WSREP: Start replication 2021-06-01 9:06:05 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:235724 2021-06-01 9:06:05 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:06:05 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:06:05 0 [Note] WSREP: backend: asio 2021-06-01 9:06:05 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:06:05 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:06:05 0 [Note] WSREP: GMCast version 0 2021-06-01 9:06:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:06:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:06:05 0 [Note] WSREP: EVS version 0 2021-06-01 9:06:05 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:06:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:06:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:06:05 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:06:06 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:06:06 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:06:06 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:06:06 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:06:06 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:06:06 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:06:06 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:06:06 0 [Note] WSREP: view(view_id(PRIM,913b953a,61) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:06:06 0 [Note] WSREP: save pc into disk 2021-06-01 9:06:06 0 [Note] WSREP: clear restored view 2021-06-01 9:06:07 0 [Note] WSREP: gcomm: connected 2021-06-01 9:06:07 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:06:07 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:06:07 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:06:07 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:06:07 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:06:07 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 9990c58d-c2b8-11eb-b037-efdebdc48b6a 2021-06-01 9:06:07 0 [Warning] WSREP: Action message in non-primary configuration from member 2 2021-06-01 9:06:07 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 9990c58d-c2b8-11eb-b037-efdebdc48b6a 2021-06-01 9:06:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9990c58d-c2b8-11eb-b037-efdebdc48b6a from 0 (os-ctrl-1) 2021-06-01 9:06:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9990c58d-c2b8-11eb-b037-efdebdc48b6a from 1 (os-ctrl-2) 2021-06-01 9:06:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9990c58d-c2b8-11eb-b037-efdebdc48b6a from 2 (os-ctrl-0) 2021-06-01 9:06:07 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 41, members = 2/3 (joined/total), act_id = 236189, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:06:07 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:06:07 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 236189) 2021-06-01 9:06:07 1 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:236189 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:235724 2021-06-01 9:06:07 1 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:06:07 1 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:236189, view# 42: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:06:07 1 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:06:07 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '251' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:235724' 2021-06-01 9:06:07 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.0632S), skipping check WSREP_SST: [INFO] Streaming with xbstream (20210601 09:06:07.288) WSREP_SST: [INFO] Using socat as streamer (20210601 09:06:07.290) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:06:07.314) 2021-06-01 9:06:07 1 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:06:07 1 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:06:07 1 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:06:07 1 [Note] WSREP: Assign initial position for certification: 236189, protocol version: 4 2021-06-01 9:06:07 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:06:07 1 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:06:07 1 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:06:07 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:06:07 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 236191) 2021-06-01 9:06:07 1 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:06:07 1 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:236189 2021-06-01 9:06:07 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:06:07 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:06:07.747) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:235724 0 (20210601 09:06:07.749) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:06:07.752) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:06:07.754) 2021-06-01 9:06:07 0 [Note] WSREP: SST complete, seqno: 235724 2021-06-01 9:06:07 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:06:07 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:06:07 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:06:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:06:07 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:06:07 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:06:07 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:06:07 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:06:07 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:06:07 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:06:07 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:06:07 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:06:07 0 [Note] InnoDB: New log files created, LSN=261530266 2021-06-01 9:06:08 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:06:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:06:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:06:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:06:08 0 [Note] InnoDB: 10.3.29 started; log sequence number 261530636; transaction id 518141 2021-06-01 9:06:08 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:06:08 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:06:08 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:06:08 2021-06-01 9:06:08 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:06:08 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:06:08 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:235724 2021-06-01 9:06:08 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:06:08 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:06:08 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:06:08 1 [Note] WSREP: Receiving IST: 465 writesets, seqnos 235724-236189 2021-06-01 9:06:08 0 [Note] WSREP: Receiving IST... 0.0% ( 0/465 events) complete. 2021-06-01 9:06:10 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:06:15 0 [Note] WSREP: Receiving IST...100.0% (465/465 events) complete. 2021-06-01 9:06:15 1 [Note] WSREP: IST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:236189 2021-06-01 9:06:15 0 [Note] WSREP: 0.0 (os-ctrl-1): State transfer from 2.0 (os-ctrl-0) complete. 2021-06-01 9:06:15 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 236270) 2021-06-01 9:06:16 0 [ERROR] WSREP: Trx 236236 tries to abort slave trx 236238. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. 210601 9:06:16 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.29-MariaDB-1:10.3.29+maria~bionic-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=1 max_threads=10002 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22055535 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5647d689bb8e] /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x5647d63308c5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fd6316ed980] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fd631328fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fd63132a921] /usr/sbin/mysqld(+0x90a914)[0x5647d64ba914] /usr/sbin/mysqld(+0x90f83c)[0x5647d64bf83c] /usr/sbin/mysqld(handle_manager+0x1f3)[0x5647d6142a83] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fd6316e26db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fd63140b71f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64055 64055 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 210601 09:06:18 mysqld_safe Number of processes running now: 0 210601 09:06:18 mysqld_safe WSREP: not restarting wsrep node automatically 210601 09:06:18 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210601 09:06:19 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210601 09:06:19 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//os-ctrl-1-recover.pid' 210601 09:06:24 mysqld_safe WSREP: Recovered position 8848e2e7-c217-11eb-adfb-4a913920f98c:236235 2021-06-01 9:06:24 0 [Note] /usr/sbin/mysqld (mysqld 10.3.29-MariaDB-1:10.3.29+maria~bionic-log) starting as process 248 ... 2021-06-01 9:06:24 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-06-01 9:06:24 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2021-06-01 9:06:24 0 [Note] WSREP: wsrep_load(): Galera 25.3.33(r15123524) by Codership Oy loaded successfully. 2021-06-01 9:06:24 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-06-01 9:06:24 0 [Note] WSREP: Found saved state: 8848e2e7-c217-11eb-adfb-4a913920f98c:-1, safe_to_bootstrap: 0 2021-06-01 9:06:24 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.33.2.33; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.33.2.33:4567; gmcast.segment = 0; gmcas 2021-06-01 9:06:24 0 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:236235 2021-06-01 9:06:24 0 [Note] WSREP: Assign initial position for certification: 236235, protocol version: -1 2021-06-01 9:06:24 0 [Note] WSREP: wsrep_sst_grab() 2021-06-01 9:06:24 0 [Note] WSREP: Start replication 2021-06-01 9:06:24 0 [Note] WSREP: Setting initial position to 8848e2e7-c217-11eb-adfb-4a913920f98c:236235 2021-06-01 9:06:24 0 [Note] WSREP: protonet asio version 0 2021-06-01 9:06:24 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-06-01 9:06:24 0 [Note] WSREP: backend: asio 2021-06-01 9:06:24 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-06-01 9:06:24 0 [Note] WSREP: restore pc from disk successfully 2021-06-01 9:06:24 0 [Note] WSREP: GMCast version 0 2021-06-01 9:06:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') listening at tcp://10.33.2.33:4567 2021-06-01 9:06:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') multicast: , ttl: 1 2021-06-01 9:06:24 0 [Note] WSREP: EVS version 0 2021-06-01 9:06:24 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.33.0.13:4567,10.33.2.33:4567,10.33.3.175:4567' 2021-06-01 9:06:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:06:24 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:06:25 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting on, nonlive peers: tcp://10.33.0.13:4567 tcp://10.33.3.175:4567 2021-06-01 9:06:26 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to a9b132d9 (tcp://10.33.0.13:4567), attempt 0 2021-06-01 9:06:26 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') reconnecting to 91e3db36 (tcp://10.33.3.175:4567), attempt 0 2021-06-01 9:06:26 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to 91e3db36 tcp://10.33.3.175:4567 2021-06-01 9:06:26 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') connection established to a9b132d9 tcp://10.33.0.13:4567 2021-06-01 9:06:26 0 [Note] WSREP: declaring 91e3db36 at tcp://10.33.3.175:4567 stable 2021-06-01 9:06:26 0 [Note] WSREP: declaring a9b132d9 at tcp://10.33.0.13:4567 stable 2021-06-01 9:06:26 0 [Note] WSREP: Node 91e3db36 state prim 2021-06-01 9:06:26 0 [Note] WSREP: view(view_id(PRIM,913b953a,63) memb { 913b953a,0 91e3db36,0 a9b132d9,0 } joined { } left { } partitioned { }) 2021-06-01 9:06:26 0 [Note] WSREP: save pc into disk 2021-06-01 9:06:26 0 [Note] WSREP: clear restored view 2021-06-01 9:06:26 0 [Note] WSREP: gcomm: connected 2021-06-01 9:06:26 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-06-01 9:06:26 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-06-01 9:06:26 0 [Note] WSREP: Opened channel 'openstack' 2021-06-01 9:06:26 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2021-06-01 9:06:26 0 [Note] WSREP: Waiting for SST to complete. 2021-06-01 9:06:26 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: a543eabc-c2b8-11eb-8c4e-c7357d5d2440 2021-06-01 9:06:26 0 [Note] WSREP: STATE EXCHANGE: sent state msg: a543eabc-c2b8-11eb-8c4e-c7357d5d2440 2021-06-01 9:06:26 0 [Note] WSREP: STATE EXCHANGE: got state msg: a543eabc-c2b8-11eb-8c4e-c7357d5d2440 from 0 (os-ctrl-1) 2021-06-01 9:06:26 0 [Note] WSREP: STATE EXCHANGE: got state msg: a543eabc-c2b8-11eb-8c4e-c7357d5d2440 from 1 (os-ctrl-2) 2021-06-01 9:06:26 0 [Note] WSREP: STATE EXCHANGE: got state msg: a543eabc-c2b8-11eb-8c4e-c7357d5d2440 from 2 (os-ctrl-0) 2021-06-01 9:06:26 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 43, members = 2/3 (joined/total), act_id = 236374, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 8848e2e7-c217-11eb-adfb-4a913920f98c 2021-06-01 9:06:26 0 [Note] WSREP: Flow-control interval: [28, 28] 2021-06-01 9:06:26 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 236374) 2021-06-01 9:06:26 2 [Note] WSREP: State transfer required: Group state: 8848e2e7-c217-11eb-adfb-4a913920f98c:236374 Local state: 8848e2e7-c217-11eb-adfb-4a913920f98c:236235 2021-06-01 9:06:26 2 [Note] WSREP: REPL Protocols: 9 (4, 2) 2021-06-01 9:06:26 2 [Note] WSREP: New cluster view: global state: 8848e2e7-c217-11eb-adfb-4a913920f98c:236374, view# 44: Primary, number of nodes: 3, my index: 0, protocol version 3 2021-06-01 9:06:26 2 [Warning] WSREP: Gap in state sequence. Need state transfer. 2021-06-01 9:06:26 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.33.2.33:4444' --datadir '/var/lib/mysql/' --parent '248' --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=8848e2e7-c217-11eb-adfb-4a913920f98c:236235' 2021-06-01 9:06:26 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.09602S), skipping check WSREP_SST: [INFO] Streaming with xbstream (20210601 09:06:26.884) WSREP_SST: [INFO] Using socat as streamer (20210601 09:06:26.887) WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20210601 09:06:26.913) 2021-06-01 9:06:27 2 [Note] WSREP: Prepared SST request: mariabackup|10.33.2.33:4444/xtrabackup_sst//1 ERROR 2002 (HY000): Can't connect to MySQL server on '10.33.2.33' (115) 2021-06-01 9:06:27 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306: 1 (Operation not permitted) 2021-06-01 9:06:27 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status Joiner --uuid 8848e2e7-c217-11eb-adfb-4a913920f98c --primary yes --index 0 --members 913b953a-c217-11eb-b0cb-5f318f54bd7d/os-ctrl-1/10.33.2.33:3306,91e3db36-c217-11eb-a61b-9fc24a9a0213/os-ctrl-2/10.33.3.175:3306,a9b132d9-c217-11eb-bb4b-86eae2403081/os-ctrl-0/10.33.0.13:3306" 2021-06-01 9:06:27 2 [Note] WSREP: Assign initial position for certification: 236374, protocol version: 4 2021-06-01 9:06:27 0 [Note] WSREP: Service thread queue flushed. 2021-06-01 9:06:27 2 [Note] WSREP: IST receiver addr using tcp://10.33.2.33:4568 2021-06-01 9:06:27 2 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.33.2.33:4568 2021-06-01 9:06:27 0 [Note] WSREP: Member 0.0 (os-ctrl-1) requested state transfer from '*any*'. Selected 2.0 (os-ctrl-0)(SYNCED) as donor. 2021-06-01 9:06:27 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 236379) 2021-06-01 9:06:27 2 [Note] WSREP: Requesting state transfer: success, donor: 2 2021-06-01 9:06:27 2 [Note] WSREP: GCache history reset: 8848e2e7-c217-11eb-adfb-4a913920f98c:0 -> 8848e2e7-c217-11eb-adfb-4a913920f98c:236374 2021-06-01 9:06:27 0 [Note] WSREP: 2.0 (os-ctrl-0): State transfer to 0.0 (os-ctrl-1) complete. 2021-06-01 9:06:27 0 [Note] WSREP: Member 2.0 (os-ctrl-0) synced with group. WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20210601 09:06:27.315) WSREP_SST: [INFO] Galera co-ords from recovery: 8848e2e7-c217-11eb-adfb-4a913920f98c:236235 0 (20210601 09:06:27.319) WSREP_SST: [INFO] Total time on joiner: 0 seconds (20210601 09:06:27.322) WSREP_SST: [INFO] Removing the sst_in_progress file (20210601 09:06:27.323) 2021-06-01 9:06:27 0 [Note] WSREP: SST complete, seqno: 236235 2021-06-01 9:06:27 0 [Note] InnoDB: Using Linux native AIO 2021-06-01 9:06:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-06-01 9:06:27 0 [Note] InnoDB: Uses event mutexes 2021-06-01 9:06:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-06-01 9:06:27 0 [Note] InnoDB: Number of pools: 1 2021-06-01 9:06:27 0 [Note] InnoDB: Using SSE2 crc32 instructions 2021-06-01 9:06:27 0 [Note] InnoDB: Initializing buffer pool, total size = 7G, instances = 8, chunk size = 128M 2021-06-01 9:06:27 0 [Note] InnoDB: Completed initialization of buffer pool 2021-06-01 9:06:27 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-06-01 9:06:27 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes 2021-06-01 9:06:27 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes 2021-06-01 9:06:27 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2021-06-01 9:06:27 0 [Note] InnoDB: New log files created, LSN=262330404 2021-06-01 9:06:27 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-06-01 9:06:27 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-06-01 9:06:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-06-01 9:06:27 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-06-01 9:06:27 0 [Note] InnoDB: 10.3.29 started; log sequence number 262330892; transaction id 519358 2021-06-01 9:06:27 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2021-06-01 9:06:27 0 [Note] InnoDB: Buffer pool(s) load completed at 210601 9:06:27 2021-06-01 9:06:27 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-06-01 9:06:27 0 [Note] Server socket created on IP: '10.33.2.33'. 2021-06-01 9:06:27 0 [Note] WSREP: Signalling provider to continue. 2021-06-01 9:06:27 0 [Note] WSREP: SST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:236235 2021-06-01 9:06:27 0 [Note] Reading of all Master_info entries succeeded 2021-06-01 9:06:27 0 [Note] Added new Master_info '' to hash table 2021-06-01 9:06:27 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-01 9:06:27 2 [Note] WSREP: Receiving IST: 139 writesets, seqnos 236235-236374 2021-06-01 9:06:27 0 [Note] WSREP: Receiving IST... 0.0% ( 0/139 events) complete. 2021-06-01 9:06:29 0 [Note] WSREP: (913b953a, 'tcp://10.33.2.33:4567') turning message relay requesting off 2021-06-01 9:06:30 0 [Note] WSREP: Receiving IST...100.0% (139/139 events) complete. 2021-06-01 9:06:30 2 [Note] WSREP: IST received: 8848e2e7-c217-11eb-adfb-4a913920f98c:236374 2021-06-01 9:06:30 0 [Note] WSREP: 0.0 (os-ctrl-1): State transfer from 2.0 (os-ctrl-0) complete. 2021-06-01 9:06:30 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 236426) 2021-06-01 9:06:30 0 [Note] WSREP: Member 0.0 (os-ctrl-1) synced with group. 2021-06-01 9:06:30 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 236437) 2021-06-01 9:06:31 2 [Note] WSREP: Synchronized with group, ready for connections