2019-07-30 8:15:48 0 [Note] mysqld (mysqld 10.4.6-MariaDB-1:10.4.6+maria~bionic) starting as process 1 ... 2019-07-30 8:15:48 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1 wsrep loader: [INFO] wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' wsrep loader: [INFO] wsrep_load(): Galera 26.4.2(r4498) by Codership Oy loaded successfully. 2019-07-30 8:15:48 0 [Note] WSREP: CRC-32C: using hardware acceleration. 2019-07-30 8:15:48 0 [Note] WSREP: Found saved state: 59d93b0e-b04c-11e9-89e1-37592b577569:317815408, safe_to_bootstrap: 0 2019-07-30 8:15:48 0 [Note] WSREP: GCache DEBUG: opened preamble: Version: 2 UUID: 59d93b0e-b04c-11e9-89e1-37592b577569 Seqno: 316244029 - 317815408 Offset: 430785064 Synced: 1 2019-07-30 8:15:48 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 59d93b0e-b04c-11e9-89e1-37592b577569, offset: 430785064 2019-07-30 8:15:48 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/2147483672 bytes) complete. 2019-07-30 8:15:49 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (2147483672/2147483672 bytes) complete. 2019-07-30 8:15:49 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 316244029-317815408 2019-07-30 8:15:49 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/2147479488 bytes) complete. 2019-07-30 8:15:49 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (2147479488/2147479488 bytes) complete. 2019-07-30 8:15:49 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 0/1571380 locked buffers 2019-07-30 8:15:49 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): used space: 2147479488/2147483648 2019-07-30 8:15:49 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = dbcluster03.our-domain.local; base_port = 5577; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; ist.recv_addr = dbcluster03.j 2019-07-30 8:15:49 0 [Note] WSREP: Start replication 2019-07-30 8:15:49 0 [Note] WSREP: Connecting with bootstrap option: 0 2019-07-30 8:15:49 0 [Note] WSREP: Setting GCS initial position to 59d93b0e-b04c-11e9-89e1-37592b577569:317815408 2019-07-30 8:15:49 0 [Note] WSREP: protonet asio version 0 2019-07-30 8:15:49 0 [Note] WSREP: Using CRC-32C for message checksums. 2019-07-30 8:15:49 0 [Note] WSREP: backend: asio 2019-07-30 8:15:49 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-07-30 8:15:49 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) 2019-07-30 8:15:49 0 [Note] WSREP: restore pc from disk failed 2019-07-30 8:15:49 0 [Note] WSREP: GMCast version 0 2019-07-30 8:15:49 0 [Note] WSREP: (3d7af5fb, 'tcp://0.0.0.0:5577') listening at tcp://0.0.0.0:5577 2019-07-30 8:15:49 0 [Note] WSREP: (3d7af5fb, 'tcp://0.0.0.0:5577') multicast: , ttl: 1 2019-07-30 8:15:49 0 [Note] WSREP: EVS version 1 2019-07-30 8:15:49 0 [Note] WSREP: gcomm: connecting to group 'mysqlext1', peer 'dbcluster01.our-domain.local:5577,dbcluster02.our-domain.local:5577,dbcluster03.our-domain.local:5577' 2019-07-30 8:15:49 0 [Note] WSREP: (3d7af5fb, 'tcp://0.0.0.0:5577') connection established to 2b762b2f tcp://172.16.0.151:5577 2019-07-30 8:15:49 0 [Note] WSREP: (3d7af5fb, 'tcp://0.0.0.0:5577') connection established to 3d7af5fb tcp://192.168.0.1:5577 2019-07-30 8:15:49 0 [Note] WSREP: (3d7af5fb, 'tcp://0.0.0.0:5577') turning message relay requesting on, nonlive peers: 2019-07-30 8:15:49 0 [Note] WSREP: (3d7af5fb, 'tcp://0.0.0.0:5577') connection established to 1fb571ea tcp://172.16.0.152:5577 2019-07-30 8:15:50 0 [Note] WSREP: EVS version upgrade 0 -> 1 2019-07-30 8:15:50 0 [Note] WSREP: declaring 1fb571ea at tcp://172.16.0.152:5577 stable 2019-07-30 8:15:50 0 [Note] WSREP: declaring 2b762b2f at tcp://172.16.0.151:5577 stable 2019-07-30 8:15:50 0 [Note] WSREP: PC protocol upgrade 0 -> 1 2019-07-30 8:15:50 0 [Note] WSREP: Node 1fb571ea state prim 2019-07-30 8:15:50 0 [Note] WSREP: view(view_id(PRIM,1fb571ea,5) memb { 1fb571ea,0 2b762b2f,0 3d7af5fb,0 } joined { } left { } partitioned { }) 2019-07-30 8:15:50 0 [Note] WSREP: save pc into disk 2019-07-30 8:15:50 0 [Note] WSREP: discarding pending addr without UUID: tcp://172.16.0.153:5577 2019-07-30 8:15:50 0 [Note] WSREP: discarding pending addr proto entry 0x5632446dadc0 2019-07-30 8:15:50 0 [Note] WSREP: gcomm: connected 2019-07-30 8:15:50 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2019-07-30 8:15:50 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2019-07-30 8:15:50 0 [Note] WSREP: Opened channel 'mysqlext1' 2019-07-30 8:15:50 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3 2019-07-30 8:15:50 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2019-07-30 8:15:50 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 3dc81324-b2a2-11e9-80a9-0bd072fb83b0 2019-07-30 8:15:50 0 [Note] WSREP: STATE EXCHANGE: got state msg: 3dc81324-b2a2-11e9-80a9-0bd072fb83b0 from 0 (dbcluster02.our-domain.local) 2019-07-30 8:15:50 0 [Note] WSREP: STATE EXCHANGE: got state msg: 3dc81324-b2a2-11e9-80a9-0bd072fb83b0 from 1 (dbcluster01.our-domain.local) 2019-07-30 8:15:50 1 [Note] WSREP: Starting rollbacker thread 1 2019-07-30 8:15:50 2 [Note] WSREP: Starting post rollbacker thread 2 2019-07-30 8:15:50 3 [Note] WSREP: Starting applier thread 3 2019-07-30 8:15:50 0 [Note] WSREP: STATE EXCHANGE: got state msg: 3dc81324-b2a2-11e9-80a9-0bd072fb83b0 from 2 (dbcluster03.our-domain.local) 2019-07-30 8:15:50 0 [Note] WSREP: Quorum results: version = 5, component = PRIMARY, conf_id = 4, members = 2/3 (joined/total), act_id = 317861583, last_appl. = 317861500, protocols = 1/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 59d93b0e-b04c-11e9-89e1-37592b577569 2019-07-30 8:15:50 0 [Note] WSREP: Writing down CC checksum: 6a45406a b9d13307 6291df65 187661b6 at offset 304 2019-07-30 8:15:50 0 [Note] WSREP: Flow-control interval: [28, 28] 2019-07-30 8:15:50 0 [Note] WSREP: Trying to continue unpaused monitor 2019-07-30 8:15:50 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 317861584) 2019-07-30 8:15:50 3 [Note] WSREP: ####### processing CC 317861584, local, ordered 2019-07-30 8:15:50 3 [Note] WSREP: ####### drain monitors upto -1 2019-07-30 8:15:50 0 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000000 of size 134217728 bytes 2019-07-30 8:15:52 0 [Note] WSREP: (3d7af5fb, 'tcp://0.0.0.0:5577') turning message relay requesting off 2019-07-30 8:16:11 3 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-07-30 8:16:11 3 [Note] WSREP: ####### My UUID: 3d7af5fb-b2a2-11e9-97cb-3bb007806590 2019-07-30 8:16:11 3 [Note] WSREP: Server dbcluster03.our-domain.local connected to cluster at position 59d93b0e-b04c-11e9-89e1-37592b577569:317861584 with ID 3d7af5fb-b2a2-11e9-97cb-3bb007806590 2019-07-30 8:16:11 3 [Note] WSREP: Server status change disconnected -> connected 2019-07-30 8:16:11 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-07-30 8:16:11 3 [Note] WSREP: State transfer required: Group state: 59d93b0e-b04c-11e9-89e1-37592b577569:317861584 Local state: 59d93b0e-b04c-11e9-89e1-37592b577569:317815408 2019-07-30 8:16:11 3 [Note] WSREP: Server status change connected -> joiner 2019-07-30 8:16:11 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-07-30 8:16:11 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'dbcluster03.our-domain.local:5454' --datadir '/var/lib/mysql/' --parent '1' '' ''' WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20190730 08:16:12.362) 2019-07-30 8:16:12 3 [Note] WSREP: Prepared SST request: mariabackup|dbcluster03.our-domain.local:5454/xtrabackup_sst//1 2019-07-30 8:16:12 3 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:0 (proto: 10), state transfer needed: yes 2019-07-30 8:16:12 0 [Note] WSREP: Service thread queue flushed. 2019-07-30 8:16:12 3 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:0, protocol version: 5 2019-07-30 8:16:12 3 [Note] WSREP: ####### IST uuid:59d93b0e-b04c-11e9-89e1-37592b577569 f: 317815409, l: 317861584, STRv: 3 2019-07-30 8:16:12 3 [Note] WSREP: IST receiver addr using tcp://dbcluster03.our-domain.local:5578 2019-07-30 8:16:12 3 [Note] WSREP: IST receiver bind using tcp://0.0.0.0:5578 2019-07-30 8:16:12 3 [Note] WSREP: Prepared IST receiver for 317815409-317861584, listening at: tcp://0.0.0.0:5578 2019-07-30 8:16:12 0 [Note] WSREP: Member 2.0 (dbcluster03.our-domain.local) requested state transfer from '*any*'. Selected 1.0 (dbcluster01.our-domain.local)(SYNCED) as donor. 2019-07-30 8:16:12 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 317861646) 2019-07-30 8:16:12 3 [Note] WSREP: Requesting state transfer: success, donor: 1 2019-07-30 8:17:52 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address 'dbcluster03.our-domain.local:5454' --datadir '/var/lib/mysql/' --parent '1' '' '': 32 (Broken pipe) 2019-07-30 8:17:52 0 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script. 2019-07-30 8:17:52 4 [Note] WSREP: SST received 2019-07-30 8:17:52 4 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1 2019-07-30 8:17:52 3 [ERROR] WSREP: Application received wrong state: Received: 00000000-0000-0000-0000-000000000000 Required: 59d93b0e-b04c-11e9-89e1-37592b577569 2019-07-30 8:17:52 3 [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required. 2019-07-30 8:17:52 3 [Note] WSREP: ReplicatorSMM::abort() 2019-07-30 8:17:52 3 [Note] WSREP: Closing send monitor... 2019-07-30 8:17:52 3 [Note] WSREP: Closed send monitor. 2019-07-30 8:17:52 3 [Note] WSREP: gcomm: terminating thread 2019-07-30 8:17:52 3 [Note] WSREP: gcomm: joining thread 2019-07-30 8:17:52 3 [Note] WSREP: gcomm: closing backend 2019-07-30 8:17:52 3 [Note] WSREP: view(view_id(NON_PRIM,1fb571ea,5) memb { 3d7af5fb,0 } joined { } left { } partitioned { 1fb571ea,0 2b762b2f,0 }) 2019-07-30 8:17:52 3 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-07-30 8:17:52 3 [Note] WSREP: view((empty)) 2019-07-30 8:17:52 3 [Note] WSREP: gcomm: closed 2019-07-30 8:17:52 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-07-30 8:17:52 0 [Note] WSREP: Writing down CC checksum: 02cbcf40 cc496178 6c79d99f 51a40b6b at offset 144 2019-07-30 8:17:52 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-07-30 8:17:52 0 [Note] WSREP: Trying to continue unpaused monitor 2019-07-30 8:17:52 0 [Note] WSREP: Received NON-PRIMARY. 2019-07-30 8:17:52 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 317861726) 2019-07-30 8:17:52 0 [Note] WSREP: Received self-leave message. 2019-07-30 8:17:52 0 [Note] WSREP: Writing down CC checksum: a08c7896 e166bd46 b78e0528 3ccc9afe at offset 64 2019-07-30 8:17:52 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-07-30 8:17:52 0 [Note] WSREP: Trying to continue unpaused monitor 2019-07-30 8:17:52 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-07-30 8:17:52 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-07-30 8:17:52 0 [Note] WSREP: RECV thread exiting 0: Success 2019-07-30 8:17:52 3 [Note] WSREP: recv_thread() joined. 2019-07-30 8:17:52 3 [Note] WSREP: Closing replication queue. 2019-07-30 8:17:52 3 [Note] WSREP: Closing slave action queue. 2019-07-30 8:17:52 3 [Note] WSREP: mysqld: Terminated. 190730 8:17:52 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic key_buffer_size=0 read_buffer_size=67108864 max_used_connections=0 max_threads=1502 thread_count=3 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 196907000 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7f34000c08 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7fcc07f9f8 thread_stack 0x49000 mysqld(my_print_stacktrace+0x2e)[0x56323c277e0e] mysqld(handle_fatal_signal+0x515)[0x56323bcf0f95] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f7fcf101890] /lib/x86_64-linux-gnu/libc.so.6(abort+0x230)[0x7f7fce4138f0] /usr/lib/galera/libgalera_smm.so(+0x71233)[0x7f7fccc06233] /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM5abortEv+0x91)[0x7f7fccdcbdf1] /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM22request_state_transferEPvRK10wsrep_uuidlPKvl+0x366)[0x7f7fccde9206] /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM19process_conf_changeEPvRK10gcs_action+0xcaf)[0x7f7fccdd705f] /usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x118)[0x7f7fccda95f8] /usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0xb8)[0x7f7fccda9858] /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x120)[0x7f7fccdd1800] /usr/lib/galera/libgalera_smm.so(galera_recv+0x2b)[0x7f7fccdef4db] mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x56323c2fce8e] mysqld(+0x7cc422)[0x56323bc68422] mysqld(_Z15start_wsrep_THDPv+0x33c)[0x56323bc56c2c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f7fcf0f66db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f7fce4f488f] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 3 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. We think the query pointer is invalid, but we will try to print it anyway. Query: Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 0 bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 1028962 1028962 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 2019-07-30 8:17:54 0 [Note] mysqld (mysqld 10.4.6-MariaDB-1:10.4.6+maria~bionic) starting as process 1 ... 2019-07-30 8:17:54 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1 wsrep loader: [INFO] wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' wsrep loader: [INFO] wsrep_load(): Galera 26.4.2(r4498) by Codership Oy loaded successfully. 2019-07-30 8:17:54 0 [Note] WSREP: CRC-32C: using hardware acceleration. 2019-07-30 8:17:54 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 0 2019-07-30 8:17:54 0 [Note] WSREP: GCache DEBUG: opened preamble: Version: 2 UUID: 59d93b0e-b04c-11e9-89e1-37592b577569 Seqno: -1 - -1 Offset: -1 Synced: 0 2019-07-30 8:17:54 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 59d93b0e-b04c-11e9-89e1-37592b577569, offset: -1 2019-07-30 8:17:54 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/2147483672 bytes) complete. 2019-07-30 8:17:55 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (2147483672/2147483672 bytes) complete. 2019-07-30 8:17:55 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 316244031-317815408 2019-07-30 8:17:55 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/2147475928 bytes) complete. 2019-07-30 8:17:56 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (2147475928/2147475928 bytes) complete. 2019-07-30 8:17:56 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 0/1571378 locked buffers 2019-07-30 8:17:56 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): used space: 2147475928/2147483648 2019-07-30 8:17:56 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = dbcluster03.our-domain.local; base_port = 5577; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; ist.recv_addr = dbcluster03.j 2019-07-30 8:17:56 0 [Note] WSREP: GCache history reset: old(59d93b0e-b04c-11e9-89e1-37592b577569:317815408 -> 00000000-0000-0000-0000-000000000000:-1 2019-07-30 8:17:56 0 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): full reset 2019-07-30 8:17:56 0 [Note] WSREP: Start replication 2019-07-30 8:17:56 0 [Note] WSREP: Connecting with bootstrap option: 0 2019-07-30 8:17:56 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1 2019-07-30 8:17:56 0 [Note] WSREP: protonet asio version 0 2019-07-30 8:17:56 0 [Note] WSREP: Using CRC-32C for message checksums. 2019-07-30 8:17:56 0 [Note] WSREP: backend: asio 2019-07-30 8:17:56 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-07-30 8:17:56 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) 2019-07-30 8:17:56 0 [Note] WSREP: restore pc from disk failed 2019-07-30 8:17:56 0 [Note] WSREP: GMCast version 0 2019-07-30 8:17:56 0 [Note] WSREP: (88ce85d0, 'tcp://0.0.0.0:5577') listening at tcp://0.0.0.0:5577 2019-07-30 8:17:56 0 [Note] WSREP: (88ce85d0, 'tcp://0.0.0.0:5577') multicast: , ttl: 1 2019-07-30 8:17:56 0 [Note] WSREP: EVS version 1 2019-07-30 8:17:56 0 [Note] WSREP: gcomm: connecting to group 'mysqlext1', peer 'dbcluster01.our-domain.local:5577,dbcluster02.our-domain.local:5577,dbcluster03.our-domain.local:5577' 2019-07-30 8:17:56 0 [Note] WSREP: (88ce85d0, 'tcp://0.0.0.0:5577') connection established to 2b762b2f tcp://172.16.0.151:5577 2019-07-30 8:17:56 0 [Note] WSREP: (88ce85d0, 'tcp://0.0.0.0:5577') turning message relay requesting on, nonlive peers: 2019-07-30 8:17:56 0 [Note] WSREP: (88ce85d0, 'tcp://0.0.0.0:5577') connection established to 88ce85d0 tcp://192.168.0.1:5577 2019-07-30 8:17:56 0 [Note] WSREP: (88ce85d0, 'tcp://0.0.0.0:5577') connection established to 1fb571ea tcp://172.16.0.152:5577 2019-07-30 8:17:56 0 [Note] WSREP: EVS version upgrade 0 -> 1 2019-07-30 8:17:56 0 [Note] WSREP: declaring 1fb571ea at tcp://172.16.0.152:5577 stable 2019-07-30 8:17:56 0 [Note] WSREP: declaring 2b762b2f at tcp://172.16.0.151:5577 stable 2019-07-30 8:17:56 0 [Note] WSREP: PC protocol upgrade 0 -> 1 2019-07-30 8:17:56 0 [Note] WSREP: Node 1fb571ea state prim 2019-07-30 8:17:56 0 [Note] WSREP: view(view_id(PRIM,1fb571ea,7) memb { 1fb571ea,0 2b762b2f,0 88ce85d0,0 } joined { } left { } partitioned { }) 2019-07-30 8:17:56 0 [Note] WSREP: save pc into disk 2019-07-30 8:17:56 0 [Note] WSREP: discarding pending addr without UUID: tcp://172.16.0.153:5577 2019-07-30 8:17:56 0 [Note] WSREP: discarding pending addr proto entry 0x561f7d04d760 2019-07-30 8:17:57 0 [Note] WSREP: gcomm: connected 2019-07-30 8:17:57 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2019-07-30 8:17:57 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2019-07-30 8:17:57 0 [Note] WSREP: Opened channel 'mysqlext1' 2019-07-30 8:17:57 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3 2019-07-30 8:17:57 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2019-07-30 8:17:57 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 892e48f5-b2a2-11e9-b32c-cbd8b020e16f 2019-07-30 8:17:57 0 [Note] WSREP: STATE EXCHANGE: got state msg: 892e48f5-b2a2-11e9-b32c-cbd8b020e16f from 1 (dbcluster01.our-domain.local) 2019-07-30 8:17:57 0 [Note] WSREP: STATE EXCHANGE: got state msg: 892e48f5-b2a2-11e9-b32c-cbd8b020e16f from 0 (dbcluster02.our-domain.local) 2019-07-30 8:17:57 2 [Note] WSREP: Starting post rollbacker thread 2 2019-07-30 8:17:57 1 [Note] WSREP: Starting rollbacker thread 1 2019-07-30 8:17:57 3 [Note] WSREP: Starting applier thread 3 2019-07-30 8:17:57 0 [Note] WSREP: STATE EXCHANGE: got state msg: 892e48f5-b2a2-11e9-b32c-cbd8b020e16f from 2 (dbcluster03.our-domain.local) 2019-07-30 8:17:57 0 [Note] WSREP: Quorum results: version = 5, component = PRIMARY, conf_id = 6, members = 2/3 (joined/total), act_id = 317861734, last_appl. = 317861500, protocols = 1/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 59d93b0e-b04c-11e9-89e1-37592b577569 2019-07-30 8:17:57 0 [Note] WSREP: Writing down CC checksum: 89f45d39 e215aaaa 15f0d78a b1b73ce8 at offset 304 2019-07-30 8:17:57 0 [Note] WSREP: Flow-control interval: [28, 28] 2019-07-30 8:17:57 0 [Note] WSREP: Trying to continue unpaused monitor 2019-07-30 8:17:57 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 317861735) 2019-07-30 8:17:57 3 [Note] WSREP: ####### processing CC 317861735, local, ordered 2019-07-30 8:17:57 3 [Note] WSREP: ####### drain monitors upto -1 2019-07-30 8:17:57 3 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-07-30 8:17:57 3 [Note] WSREP: ####### My UUID: 88ce85d0-b2a2-11e9-8b6e-da4ac98b904e 2019-07-30 8:17:57 3 [Note] WSREP: Server dbcluster03.our-domain.local connected to cluster at position 59d93b0e-b04c-11e9-89e1-37592b577569:317861735 with ID 88ce85d0-b2a2-11e9-8b6e-da4ac98b904e 2019-07-30 8:17:57 3 [Note] WSREP: Server status change disconnected -> connected 2019-07-30 8:17:57 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-07-30 8:17:57 3 [Note] WSREP: State transfer required: Group state: 59d93b0e-b04c-11e9-89e1-37592b577569:317861735 Local state: 00000000-0000-0000-0000-000000000000:-1 2019-07-30 8:17:57 3 [Note] WSREP: Server status change connected -> joiner 2019-07-30 8:17:57 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-07-30 8:17:57 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'dbcluster03.our-domain.local:5454' --datadir '/var/lib/mysql/' --parent '1' '' ''' WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20190730 08:17:57.602) 2019-07-30 8:17:57 3 [Note] WSREP: Prepared SST request: mariabackup|dbcluster03.our-domain.local:5454/xtrabackup_sst//1 2019-07-30 8:17:57 3 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:0 (proto: 10), state transfer needed: yes 2019-07-30 8:17:57 0 [Note] WSREP: Service thread queue flushed. 2019-07-30 8:17:57 3 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:0, protocol version: 5 2019-07-30 8:17:57 3 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 317861735, STRv: 3 2019-07-30 8:17:57 3 [Note] WSREP: IST receiver addr using tcp://dbcluster03.our-domain.local:5578 2019-07-30 8:17:57 3 [Note] WSREP: IST receiver bind using tcp://0.0.0.0:5578 2019-07-30 8:17:57 3 [Note] WSREP: Prepared IST receiver for 0-317861735, listening at: tcp://0.0.0.0:5578 2019-07-30 8:17:57 0 [Note] WSREP: Member 2.0 (dbcluster03.our-domain.local) requested state transfer from '*any*'. Selected 0.0 (dbcluster02.our-domain.local)(SYNCED) as donor. 2019-07-30 8:17:57 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 317861735) 2019-07-30 8:17:57 3 [Note] WSREP: Requesting state transfer: success, donor: 0 2019-07-30 8:17:57 3 [Note] WSREP: Resetting GCache seqno map due to different histories. 2019-07-30 8:17:57 3 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0 -> 59d93b0e-b04c-11e9-89e1-37592b577569:317861735 2019-07-30 8:17:59 0 [Note] WSREP: (88ce85d0, 'tcp://0.0.0.0:5577') turning message relay requesting off 2019-07-30 8:22:40 0 [Note] WSREP: 0.0 (dbcluster02.our-domain.local): State transfer to 2.0 (dbcluster03.our-domain.local) complete. 2019-07-30 8:22:40 0 [Note] WSREP: Member 0.0 (dbcluster02.our-domain.local) synced with group. 2019-07-30 8:22:50 4 [Note] WSREP: SST received 2019-07-30 8:22:50 4 [Note] WSREP: Server status change joiner -> initializing 2019-07-30 8:22:50 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-07-30 8:22:50 0 [Note] mysqld: Aria engine: starting recovery recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (0.0 seconds); tables to flush: 3 2 1 0 (0.0 seconds); 2019-07-30 8:22:50 0 [Note] mysqld: Aria engine: recovery done 2019-07-30 8:22:50 0 [Note] InnoDB: Using Linux native AIO 2019-07-30 8:22:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-07-30 8:22:50 0 [Note] InnoDB: Uses event mutexes 2019-07-30 8:22:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2019-07-30 8:22:50 0 [Note] InnoDB: Number of pools: 1 2019-07-30 8:22:50 0 [Note] InnoDB: Using SSE2 crc32 instructions 2019-07-30 8:22:50 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) 2019-07-30 8:22:50 0 [Note] InnoDB: Initializing buffer pool, total size = 66G, instances = 24, chunk size = 128M 2019-07-30 8:22:52 0 [Note] InnoDB: Completed initialization of buffer pool 2019-07-30 8:22:52 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2019-07-30 8:22:52 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 1073741824 bytes 2019-07-30 8:22:52 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 1073741824 bytes 2019-07-30 8:22:52 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2019-07-30 8:22:52 0 [Note] InnoDB: New log files created, LSN=398314177731 2019-07-30 8:22:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2019-07-30 8:22:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2019-07-30 8:22:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2019-07-30 8:22:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2019-07-30 8:22:53 0 [Note] InnoDB: Waiting for purge to start 2019-07-30 8:22:53 0 [Note] InnoDB: 10.4.6 started; log sequence number 398314178060; transaction id 637240118 2019-07-30 8:22:53 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2019-07-30 8:22:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2019-07-30 8:22:53 0 [Note] Server socket created on IP: '::'. 2019-07-30 8:22:53 0 [Note] WSREP: wsrep_init_schema_and_SR (nil) 2019-07-30 8:22:53 0 [Note] WSREP: Server initialized 2019-07-30 8:22:53 0 [Note] WSREP: Server status change initializing -> initialized 2019-07-30 8:22:53 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-07-30 8:22:53 4 [Note] WSREP: Server status change initialized -> joined 2019-07-30 8:22:53 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-07-30 8:22:53 4 [Note] WSREP: Recovered position from storage: 59d93b0e-b04c-11e9-89e1-37592b577569:317913118 2019-07-30 8:22:53 11 [Note] WSREP: Starting applier thread 11 2019-07-30 8:22:53 14 [Note] WSREP: Starting applier thread 14 2019-07-30 8:22:53 12 [Note] WSREP: Starting applier thread 12 2019-07-30 8:22:53 15 [Note] WSREP: Starting applier thread 15 2019-07-30 8:22:53 13 [Note] WSREP: Starting applier thread 13 2019-07-30 8:22:53 17 [Note] WSREP: Starting applier thread 17 2019-07-30 8:22:53 16 [Note] WSREP: Starting applier thread 16 2019-07-30 8:22:53 4 [Note] WSREP: Recovered view from SST: id: 59d93b0e-b04c-11e9-89e1-37592b577569:317861735 status: primary protocol_version: 4 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO final: no own_index: 2 members(3): 0: 1fb571ea-b1ec-11e9-9e17-f6900a741833, dbcluster02.our-domain.local 1: 2b762b2f-b1ed-11e9-8fa0-1a39059e097c, dbcluster01.our-domain.local 2: 88ce85d0-b2a2-11e9-8b6e-da4ac98b904e, dbcluster03.our-domain.local 2019-07-30 8:22:53 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-07-30 8:22:53 19 [Note] WSREP: Recovered cluster id 59d93b0e-b04c-11e9-89e1-37592b577569 2019-07-30 8:22:53 0 [Note] Reading of all Master_info entries succeeded 2019-07-30 8:22:53 0 [Note] Added new Master_info '' to hash table 2019-07-30 8:22:53 0 [Note] mysqld: ready for connections. Version: '10.4.6-MariaDB-1:10.4.6+maria~bionic' socket: '/var/run/mysqld/mysqld.sock' port: 33307 mariadb.org binary distribution 2019-07-30 8:22:53 4 [Note] WSREP: SST received: 59d93b0e-b04c-11e9-89e1-37592b577569:317913118 2019-07-30 8:22:53 3 [Note] WSREP: Installed new state from SST: 59d93b0e-b04c-11e9-89e1-37592b577569:317913118 2019-07-30 8:22:53 3 [Note] WSREP: Cert. index preload up to 317913118 2019-07-30 8:22:53 0 [Note] WSREP: ####### IST applying starts with 317913119 2019-07-30 8:22:53 0 [Note] WSREP: ####### IST current seqno initialized to 317861501 2019-07-30 8:22:53 0 [Note] WSREP: Receiving IST... 0.0% ( 0/235 events) complete. 2019-07-30 8:22:53 0 [Note] WSREP: IST preload starting at 317861501 2019-07-30 8:22:53 0 [Note] WSREP: Service thread queue flushed. 2019-07-30 8:22:53 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:317861500, protocol version: 5 2019-07-30 8:22:53 0 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-07-30 8:22:53 0 [Note] WSREP: ####### Adjusting cert position: 317861583 -> 317861584 2019-07-30 8:22:53 0 [Note] WSREP: Service thread queue flushed. 2019-07-30 8:22:53 0 [Note] WSREP: Lowest cert indnex boundary for CC from preload: 317861501 2019-07-30 8:22:53 0 [Note] WSREP: Min available from gcache for CC from preload: 317861501 2019-07-30 8:22:53 0 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-07-30 8:22:53 0 [Note] WSREP: ####### Adjusting cert position: 317861726 -> 317861727 2019-07-30 8:22:53 0 [Note] WSREP: Service thread queue flushed. 2019-07-30 8:22:53 0 [Note] WSREP: Lowest cert indnex boundary for CC from preload: 317861501 2019-07-30 8:22:53 0 [Note] WSREP: Min available from gcache for CC from preload: 317861501 2019-07-30 8:22:53 0 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-07-30 8:22:53 0 [Note] WSREP: ####### Adjusting cert position: 317861734 -> 317861735 2019-07-30 8:22:53 0 [Note] WSREP: Service thread queue flushed. 2019-07-30 8:22:53 0 [Note] WSREP: Lowest cert indnex boundary for CC from preload: 317861501 2019-07-30 8:22:53 0 [Note] WSREP: Min available from gcache for CC from preload: 317861501 2019-07-30 8:22:53 0 [Note] WSREP: Receiving IST...100.0% (235/235 events) complete. 2019-07-30 8:22:53 3 [Note] WSREP: Cert. index preloaded up to 317861735 2019-07-30 8:22:53 0 [Note] WSREP: 2.0 (dbcluster03.our-domain.local): State transfer from 0.0 (dbcluster02.our-domain.local) complete. 2019-07-30 8:22:53 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 317917259) 2019-07-30 8:22:55 0 [Note] WSREP: Member 2.0 (dbcluster03.our-domain.local) synced with group. 2019-07-30 8:22:55 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 317918135) 2019-07-30 8:22:55 13 [Note] WSREP: Server dbcluster03.our-domain.local synced with group 2019-07-30 8:22:55 13 [Note] WSREP: Server status change joined -> synced 2019-07-30 8:22:55 13 [Note] WSREP: Synchronized with group, ready for connections 2019-07-30 8:22:55 13 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.