150327 22:29:04 [ERROR] Master '5.5': Slave SQL: Could not execute Delete_rows_v1 event on table X_20150210.feed_lookup; Can't find record in 'feed_lookup', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log tulwaurt005-mysql-bin.000687, end_log_pos 649583202, Internal MariaDB error code: 1032
150327 22:29:04 [Warning] Master '5.5': Slave: Can't find record in 'feed_lookup' Error_code: 1032
150327 22:29:04 [ERROR] Master '5.5': Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'tulwaurt005-mysql-bin.000687' position 649581827
150327 22:29:04 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.0.17-MariaDB-log
key_buffer_size=209715200
read_buffer_size=262144
max_used_connections=6
max_threads=3202
thread_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7648478 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7fb38902d008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fb370d5b870 thread_stack 0x48000
/opt/mariadb-10/sbin/mysqld(my_print_stacktrace+0x2b)[0xb743db]
/opt/mariadb-10/sbin/mysqld(handle_fatal_signal+0x398)[0x726a38]
/lib64/libpthread.so.0[0x3353a0f500]
/opt/mariadb-10/sbin/mysqld(_ZN14rpl_group_infoD2Ev+0x18)[0x68e148]
/opt/mariadb-10/sbin/mysqld(handle_slave_sql+0x21c)[0x56576c]
/lib64/libpthread.so.0[0x3353a07851]
/lib64/libc.so.6(clone+0x6d)[0x33536e890d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 89
Status: NOT_KILLED

Optimizer switch: index_merge=off,index_merge_union=off,index_merge_sort_union=off,index_merge_intersection=off,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=off,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150327 22:29:05 mysqld_safe Number of processes running now: 0
150327 22:29:05 mysqld_safe mysqld restarted
150327 22:29:05 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150327 22:29:05 [Note] InnoDB: The InnoDB memory heap is disabled
150327 22:29:05 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150327 22:29:05 [Note] InnoDB: Memory barrier is not used
150327 22:29:05 [Note] InnoDB: Compressed tables use zlib 1.2.3
150327 22:29:05 [Note] InnoDB: Using Linux native AIO
150327 22:29:05 [Note] InnoDB: Using CPU crc32 instructions
150327 22:29:05 [Note] InnoDB: Initializing buffer pool, size = 24.0G
150327 22:29:06 [Note] InnoDB: Completed initialization of buffer pool
150327 22:29:07 [Note] InnoDB: Highest supported file format is Barracuda.
150327 22:29:07 [Note] InnoDB: The log sequence numbers 326906981146 and 326906981146 in ibdata files do not match the log sequence number 328140590060 in the ib_logfiles!
150327 22:29:07 [Note] InnoDB: Database was not shutdown normally!
150327 22:29:07 [Note] InnoDB: Starting crash recovery.
150327 22:29:07 [Note] InnoDB: Reading tablespace information from the .ibd files...
150327 22:29:07 [Note] InnoDB: Restoring possible half-written data pages 
150327 22:29:07 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 381904073, file name ./tulwaurt005-mysql-bin.000177
150327 22:29:07 [Note] InnoDB: 128 rollback segment(s) are active.
150327 22:29:07 [Note] InnoDB: Waiting for purge to start
150327 22:29:07 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 328140590060
2015-03-27 22:29:07 7f77babf9700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
150327 22:29:08 [Note] Plugin 'FEEDBACK' is disabled.
150327 22:29:08 [Note] Recovering after a crash using tulwaurt005-mysql-bin
150327 22:29:09 [Note] Starting crash recovery...
150327 22:29:09 [Note] Crash recovery finished.
150327 22:29:09 [Note] Server socket created on IP: '::'.
150327 22:29:09 [Note] Event Scheduler: Loaded 0 events
150327 22:29:09 [Note] /opt/mariadb-10/sbin/mysqld: ready for connections.
Version: '10.0.17-MariaDB-log'  socket: '/var/lib/mariadb/mysql-10.sock'  port: 3310  MariaDB Server
2015-03-27 22:29:28 7f77babf9700 InnoDB: Buffer pool(s) load completed at 150327 22:29:28
150327 22:35:54 [Note] Master '5.5': Slave SQL thread initialized, starting replication in log 'tulwaurt005-mysql-bin.000687' at position 649581827, relay log './tulwaurt005-relay-bin-5@002e5.000004' position: 649582126
150327 22:35:54 [ERROR] Master '5.5': Slave SQL: Could not execute Delete_rows_v1 event on table X_20150210.feed_lookup; Can't find record in 'feed_lookup', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log tulwaurt005-mysql-bin.000687, end_log_pos 649583202, Internal MariaDB error code: 1032
150327 22:35:54 [Warning] Master '5.5': Slave: Can't find record in 'feed_lookup' Error_code: 1032
150327 22:35:54 [ERROR] Master '5.5': Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'tulwaurt005-mysql-bin.000687' position 649581827
150327 22:35:54 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.0.17-MariaDB-log
key_buffer_size=209715200
read_buffer_size=262144
max_used_connections=5
max_threads=3202
thread_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7648478 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7f77d882d008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f77bec8e870 thread_stack 0x48000
150327 22:35:54 [Note] Master '5.5': Slave I/O thread: connected to master 'repl@localhost:3306',replication started in log 'tulwaurt005-mysql-bin.000702' at position 957099125
/opt/mariadb-10/sbin/mysqld(my_print_stacktrace+0x2b)[0xb743db]
/opt/mariadb-10/sbin/mysqld(handle_fatal_signal+0x398)[0x726a38]
/lib64/libpthread.so.0[0x3353a0f500]
/opt/mariadb-10/sbin/mysqld(_ZN14rpl_group_infoD2Ev+0x18)[0x68e148]
/opt/mariadb-10/sbin/mysqld(handle_slave_sql+0x21c)[0x56576c]
/lib64/libpthread.so.0[0x3353a07851]
/lib64/libc.so.6(clone+0x6d)[0x33536e890d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 31
Status: NOT_KILLED

Optimizer switch: index_merge=off,index_merge_union=off,index_merge_sort_union=off,index_merge_intersection=off,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=off,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150327 22:35:54 mysqld_safe Number of processes running now: 0
150327 22:35:54 mysqld_safe mysqld restarted
150327 22:35:54 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150327 22:35:54 [Note] InnoDB: The InnoDB memory heap is disabled
150327 22:35:54 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150327 22:35:54 [Note] InnoDB: Memory barrier is not used
150327 22:35:54 [Note] InnoDB: Compressed tables use zlib 1.2.3
150327 22:35:54 [Note] InnoDB: Using Linux native AIO
150327 22:35:54 [Note] InnoDB: Using CPU crc32 instructions
150327 22:35:54 [Note] InnoDB: Initializing buffer pool, size = 24.0G
150327 22:35:55 [Note] InnoDB: Completed initialization of buffer pool
150327 22:35:55 [Note] InnoDB: Highest supported file format is Barracuda.
150327 22:35:55 [Note] InnoDB: The log sequence numbers 326906981146 and 326906981146 in ibdata files do not match the log sequence number 328140591985 in the ib_logfiles!
150327 22:35:55 [Note] InnoDB: Database was not shutdown normally!
150327 22:35:55 [Note] InnoDB: Starting crash recovery.
150327 22:35:55 [Note] InnoDB: Reading tablespace information from the .ibd files...
150327 22:35:55 [Note] InnoDB: Restoring possible half-written data pages 
150327 22:35:55 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 381904073, file name ./tulwaurt005-mysql-bin.000177
150327 22:35:56 [Note] InnoDB: 128 rollback segment(s) are active.
150327 22:35:56 [Note] InnoDB: Waiting for purge to start
150327 22:35:56 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 328140591985
2015-03-27 22:35:56 7ef8e03f9700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
150327 22:35:56 [Note] Plugin 'FEEDBACK' is disabled.
150327 22:35:56 [Note] Recovering after a crash using tulwaurt005-mysql-bin
150327 22:35:56 [Note] Starting crash recovery...
150327 22:35:56 [Note] Crash recovery finished.
150327 22:35:56 [Note] Server socket created on IP: '::'.
150327 22:35:56 [Note] Event Scheduler: Loaded 0 events
150327 22:35:56 [Note] /opt/mariadb-10/sbin/mysqld: ready for connections.
Version: '10.0.17-MariaDB-log'  socket: '/var/lib/mariadb/mysql-10.sock'  port: 3310  MariaDB Server
2015-03-27 22:36:16 7ef8e03f9700 InnoDB: Buffer pool(s) load completed at 150327 22:36:16
150327 22:38:49 [Note] /opt/mariadb-10/sbin/mysqld: Normal shutdown

150327 22:38:49 [Note] Event Scheduler: Purging the queue. 0 events
150327 22:38:49 [Note] InnoDB: FTS optimize thread exiting.
150327 22:38:49 [Note] InnoDB: Starting shutdown...
2015-03-27 22:38:49 7ef8e03f9700 InnoDB: Dumping buffer pool(s) to .//ib_buffer_pool
2015-03-27 22:38:49 7ef8e03f9700 InnoDB: Buffer pool(s) dump completed at 150327 22:38:49
150327 22:38:51 [Note] InnoDB: Shutdown completed; log sequence number 328140595365
150327 22:38:51 [Note] /opt/mariadb-10/sbin/mysqld: Shutdown complete

150327 22:38:51 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld-10.pid ended
150327 22:38:53 mysqld_safe Starting mysqld daemon with databases from /u01/mariadb-10
150327 22:38:53 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150327 22:38:53 [Note] InnoDB: The InnoDB memory heap is disabled
150327 22:38:53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150327 22:38:53 [Note] InnoDB: Memory barrier is not used
150327 22:38:53 [Note] InnoDB: Compressed tables use zlib 1.2.3
150327 22:38:53 [Note] InnoDB: Using Linux native AIO
150327 22:38:53 [Note] InnoDB: Using CPU crc32 instructions
150327 22:38:53 [Note] InnoDB: Initializing buffer pool, size = 24.0G
150327 22:38:54 [Note] InnoDB: Completed initialization of buffer pool
150327 22:38:54 [Note] InnoDB: Highest supported file format is Barracuda.
150327 22:38:55 [Note] InnoDB: 128 rollback segment(s) are active.
150327 22:38:55 [Note] InnoDB: Waiting for purge to start
150327 22:38:55 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 328140595365
2015-03-27 22:38:55 7fc9551f8700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
150327 22:38:55 [Note] Plugin 'FEEDBACK' is disabled.
150327 22:38:55 [Note] Server socket created on IP: '::'.
150327 22:38:55 [Note] Event Scheduler: Loaded 0 events
150327 22:38:55 [Note] /opt/mariadb-10/sbin/mysqld: ready for connections.
Version: '10.0.17-MariaDB-log'  socket: '/var/lib/mariadb/mysql-10.sock'  port: 3310  MariaDB Server
150327 22:39:10 [Note] Master '5.5': Slave SQL thread initialized, starting replication in log 'tulwaurt005-mysql-bin.000687' at position 649581827, relay log './tulwaurt005-relay-bin-5@002e5.000004' position: 649582126
150327 22:39:10 [ERROR] Master '5.5': Slave SQL: Could not execute Delete_rows_v1 event on table X_20150210.feed_lookup; Can't find record in 'feed_lookup', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log tulwaurt005-mysql-bin.000687, end_log_pos 649583202, Internal MariaDB error code: 1032
150327 22:39:10 [Warning] Master '5.5': Slave: Can't find record in 'feed_lookup' Error_code: 1032
150327 22:39:10 [ERROR] Master '5.5': Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'tulwaurt005-mysql-bin.000687' position 649581827
150327 22:39:10 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.0.17-MariaDB-log
key_buffer_size=209715200
read_buffer_size=262144
max_used_connections=1
max_threads=3202
thread_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7648478 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7fc97002d008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fc9523ef870 thread_stack 0x48000
/opt/mariadb-10/sbin/mysqld(my_print_stacktrace+0x2b)[0xb743db]
/opt/mariadb-10/sbin/mysqld(handle_fatal_signal+0x398)[0x726a38]
/lib64/libpthread.so.0[0x3353a0f500]
150327 22:39:10 [Note] Master '5.5': Slave I/O thread: connected to master 'repl@localhost:3306',replication started in log 'tulwaurt005-mysql-bin.000702' at position 957219213
/opt/mariadb-10/sbin/mysqld(_ZN14rpl_group_infoD2Ev+0x18)[0x68e148]
/opt/mariadb-10/sbin/mysqld(handle_slave_sql+0x21c)[0x56576c]
/lib64/libpthread.so.0[0x3353a07851]
/lib64/libc.so.6(clone+0x6d)[0x33536e890d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 25
Status: NOT_KILLED

Optimizer switch: index_merge=off,index_merge_union=off,index_merge_sort_union=off,index_merge_intersection=off,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=off,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150327 22:39:11 mysqld_safe Number of processes running now: 0
150327 22:39:11 mysqld_safe mysqld restarted
150327 22:39:11 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150327 22:39:11 [Note] InnoDB: The InnoDB memory heap is disabled
150327 22:39:11 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150327 22:39:11 [Note] InnoDB: Memory barrier is not used
150327 22:39:11 [Note] InnoDB: Compressed tables use zlib 1.2.3
150327 22:39:11 [Note] InnoDB: Using Linux native AIO
150327 22:39:11 [Note] InnoDB: Using CPU crc32 instructions
150327 22:39:11 [Note] InnoDB: Initializing buffer pool, size = 24.0G
150327 22:39:12 [Note] InnoDB: Completed initialization of buffer pool
150327 22:39:12 [Note] InnoDB: Highest supported file format is Barracuda.
150327 22:39:12 [Note] InnoDB: The log sequence numbers 328140595365 and 328140595365 in ibdata files do not match the log sequence number 328140595375 in the ib_logfiles!
150327 22:39:12 [Note] InnoDB: Database was not shutdown normally!
150327 22:39:12 [Note] InnoDB: Starting crash recovery.
150327 22:39:12 [Note] InnoDB: Reading tablespace information from the .ibd files...
150327 22:39:12 [Note] InnoDB: Restoring possible half-written data pages 
150327 22:39:12 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 381904073, file name ./tulwaurt005-mysql-bin.000177
150327 22:39:13 [Note] InnoDB: 128 rollback segment(s) are active.
150327 22:39:13 [Note] InnoDB: Waiting for purge to start
150327 22:39:13 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 328140595375
2015-03-27 22:39:13 7f16a93f9700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
150327 22:39:13 [Note] Plugin 'FEEDBACK' is disabled.
150327 22:39:13 [Note] Recovering after a crash using tulwaurt005-mysql-bin
150327 22:39:13 [Note] Starting crash recovery...
150327 22:39:13 [Note] Crash recovery finished.
150327 22:39:13 [Note] Server socket created on IP: '::'.
150327 22:39:13 [Note] Event Scheduler: Loaded 0 events
150327 22:39:13 [Note] /opt/mariadb-10/sbin/mysqld: ready for connections.
Version: '10.0.17-MariaDB-log'  socket: '/var/lib/mariadb/mysql-10.sock'  port: 3310  MariaDB Server
2015-03-27 22:39:32 7f16a93f9700 InnoDB: Buffer pool(s) load completed at 150327 22:39:32
