140819 13:21:54 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=41 max_threads=1002 thread_count=27 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f40f01d2008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4293150de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] /data/mysql/bin/mysqld(end_of_list+0x0)[0x13ec120] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f40d76f4020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 91938 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140819 13:21:57 mysqld_safe Number of processes running now: 0 140819 13:21:57 mysqld_safe mysqld restarted 140819 13:21:58 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 13:21:58 [Note] InnoDB: The InnoDB memory heap is disabled 140819 13:21:58 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 13:21:58 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 13:21:58 [Note] InnoDB: Using Linux native AIO 140819 13:21:58 [Note] InnoDB: Not using CPU crc32 instructions 140819 13:21:58 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 13:21:59 [Note] InnoDB: Completed initialization of buffer pool 140819 13:22:00 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 13:22:00 [Note] InnoDB: Highest supported file format is Barracuda. 140819 13:22:00 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661110609022 140819 13:22:00 [Note] InnoDB: Database was not shutdown normally! 140819 13:22:00 [Note] InnoDB: Starting crash recovery. 140819 13:22:00 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 13:22:19 [Note] InnoDB: Restoring possible half-written data pages 140819 13:22:19 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661115851776 InnoDB: Doing recovery: scanned up to log sequence number 7661121094656 InnoDB: Doing recovery: scanned up to log sequence number 7661126337536 InnoDB: Doing recovery: scanned up to log sequence number 7661131580416 InnoDB: Doing recovery: scanned up to log sequence number 7661136823296 InnoDB: Doing recovery: scanned up to log sequence number 7661142066176 InnoDB: Doing recovery: scanned up to log sequence number 7661145751591 InnoDB: Transaction 916612606 was in the XA prepared state. InnoDB: Transaction 916612606 was in the XA prepared state. InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 0 row operations to undo InnoDB: Trx id counter is 916612864 140819 13:22:32 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 199164979, file name /mysql_logs/mysql_products_3104/mysql-bin.013333 140819 13:22:35 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-19 13:22:35 7f639d7ff700 InnoDB: Rollback of non-prepared transactions completed 140819 13:22:37 [Note] InnoDB: Waiting for purge to start 140819 13:22:37 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661145751591 140819 13:22:37 [Note] Plugin 'FEEDBACK' is disabled. 140819 13:22:38 server_audit: server_audit_excl_users set to 'root'. 140819 13:22:38 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 13:22:38 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 13:22:38 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 13:22:45 [Note] Starting crash recovery... 2014-08-19 13:22:45 7f6575e53760 InnoDB: Starting recovery for XA transactions... 2014-08-19 13:22:45 7f6575e53760 InnoDB: Transaction 916612606 in prepared state after recovery 2014-08-19 13:22:45 7f6575e53760 InnoDB: Transaction contains changes to 499 rows 2014-08-19 13:22:45 7f6575e53760 InnoDB: 1 transactions in prepared state after recovery 140819 13:22:45 [Note] Found 1 prepared transaction(s) in InnoDB 140819 13:22:45 [Note] Crash recovery finished. 140819 13:22:45 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 13:22:45 7f6575e53760 InnoDB: Error: table `mysql_tmp`.`#sql5992_16a83_1` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 2014-08-19 13:22:45 7f6575e53760 InnoDB: Error: table `mysql_tmp`.`#sql5992_16722_39` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 2014-08-19 13:22:45 7f6575e53760 InnoDB: Error: table `mysql_tmp`.`#sql5992_16a83_0` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 13:22:45 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired 140819 13:22:45 [Warning] Checking table: './mysql/user' 140819 13:22:45 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly 140819 13:22:45 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired 140819 13:22:45 [Warning] Checking table: './mysql/db' 140819 13:22:45 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly 140819 13:22:47 [Note] Event Scheduler: Loaded 0 events 140819 13:22:47 [Note] Event Scheduler: scheduler thread started with id 2 140819 13:22:47 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server pure virtual method called terminate called without an active exception 140819 13:23:52 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=12 max_threads=1002 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f63a43bb008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f655277fde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] /lib64/libc.so.6(gsignal+0x35)[0x30ec632885] /lib64/libc.so.6(abort+0x175)[0x30ec634065] /usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x30f06bea7d] /usr/lib64/libstdc++.so.6[0x30f06bcc06] /usr/lib64/libstdc++.so.6[0x30f06bcc33] /usr/lib64/libstdc++.so.6[0x30f06bd55f] sql/sql_select.cc:21500(count_field_types(st_select_lex*, TMP_TABLE_PARAM*, List&, bool))[0x5b6455] sql/sql_select.cc:997(JOIN::prepare_stage2())[0x5c13da] sql/sql_select.cc:970(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4ef4] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f63a394d020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 9 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140819 13:23:53 mysqld_safe Number of processes running now: 0 140819 13:23:53 mysqld_safe mysqld restarted 140819 13:23:54 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 13:23:54 [Note] InnoDB: The InnoDB memory heap is disabled 140819 13:23:54 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 13:23:54 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 13:23:54 [Note] InnoDB: Using Linux native AIO 140819 13:23:54 [Note] InnoDB: Not using CPU crc32 instructions 140819 13:23:54 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 13:23:55 [Note] InnoDB: Completed initialization of buffer pool 140819 13:23:55 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 13:23:55 [Note] InnoDB: Highest supported file format is Barracuda. 140819 13:23:55 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661178508877 140819 13:23:55 [Note] InnoDB: Database was not shutdown normally! 140819 13:23:55 [Note] InnoDB: Starting crash recovery. 140819 13:23:55 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 13:23:56 [Note] InnoDB: Restoring possible half-written data pages 140819 13:23:56 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661183751680 InnoDB: Doing recovery: scanned up to log sequence number 7661188994560 InnoDB: Doing recovery: scanned up to log sequence number 7661194237440 InnoDB: Doing recovery: scanned up to log sequence number 7661199480320 InnoDB: Doing recovery: scanned up to log sequence number 7661204723200 InnoDB: Doing recovery: scanned up to log sequence number 7661209966080 InnoDB: Doing recovery: scanned up to log sequence number 7661215208960 InnoDB: Doing recovery: scanned up to log sequence number 7661220451840 InnoDB: Doing recovery: scanned up to log sequence number 7661225694720 InnoDB: Doing recovery: scanned up to log sequence number 7661229028280 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 383 row operations to undo InnoDB: Trx id counter is 916613632 140819 13:24:00 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 7817508, file name /mysql_logs/mysql_products_3104/mysql-bin.013334 140819 13:24:04 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-19 13:24:04 7f33c53ff700 InnoDB: Rolling back trx with id 916613293, 383 rows to undo 140819 13:24:05 [Note] InnoDB: Rollback of trx with id 916613293 completed 2014-08-19 13:24:05 7f33c53ff700 InnoDB: Rollback of non-prepared transactions completed 140819 13:24:08 [Note] InnoDB: Waiting for purge to start 140819 13:24:08 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661229028280 140819 13:24:08 [Note] Plugin 'FEEDBACK' is disabled. 140819 13:24:08 server_audit: server_audit_excl_users set to 'root'. 140819 13:24:08 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 13:24:08 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 13:24:08 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 13:24:08 [Note] Starting crash recovery... 140819 13:24:08 [Note] Crash recovery finished. 140819 13:24:08 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 13:24:08 7f359d7ee760 InnoDB: Error: table `mysql_tmp`.`#sql64f6_9_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 13:24:08 [Note] Event Scheduler: Loaded 0 events 140819 13:24:08 [Note] Event Scheduler: scheduler thread started with id 2 140819 13:24:08 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server pure virtual method called terminate called without an active exception 140819 13:25:00 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=8 max_threads=1002 thread_count=9 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f33cbf4c008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f359c16bde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] /lib64/libc.so.6(gsignal+0x35)[0x30ec632885] /lib64/libc.so.6(abort+0x175)[0x30ec634065] /usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x30f06bea7d] /usr/lib64/libstdc++.so.6[0x30f06bcc06] /usr/lib64/libstdc++.so.6[0x30f06bcc33] /usr/lib64/libstdc++.so.6[0x30f06bd55f] sql/sql_select.cc:21500(count_field_types(st_select_lex*, TMP_TABLE_PARAM*, List&, bool))[0x5b6455] sql/sql_select.cc:997(JOIN::prepare_stage2())[0x5c13da] sql/sql_select.cc:970(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4ef4] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f33c8250020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 9 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140819 13:25:01 mysqld_safe Number of processes running now: 0 140819 13:25:01 mysqld_safe mysqld restarted 140819 13:25:02 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 13:25:02 [Note] InnoDB: The InnoDB memory heap is disabled 140819 13:25:02 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 13:25:02 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 13:25:02 [Note] InnoDB: Using Linux native AIO 140819 13:25:02 [Note] InnoDB: Not using CPU crc32 instructions 140819 13:25:02 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 13:25:03 [Note] InnoDB: Completed initialization of buffer pool 140819 13:25:03 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 13:25:03 [Note] InnoDB: Highest supported file format is Barracuda. 140819 13:25:03 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661229204543 140819 13:25:03 [Note] InnoDB: Database was not shutdown normally! 140819 13:25:03 [Note] InnoDB: Starting crash recovery. 140819 13:25:03 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 13:25:04 [Note] InnoDB: Restoring possible half-written data pages 140819 13:25:04 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661230702393 140819 13:25:06 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 920293, file name /mysql_logs/mysql_products_3104/mysql-bin.013335 140819 13:25:08 [Note] InnoDB: 128 rollback segment(s) are active. 140819 13:25:08 [Note] InnoDB: Waiting for purge to start 140819 13:25:08 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661230702393 140819 13:25:08 [Note] Plugin 'FEEDBACK' is disabled. 140819 13:25:08 server_audit: server_audit_excl_users set to 'root'. 140819 13:25:08 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 13:25:08 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 13:25:08 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 13:25:08 [Note] Starting crash recovery... 140819 13:25:08 [Note] Crash recovery finished. 140819 13:25:08 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 13:25:08 7f46a208e760 InnoDB: Error: table `mysql_tmp`.`#sql677c_9_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 13:25:08 [Note] Event Scheduler: Loaded 0 events 140819 13:25:08 [Note] Event Scheduler: scheduler thread started with id 2 140819 13:25:08 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server pure virtual method called terminate called without an active exception 140819 13:29:12 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=17 max_threads=1002 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f44c9ebd008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f46a0d22de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] /lib64/libc.so.6(gsignal+0x35)[0x30ec632885] /lib64/libc.so.6(abort+0x175)[0x30ec634065] /usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x30f06bea7d] /usr/lib64/libstdc++.so.6[0x30f06bcc06] /usr/lib64/libstdc++.so.6[0x30f06bcc33] /usr/lib64/libstdc++.so.6[0x30f06bd55f] sql/sql_select.cc:21500(count_field_types(st_select_lex*, TMP_TABLE_PARAM*, List&, bool))[0x5b6455] sql/sql_select.cc:997(JOIN::prepare_stage2())[0x5c13da] sql/sql_select.cc:970(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4ef4] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f44cc9f9020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 5 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140819 13:29:13 mysqld_safe Number of processes running now: 0 140819 13:29:13 mysqld_safe mysqld restarted 140819 13:29:14 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 13:29:14 [Note] InnoDB: The InnoDB memory heap is disabled 140819 13:29:14 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 13:29:14 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 13:29:14 [Note] InnoDB: Using Linux native AIO 140819 13:29:14 [Note] InnoDB: Not using CPU crc32 instructions 140819 13:29:14 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 13:29:14 [Note] InnoDB: Completed initialization of buffer pool 140819 13:29:14 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 13:29:14 [Note] InnoDB: Highest supported file format is Barracuda. 140819 13:29:14 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661257979094 140819 13:29:14 [Note] InnoDB: Database was not shutdown normally! 140819 13:29:14 [Note] InnoDB: Starting crash recovery. 140819 13:29:14 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 13:29:16 [Note] InnoDB: Restoring possible half-written data pages 140819 13:29:16 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661263221760 InnoDB: Doing recovery: scanned up to log sequence number 7661268464640 InnoDB: Doing recovery: scanned up to log sequence number 7661273707520 InnoDB: Doing recovery: scanned up to log sequence number 7661278950400 InnoDB: Doing recovery: scanned up to log sequence number 7661284193280 InnoDB: Doing recovery: scanned up to log sequence number 7661284833194 InnoDB: Transaction 916615632 was in the XA prepared state. InnoDB: Transaction 916615632 was in the XA prepared state. InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 0 row operations to undo InnoDB: Trx id counter is 916615936 140819 13:29:17 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 33278390, file name /mysql_logs/mysql_products_3104/mysql-bin.013336 140819 13:29:21 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-19 13:29:21 7f57cc3ff700 InnoDB: Rollback of non-prepared transactions completed 140819 13:29:24 [Note] InnoDB: Waiting for purge to start 140819 13:29:24 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661284833194 140819 13:29:24 [Note] Plugin 'FEEDBACK' is disabled. 140819 13:29:24 server_audit: server_audit_excl_users set to 'root'. 140819 13:29:24 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 13:29:24 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 13:29:24 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 13:29:24 [Note] Starting crash recovery... 2014-08-19 13:29:24 7f59a49be760 InnoDB: Starting recovery for XA transactions... 2014-08-19 13:29:24 7f59a49be760 InnoDB: Transaction 916615632 in prepared state after recovery 2014-08-19 13:29:24 7f59a49be760 InnoDB: Transaction contains changes to 499 rows 2014-08-19 13:29:24 7f59a49be760 InnoDB: 1 transactions in prepared state after recovery 140819 13:29:24 [Note] Found 1 prepared transaction(s) in InnoDB 140819 13:29:25 [Note] Crash recovery finished. 140819 13:29:25 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 13:29:25 7f59a49be760 InnoDB: Error: table `mysql_tmp`.`#sql6893_5_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 13:29:25 [Note] Event Scheduler: Loaded 0 events 140819 13:29:25 [Note] Event Scheduler: scheduler thread started with id 2 140819 13:29:25 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server pure virtual method called terminate called without an active exception 140819 13:30:40 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=9 max_threads=1002 thread_count=9 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f57d2fdf008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f59a35b4de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] /lib64/libc.so.6(gsignal+0x35)[0x30ec632885] /lib64/libc.so.6(abort+0x175)[0x30ec634065] /usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x30f06bea7d] /usr/lib64/libstdc++.so.6[0x30f06bcc06] /usr/lib64/libstdc++.so.6[0x30f06bcc33] /usr/lib64/libstdc++.so.6[0x30f06bd55f] sql/sql_select.cc:21500(count_field_types(st_select_lex*, TMP_TABLE_PARAM*, List&, bool))[0x5b6455] sql/sql_select.cc:997(JOIN::prepare_stage2())[0x5c13da] sql/sql_select.cc:970(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4ef4] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f57cf4ba020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 7 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140819 13:30:42 mysqld_safe Number of processes running now: 0 140819 13:30:42 mysqld_safe mysqld restarted 140819 13:30:42 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 13:30:42 [Note] InnoDB: The InnoDB memory heap is disabled 140819 13:30:42 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 13:30:42 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 13:30:42 [Note] InnoDB: Using Linux native AIO 140819 13:30:42 [Note] InnoDB: Not using CPU crc32 instructions 140819 13:30:42 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 13:30:43 [Note] InnoDB: Completed initialization of buffer pool 140819 13:30:44 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 13:30:44 [Note] InnoDB: Highest supported file format is Barracuda. 140819 13:30:44 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661284943942 140819 13:30:44 [Note] InnoDB: Database was not shutdown normally! 140819 13:30:44 [Note] InnoDB: Starting crash recovery. 140819 13:30:44 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 13:30:45 [Note] InnoDB: Restoring possible half-written data pages 140819 13:30:45 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661284951846 140819 13:30:46 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 758, file name /mysql_logs/mysql_products_3104/mysql-bin.013337 140819 13:30:48 [Note] InnoDB: 128 rollback segment(s) are active. 140819 13:30:48 [Note] InnoDB: Waiting for purge to start 140819 13:30:48 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661284951846 140819 13:30:48 [Note] Plugin 'FEEDBACK' is disabled. 140819 13:30:48 server_audit: server_audit_excl_users set to 'root'. 140819 13:30:48 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 13:30:48 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 13:30:48 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 13:30:48 [Note] Starting crash recovery... 140819 13:30:48 [Note] Crash recovery finished. 140819 13:30:48 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 13:30:48 7f35d3b81760 InnoDB: Error: table `mysql_tmp`.`#sql6f1d_7_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 13:30:48 [Note] Event Scheduler: Loaded 0 events 140819 13:30:48 [Note] Event Scheduler: scheduler thread started with id 2 140819 13:30:48 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140819 13:55:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=23 max_threads=1002 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f3401e8e008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 140819 13:55:41 mysqld_safe Number of processes running now: 0 140819 13:55:41 mysqld_safe mysqld restarted 140819 13:55:42 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 13:55:42 [Note] InnoDB: The InnoDB memory heap is disabled 140819 13:55:42 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 13:55:42 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 13:55:42 [Note] InnoDB: Using Linux native AIO 140819 13:55:42 [Note] InnoDB: Not using CPU crc32 instructions 140819 13:55:42 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 13:55:43 [Note] InnoDB: Completed initialization of buffer pool 140819 13:55:43 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 13:55:43 [Note] InnoDB: Highest supported file format is Barracuda. 140819 13:55:43 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661352357369 140819 13:55:43 [Note] InnoDB: Database was not shutdown normally! 140819 13:55:43 [Note] InnoDB: Starting crash recovery. 140819 13:55:43 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 13:55:45 [Note] InnoDB: Restoring possible half-written data pages 140819 13:55:45 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661357599744 InnoDB: Doing recovery: scanned up to log sequence number 7661362842624 InnoDB: Doing recovery: scanned up to log sequence number 7661368085504 InnoDB: Doing recovery: scanned up to log sequence number 7661373328384 InnoDB: Doing recovery: scanned up to log sequence number 7661374735718 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 59 row operations to undo InnoDB: Trx id counter is 916619776 140819 13:55:51 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 51258296, file name /mysql_logs/mysql_products_3104/mysql-bin.013338 140819 13:55:54 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-19 13:55:54 7f3ed13ff700 InnoDB: Rolling back trx with id 916619287, 59 rows to undo 140819 13:55:55 [Note] InnoDB: Rollback of trx with id 916619287 completed 2014-08-19 13:55:55 7f3ed13ff700 InnoDB: Rollback of non-prepared transactions completed 140819 13:55:55 [Note] InnoDB: Waiting for purge to start 140819 13:55:55 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661374735718 140819 13:55:55 [Note] Plugin 'FEEDBACK' is disabled. 140819 13:55:55 server_audit: server_audit_excl_users set to 'root'. 140819 13:55:55 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 13:55:55 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 13:55:55 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 13:55:56 [Note] Starting crash recovery... 140819 13:55:56 [Note] Crash recovery finished. 140819 13:55:57 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 13:55:57 7f40a98b9760 InnoDB: Error: table `mysql_tmp`.`#sql7481_5_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 2014-08-19 13:55:57 7f40a98b9760 InnoDB: Error: table `mysql_tmp`.`#sql7481_33_0` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 13:55:57 [Note] Event Scheduler: Loaded 0 events 140819 13:55:57 [Note] Event Scheduler: scheduler thread started with id 2 140819 13:55:57 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140819 14:23:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=26 max_threads=1002 thread_count=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f3ed7f92008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 140819 14:23:45 mysqld_safe Number of processes running now: 0 140819 14:23:45 mysqld_safe mysqld restarted 140819 14:23:46 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 14:23:46 [Note] InnoDB: The InnoDB memory heap is disabled 140819 14:23:46 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 14:23:46 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 14:23:46 [Note] InnoDB: Using Linux native AIO 140819 14:23:46 [Note] InnoDB: Not using CPU crc32 instructions 140819 14:23:46 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 14:23:46 [Note] InnoDB: Completed initialization of buffer pool 140819 14:23:47 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 14:23:47 [Note] InnoDB: Highest supported file format is Barracuda. 140819 14:23:47 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661815746983 140819 14:23:47 [Note] InnoDB: Database was not shutdown normally! 140819 14:23:47 [Note] InnoDB: Starting crash recovery. 140819 14:23:47 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 14:23:48 [Note] InnoDB: Restoring possible half-written data pages 140819 14:23:48 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661820989440 InnoDB: Doing recovery: scanned up to log sequence number 7661826232320 InnoDB: Doing recovery: scanned up to log sequence number 7661831475200 InnoDB: Doing recovery: scanned up to log sequence number 7661836718080 InnoDB: Doing recovery: scanned up to log sequence number 7661841960960 InnoDB: Doing recovery: scanned up to log sequence number 7661845008527 140819 14:23:59 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 11400734, file name /mysql_logs/mysql_products_3104/mysql-bin.013340 140819 14:24:04 [Note] InnoDB: 128 rollback segment(s) are active. 140819 14:24:07 [Note] InnoDB: Waiting for purge to start 140819 14:24:07 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661845008527 140819 14:24:07 [Note] Plugin 'FEEDBACK' is disabled. 140819 14:24:07 server_audit: server_audit_excl_users set to 'root'. 140819 14:24:07 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 14:24:07 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 14:24:07 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 14:24:08 [Note] Starting crash recovery... 140819 14:24:08 [Note] Crash recovery finished. 140819 14:24:08 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 14:24:08 7f732f9e1760 InnoDB: Error: table `mysql_tmp`.`#sql1b93_11_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 14:24:08 [Note] Event Scheduler: Loaded 0 events 140819 14:24:08 [Note] Event Scheduler: scheduler thread started with id 2 140819 14:24:08 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140819 14:32:38 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=20 max_threads=1002 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f715dfd3008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f732e5fdde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] [0x7f7154510100] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7159e39020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 6 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140819 14:32:39 mysqld_safe Number of processes running now: 0 140819 14:32:39 mysqld_safe mysqld restarted 140819 14:32:40 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 14:32:40 [Note] InnoDB: The InnoDB memory heap is disabled 140819 14:32:40 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 14:32:40 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 14:32:40 [Note] InnoDB: Using Linux native AIO 140819 14:32:40 [Note] InnoDB: Not using CPU crc32 instructions 140819 14:32:40 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 14:32:41 [Note] InnoDB: Completed initialization of buffer pool 140819 14:32:41 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 14:32:41 [Note] InnoDB: Highest supported file format is Barracuda. 140819 14:32:41 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661918670712 140819 14:32:41 [Note] InnoDB: Database was not shutdown normally! 140819 14:32:41 [Note] InnoDB: Starting crash recovery. 140819 14:32:41 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 14:32:42 [Note] InnoDB: Restoring possible half-written data pages 140819 14:32:42 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661918680276 140819 14:32:44 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 44793362, file name /mysql_logs/mysql_products_3104/mysql-bin.013341 140819 14:32:47 [Note] InnoDB: 128 rollback segment(s) are active. 140819 14:32:47 [Note] InnoDB: Waiting for purge to start 140819 14:32:47 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661918680276 140819 14:32:47 [Note] Plugin 'FEEDBACK' is disabled. 140819 14:32:47 server_audit: server_audit_excl_users set to 'root'. 140819 14:32:47 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 14:32:47 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 14:32:47 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 14:32:47 [Note] Starting crash recovery... 140819 14:32:47 [Note] Crash recovery finished. 140819 14:32:47 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 14:32:47 7fa6bb1e6760 InnoDB: Error: table `mysql_tmp`.`#sql4387_6_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 14:32:47 [Note] Event Scheduler: Loaded 0 events 140819 14:32:47 [Note] Event Scheduler: scheduler thread started with id 2 140819 14:32:47 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140819 14:39:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=15 max_threads=1002 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7fa4e965f008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa6b9db4de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] [0x7fa4e6880100] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa4e6db0020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 7 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140819 14:39:34 mysqld_safe Number of processes running now: 0 140819 14:39:34 mysqld_safe mysqld restarted 140819 14:39:34 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140819 14:39:34 [Note] InnoDB: The InnoDB memory heap is disabled 140819 14:39:34 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140819 14:39:34 [Note] InnoDB: Compressed tables use zlib 1.2.3 140819 14:39:34 [Note] InnoDB: Using Linux native AIO 140819 14:39:34 [Note] InnoDB: Not using CPU crc32 instructions 140819 14:39:34 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140819 14:39:35 [Note] InnoDB: Completed initialization of buffer pool 140819 14:39:36 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140819 14:39:36 [Note] InnoDB: Highest supported file format is Barracuda. 140819 14:39:36 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7661926484540 140819 14:39:36 [Note] InnoDB: Database was not shutdown normally! 140819 14:39:36 [Note] InnoDB: Starting crash recovery. 140819 14:39:36 [Note] InnoDB: Reading tablespace information from the .ibd files... 140819 14:39:37 [Note] InnoDB: Restoring possible half-written data pages 140819 14:39:37 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7661931727360 InnoDB: Doing recovery: scanned up to log sequence number 7661936970240 InnoDB: Doing recovery: scanned up to log sequence number 7661942213120 InnoDB: Doing recovery: scanned up to log sequence number 7661947456000 InnoDB: Doing recovery: scanned up to log sequence number 7661951072903 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 52 row operations to undo InnoDB: Trx id counter is 916647680 140819 14:39:41 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 51186792, file name /mysql_logs/mysql_products_3104/mysql-bin.013342 140819 14:39:47 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-19 14:39:47 7fb459bff700 InnoDB: Rolling back trx with id 916647193, 52 rows to undo 140819 14:39:47 [Note] InnoDB: Rollback of trx with id 916647193 completed 2014-08-19 14:39:47 7fb459bff700 InnoDB: Rollback of non-prepared transactions completed 140819 14:39:52 [Note] InnoDB: Waiting for purge to start 140819 14:39:52 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7661951072903 140819 14:39:52 [Note] Plugin 'FEEDBACK' is disabled. 140819 14:39:52 server_audit: server_audit_excl_users set to 'root'. 140819 14:39:52 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140819 14:39:52 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140819 14:39:52 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140819 14:39:53 [Note] Starting crash recovery... 140819 14:39:53 [Note] Crash recovery finished. 140819 14:39:53 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-19 14:39:53 7fb632238760 InnoDB: Error: table `mysql_tmp`.`#sql50c9_7_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140819 14:39:53 [Note] Event Scheduler: Loaded 0 events 140819 14:39:53 [Note] Event Scheduler: scheduler thread started with id 2 140819 14:39:53 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140820 15:06:44 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=42 max_threads=1002 thread_count=37 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7fb44bcfe008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb60e85cde0 thread_stack 0x48000 140820 15:06:46 mysqld_safe Number of processes running now: 0 140820 15:06:46 mysqld_safe mysqld restarted 140820 15:06:48 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140820 15:06:48 [Note] InnoDB: The InnoDB memory heap is disabled 140820 15:06:48 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140820 15:06:48 [Note] InnoDB: Compressed tables use zlib 1.2.3 140820 15:06:48 [Note] InnoDB: Using Linux native AIO 140820 15:06:48 [Note] InnoDB: Not using CPU crc32 instructions 140820 15:06:48 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140820 15:06:48 [Note] InnoDB: Completed initialization of buffer pool 140820 15:06:48 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140820 15:06:48 [Note] InnoDB: Highest supported file format is Barracuda. 140820 15:06:49 [Note] InnoDB: The log sequence numbers 7569990467767 and 7569990467767 in ibdata files do not match the log sequence number 7682532559732 in the ib_logfiles! 140820 15:06:49 [Note] InnoDB: Database was not shutdown normally! 140820 15:06:49 [Note] InnoDB: Starting crash recovery. 140820 15:06:49 [Note] InnoDB: Reading tablespace information from the .ibd files... 140820 15:07:00 [Note] InnoDB: Restoring possible half-written data pages 140820 15:07:00 [Note] InnoDB: from the doublewrite buffer... InnoDB: Last MySQL binlog file position 0 50721724, file name /mysql_logs/mysql_products_3104/mysql-bin.013403 140820 15:07:06 [Note] InnoDB: 128 rollback segment(s) are active. 140820 15:07:06 [Note] InnoDB: Waiting for purge to start 140820 15:07:06 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7682532559732 140820 15:07:06 [Note] Plugin 'FEEDBACK' is disabled. 140820 15:07:06 server_audit: server_audit_excl_users set to 'root'. 140820 15:07:06 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140820 15:07:06 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140820 15:07:06 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140820 15:07:07 [Note] Starting crash recovery... 140820 15:07:07 [Note] Crash recovery finished. 140820 15:07:07 [Note] Server socket created on IP: '0.0.0.0'. 140820 15:07:08 [Note] Event Scheduler: Loaded 0 events 140820 15:07:08 [Note] Event Scheduler: scheduler thread started with id 2 140820 15:07:08 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140820 15:07:30 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=13 max_threads=1002 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f5394e47008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f556697fde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c6186] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f5390d4d020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 5 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140820 15:07:31 mysqld_safe Number of processes running now: 0 140820 15:07:31 mysqld_safe mysqld restarted 140820 15:07:32 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140820 15:07:32 [Note] InnoDB: The InnoDB memory heap is disabled 140820 15:07:32 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140820 15:07:32 [Note] InnoDB: Compressed tables use zlib 1.2.3 140820 15:07:32 [Note] InnoDB: Using Linux native AIO 140820 15:07:32 [Note] InnoDB: Not using CPU crc32 instructions 140820 15:07:32 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140820 15:07:33 [Note] InnoDB: Completed initialization of buffer pool 140820 15:07:33 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140820 15:07:33 [Note] InnoDB: Highest supported file format is Barracuda. 140820 15:07:33 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7682532578005 140820 15:07:33 [Note] InnoDB: Database was not shutdown normally! 140820 15:07:33 [Note] InnoDB: Starting crash recovery. 140820 15:07:33 [Note] InnoDB: Reading tablespace information from the .ibd files... 140820 15:07:34 [Note] InnoDB: Restoring possible half-written data pages 140820 15:07:34 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7682532586338 140820 15:07:34 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 50721724, file name /mysql_logs/mysql_products_3104/mysql-bin.013403 140820 15:07:36 [Note] InnoDB: 128 rollback segment(s) are active. 140820 15:07:36 [Note] InnoDB: Waiting for purge to start 140820 15:07:36 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7682532586338 140820 15:07:36 [Note] Plugin 'FEEDBACK' is disabled. 140820 15:07:36 server_audit: server_audit_excl_users set to 'root'. 140820 15:07:36 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140820 15:07:36 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140820 15:07:36 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140820 15:07:36 [Note] Starting crash recovery... 140820 15:07:36 [Note] Crash recovery finished. 140820 15:07:36 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-20 15:07:36 7fb63d33a760 InnoDB: Error: table `mysql_tmp`.`#sql208e_5_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140820 15:07:36 [Note] Event Scheduler: Loaded 0 events 140820 15:07:36 [Note] Event Scheduler: scheduler thread started with id 2 140820 15:07:36 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140820 15:08:48 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=15 max_threads=1002 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7fb46b4d0008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 140820 15:08:48 mysqld_safe Number of processes running now: 0 140820 15:08:48 mysqld_safe mysqld restarted 140820 15:08:49 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140820 15:08:49 [Note] InnoDB: The InnoDB memory heap is disabled 140820 15:08:49 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140820 15:08:49 [Note] InnoDB: Compressed tables use zlib 1.2.3 140820 15:08:49 [Note] InnoDB: Using Linux native AIO 140820 15:08:49 [Note] InnoDB: Not using CPU crc32 instructions 140820 15:08:49 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140820 15:08:50 [Note] InnoDB: Completed initialization of buffer pool 140820 15:08:50 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140820 15:08:50 [Note] InnoDB: Highest supported file format is Barracuda. 140820 15:08:50 [Note] InnoDB: The log sequence numbers 7569990467767 and 7569990467767 in ibdata files do not match the log sequence number 7682532626874 in the ib_logfiles! 140820 15:08:50 [Note] InnoDB: Database was not shutdown normally! 140820 15:08:50 [Note] InnoDB: Starting crash recovery. 140820 15:08:50 [Note] InnoDB: Reading tablespace information from the .ibd files... 140820 15:08:51 [Note] InnoDB: Restoring possible half-written data pages 140820 15:08:51 [Note] InnoDB: from the doublewrite buffer... InnoDB: Last MySQL binlog file position 0 598, file name /mysql_logs/mysql_products_3104/mysql-bin.013405 140820 15:08:52 [Note] InnoDB: 128 rollback segment(s) are active. 140820 15:08:52 [Note] InnoDB: Waiting for purge to start 140820 15:08:52 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7682532626874 140820 15:08:52 [Note] Plugin 'FEEDBACK' is disabled. 140820 15:08:52 server_audit: server_audit_excl_users set to 'root'. 140820 15:08:52 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140820 15:08:52 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140820 15:08:52 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140820 15:08:52 [Note] Starting crash recovery... 140820 15:08:52 [Note] Crash recovery finished. 140820 15:08:52 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-20 15:08:52 7f1a99543760 InnoDB: Error: table `mysql_tmp`.`#sql2164_18_1` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 2014-08-20 15:08:52 7f1a99543760 InnoDB: Error: table `mysql_tmp`.`#sql2164_18_2` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140820 15:08:52 [Note] Event Scheduler: Loaded 0 events 140820 15:08:52 [Note] Event Scheduler: scheduler thread started with id 2 140820 15:08:52 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140820 15:09:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=12 max_threads=1002 thread_count=10 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f18c7bf5008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1a9805bde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c61a1] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f18c4f108d0): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MktVenIds ttmvIds ON mv.marketVenueId=ttmvIds.ttv_MarketVenueId WHERE mi.marketSequence = 1 AND p.rowStatusId < NAME_CONST('compareStatus',2) AND pt.productTypeCode = 'DBT' AND i.rowStatusId < NAME_CONST('compareStatus',2) AND mkt.rowStatusId < NAME_CONST('compareStatus',2) AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= IF ( NAME_CONST('IncludeInActive',_utf8'N' COLLATE 'utf8_general_ci')='Y', 0, CONVERT(CURDATE(),UNSIGNED)) GROUP BY mv.marketId Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140820 15:09:42 mysqld_safe Number of processes running now: 0 140820 15:09:42 mysqld_safe mysqld restarted 140820 15:09:43 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140820 15:09:43 [Note] InnoDB: The InnoDB memory heap is disabled 140820 15:09:43 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140820 15:09:43 [Note] InnoDB: Compressed tables use zlib 1.2.3 140820 15:09:43 [Note] InnoDB: Using Linux native AIO 140820 15:09:43 [Note] InnoDB: Not using CPU crc32 instructions 140820 15:09:43 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140820 15:09:44 [Note] InnoDB: Completed initialization of buffer pool 140820 15:09:44 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140820 15:09:44 [Note] InnoDB: Highest supported file format is Barracuda. 140820 15:09:44 [Note] InnoDB: The log sequence numbers 7569990467767 and 7569990467767 in ibdata files do not match the log sequence number 7682532647813 in the ib_logfiles! 140820 15:09:44 [Note] InnoDB: Database was not shutdown normally! 140820 15:09:44 [Note] InnoDB: Starting crash recovery. 140820 15:09:44 [Note] InnoDB: Reading tablespace information from the .ibd files... 140820 15:09:45 [Note] InnoDB: Restoring possible half-written data pages 140820 15:09:45 [Note] InnoDB: from the doublewrite buffer... InnoDB: Last MySQL binlog file position 0 598, file name /mysql_logs/mysql_products_3104/mysql-bin.013405 140820 15:09:46 [Note] InnoDB: 128 rollback segment(s) are active. 140820 15:09:46 [Note] InnoDB: Waiting for purge to start 140820 15:09:46 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7682532647813 140820 15:09:46 [Note] Plugin 'FEEDBACK' is disabled. 140820 15:09:46 server_audit: server_audit_excl_users set to 'root'. 140820 15:09:46 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140820 15:09:46 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140820 15:09:46 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140820 15:09:46 [Note] Starting crash recovery... 140820 15:09:46 [Note] Crash recovery finished. 140820 15:09:46 [Note] Server socket created on IP: '0.0.0.0'. 140820 15:09:46 [Note] Event Scheduler: Loaded 0 events 140820 15:09:46 [Note] Event Scheduler: scheduler thread started with id 2 140820 15:09:46 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140820 16:55:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=28 max_threads=1002 thread_count=20 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7ff1624b5008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff310a12de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff14eeb5020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 28 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140820 16:55:09 mysqld_safe Number of processes running now: 0 140820 16:55:09 mysqld_safe mysqld restarted 140820 16:55:10 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140820 16:55:10 [Note] InnoDB: The InnoDB memory heap is disabled 140820 16:55:10 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140820 16:55:10 [Note] InnoDB: Compressed tables use zlib 1.2.3 140820 16:55:10 [Note] InnoDB: Using Linux native AIO 140820 16:55:10 [Note] InnoDB: Not using CPU crc32 instructions 140820 16:55:10 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140820 16:55:11 [Note] InnoDB: Completed initialization of buffer pool 140820 16:55:12 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140820 16:55:12 [Note] InnoDB: Highest supported file format is Barracuda. 140820 16:55:12 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7682661775010 140820 16:55:12 [Note] InnoDB: Database was not shutdown normally! 140820 16:55:12 [Note] InnoDB: Starting crash recovery. 140820 16:55:12 [Note] InnoDB: Reading tablespace information from the .ibd files... 140820 16:55:15 [Note] InnoDB: Restoring possible half-written data pages 140820 16:55:15 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7682661783191 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 2 row operations to undo InnoDB: Trx id counter is 918420480 140820 16:55:30 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 29671805, file name /mysql_logs/mysql_products_3104/mysql-bin.013407 140820 16:55:32 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-20 16:55:32 7effe5fff700 InnoDB: Rolling back trx with id 918365285, 2 rows to undo 140820 16:55:32 [Note] InnoDB: Rollback of trx with id 918365285 completed 2014-08-20 16:55:32 7effe5fff700 InnoDB: Rollback of non-prepared transactions completed 140820 16:55:32 [Note] InnoDB: Waiting for purge to start 140820 16:55:32 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7682661783191 140820 16:55:32 [Note] Plugin 'FEEDBACK' is disabled. 140820 16:55:32 server_audit: server_audit_excl_users set to 'root'. 140820 16:55:32 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140820 16:55:32 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140820 16:55:32 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140820 16:55:33 [Note] Starting crash recovery... 140820 16:55:33 [Note] Crash recovery finished. 140820 16:55:34 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-20 16:55:35 7f01be5d4760 InnoDB: Error: table `mysql_tmp`.`#sql23a4_2db_2` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 2014-08-20 16:55:35 7f01be5d4760 InnoDB: Error: table `mysql_tmp`.`#sql23a4_1c_16` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 2014-08-20 16:55:35 7f01be5d4760 InnoDB: Error: table `mysql_tmp`.`#sql23a4_2db_1` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140820 16:55:35 [Note] Event Scheduler: Loaded 0 events 140820 16:55:35 [Note] Event Scheduler: scheduler thread started with id 2 140820 16:55:35 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140820 16:58:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=8 max_threads=1002 thread_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7effecba2008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f01be4adde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c61a1] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7effe87fe8d0): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MktVenIds ttmvIds ON mv.marketVenueId=ttmvIds.ttv_MarketVenueId WHERE mi.marketSequence = 1 AND p.rowStatusId < NAME_CONST('compareStatus',2) AND pt.productTypeCode = 'DBT' AND i.rowStatusId < NAME_CONST('compareStatus',2) AND mkt.rowStatusId < NAME_CONST('compareStatus',2) AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= IF ( NAME_CONST('IncludeInActive',_utf8'N' COLLATE 'utf8_general_ci')='Y', 0, CONVERT(CURDATE(),UNSIGNED)) GROUP BY mv.marketId Connection ID (thread ID): 15 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140820 16:58:07 mysqld_safe Number of processes running now: 0 140820 16:58:07 mysqld_safe mysqld restarted 140820 16:58:08 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140820 16:58:08 [Note] InnoDB: The InnoDB memory heap is disabled 140820 16:58:08 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140820 16:58:08 [Note] InnoDB: Compressed tables use zlib 1.2.3 140820 16:58:08 [Note] InnoDB: Using Linux native AIO 140820 16:58:08 [Note] InnoDB: Not using CPU crc32 instructions 140820 16:58:08 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140820 16:58:09 [Note] InnoDB: Completed initialization of buffer pool 140820 16:58:09 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140820 16:58:09 [Note] InnoDB: Highest supported file format is Barracuda. 140820 16:58:09 [Note] InnoDB: The log sequence numbers 7569990467767 and 7569990467767 in ibdata files do not match the log sequence number 7682661800341 in the ib_logfiles! 140820 16:58:09 [Note] InnoDB: Database was not shutdown normally! 140820 16:58:09 [Note] InnoDB: Starting crash recovery. 140820 16:58:09 [Note] InnoDB: Reading tablespace information from the .ibd files... 140820 16:58:10 [Note] InnoDB: Restoring possible half-written data pages 140820 16:58:10 [Note] InnoDB: from the doublewrite buffer... InnoDB: Last MySQL binlog file position 0 870, file name /mysql_logs/mysql_products_3104/mysql-bin.013408 140820 16:58:13 [Note] InnoDB: 128 rollback segment(s) are active. 140820 16:58:13 [Note] InnoDB: Waiting for purge to start 140820 16:58:13 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7682661800341 140820 16:58:13 [Note] Plugin 'FEEDBACK' is disabled. 140820 16:58:13 server_audit: server_audit_excl_users set to 'root'. 140820 16:58:13 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140820 16:58:13 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140820 16:58:13 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140820 16:58:13 [Note] Starting crash recovery... 140820 16:58:13 [Note] Crash recovery finished. 140820 16:58:13 [Note] Server socket created on IP: '0.0.0.0'. 140820 16:58:13 [Note] Event Scheduler: Loaded 0 events 140820 16:58:13 [Note] Event Scheduler: scheduler thread started with id 2 140820 16:58:13 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140820 17:02:54 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=17 max_threads=1002 thread_count=18 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f629d4ac008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f644b937de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c61a1] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f629b2a68d0): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MktVenIds ttmvIds ON mv.marketVenueId=ttmvIds.ttv_MarketVenueId WHERE mi.marketSequence = 1 AND p.rowStatusId < NAME_CONST('compareStatus',2) AND pt.productTypeCode = 'DBT' AND i.rowStatusId < NAME_CONST('compareStatus',2) AND mkt.rowStatusId < NAME_CONST('compareStatus',2) AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= IF ( NAME_CONST('IncludeInActive',_utf8'N' COLLATE 'utf8_general_ci')='Y', 0, CONVERT(CURDATE(),UNSIGNED)) GROUP BY mv.marketId Connection ID (thread ID): 49 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140820 17:02:55 mysqld_safe Number of processes running now: 0 140820 17:02:55 mysqld_safe mysqld restarted 140820 17:02:56 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140820 17:02:56 [Note] InnoDB: The InnoDB memory heap is disabled 140820 17:02:56 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140820 17:02:56 [Note] InnoDB: Compressed tables use zlib 1.2.3 140820 17:02:56 [Note] InnoDB: Using Linux native AIO 140820 17:02:56 [Note] InnoDB: Not using CPU crc32 instructions 140820 17:02:56 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140820 17:02:57 [Note] InnoDB: Completed initialization of buffer pool 140820 17:02:57 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140820 17:02:57 [Note] InnoDB: Highest supported file format is Barracuda. 140820 17:02:57 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7682662248558 140820 17:02:57 [Note] InnoDB: Database was not shutdown normally! 140820 17:02:57 [Note] InnoDB: Starting crash recovery. 140820 17:02:57 [Note] InnoDB: Reading tablespace information from the .ibd files... 140820 17:02:59 [Note] InnoDB: Restoring possible half-written data pages 140820 17:02:59 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7682665590161 InnoDB: Transaction 918423385 was in the XA prepared state. InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 0 row operations to undo InnoDB: Trx id counter is 918423808 140820 17:03:04 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 2154146, file name /mysql_logs/mysql_products_3104/mysql-bin.013409 140820 17:03:07 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-20 17:03:07 7fb5b8bff700 InnoDB: Rollback of non-prepared transactions completed 140820 17:03:10 [Note] InnoDB: Waiting for purge to start 140820 17:03:10 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7682665590161 140820 17:03:10 [Note] Plugin 'FEEDBACK' is disabled. 140820 17:03:10 server_audit: server_audit_excl_users set to 'root'. 140820 17:03:10 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140820 17:03:10 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140820 17:03:10 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140820 17:03:10 [Note] Starting crash recovery... 2014-08-20 17:03:10 7fb791144760 InnoDB: Starting recovery for XA transactions... 2014-08-20 17:03:10 7fb791144760 InnoDB: Transaction 918423385 in prepared state after recovery 2014-08-20 17:03:10 7fb791144760 InnoDB: Transaction contains changes to 88 rows 2014-08-20 17:03:10 7fb791144760 InnoDB: 1 transactions in prepared state after recovery 140820 17:03:10 [Note] Found 1 prepared transaction(s) in InnoDB 140820 17:03:11 [Note] Crash recovery finished. 140820 17:03:12 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-20 17:03:12 7fb791144760 InnoDB: Error: table `mysql_tmp`.`#sql4852_2e_14` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 2014-08-20 17:03:12 7fb791144760 InnoDB: Error: table `mysql_tmp`.`#sql4852_2b_6` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140820 17:03:12 [Note] Event Scheduler: Loaded 0 events 140820 17:03:12 [Note] Event Scheduler: scheduler thread started with id 2 140820 17:03:12 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140820 17:07:17 [Note] /data/mysql/bin/mysqld: Normal shutdown 140820 17:07:17 [Note] Event Scheduler: Killing the scheduler thread, thread id 2 140820 17:07:17 [Note] Event Scheduler: Waiting for the scheduler thread to reply 140820 17:07:17 [Note] Event Scheduler: Stopped 140820 17:07:17 [Note] Event Scheduler: Purging the queue. 0 events 140820 17:07:17 server_audit: STOPPED 140820 17:07:17 [Note] InnoDB: FTS optimize thread exiting. 140820 17:07:17 [Note] InnoDB: Starting shutdown... 140820 17:07:19 [Note] InnoDB: Shutdown completed; log sequence number 7682665707857 140820 17:07:19 [Note] /data/mysql/bin/mysqld: Shutdown complete 140820 17:07:20 mysqld_safe mysqld from pid file /data/mysql/socket/products_3104.pid ended 140820 17:07:27 mysqld_safe Starting mysqld daemon with databases from /mysql_data/mysql_products_3104 140820 17:07:28 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140820 17:07:28 [Note] InnoDB: The InnoDB memory heap is disabled 140820 17:07:28 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140820 17:07:28 [Note] InnoDB: Compressed tables use zlib 1.2.3 140820 17:07:28 [Note] InnoDB: Using Linux native AIO 140820 17:07:28 [Note] InnoDB: Not using CPU crc32 instructions 140820 17:07:28 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140820 17:07:28 [Note] InnoDB: Completed initialization of buffer pool 140820 17:07:29 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140820 17:07:29 [Note] InnoDB: Highest supported file format is Barracuda. 140820 17:07:32 [Note] InnoDB: 128 rollback segment(s) are active. 140820 17:07:33 [Note] InnoDB: Waiting for purge to start 140820 17:07:33 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7682665707857 140820 17:07:33 [Note] Plugin 'FEEDBACK' is disabled. 140820 17:07:33 server_audit: server_audit_excl_users set to 'root'. 140820 17:07:33 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140820 17:07:33 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140820 17:07:34 [Note] Server socket created on IP: '0.0.0.0'. 140820 17:07:34 [Note] Event Scheduler: Loaded 0 events 140820 17:07:34 [Note] Event Scheduler: scheduler thread started with id 2 140820 17:07:34 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140821 15:14:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=37 max_threads=1002 thread_count=21 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f0ca57bd008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0ca376cde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0cb8f82020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 6937 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140821 15:14:09 mysqld_safe Number of processes running now: 0 140821 15:14:09 mysqld_safe mysqld restarted 140821 15:14:09 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140821 15:14:09 [Note] InnoDB: The InnoDB memory heap is disabled 140821 15:14:09 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140821 15:14:09 [Note] InnoDB: Compressed tables use zlib 1.2.3 140821 15:14:09 [Note] InnoDB: Using Linux native AIO 140821 15:14:09 [Note] InnoDB: Not using CPU crc32 instructions 140821 15:14:09 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140821 15:14:10 [Note] InnoDB: Completed initialization of buffer pool 140821 15:14:10 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140821 15:14:10 [Note] InnoDB: Highest supported file format is Barracuda. 140821 15:14:10 [Note] InnoDB: The log sequence numbers 7682665707857 and 7682665707857 in ibdata files do not match the log sequence number 7684618754799 in the ib_logfiles! 140821 15:14:10 [Note] InnoDB: Database was not shutdown normally! 140821 15:14:10 [Note] InnoDB: Starting crash recovery. 140821 15:14:10 [Note] InnoDB: Reading tablespace information from the .ibd files... 140821 15:14:13 [Note] InnoDB: Restoring possible half-written data pages 140821 15:14:13 [Note] InnoDB: from the doublewrite buffer... InnoDB: Last MySQL binlog file position 0 46378444, file name /mysql_logs/mysql_products_3104/mysql-bin.013411 140821 15:14:18 [Note] InnoDB: 128 rollback segment(s) are active. 140821 15:14:18 [Note] InnoDB: Waiting for purge to start 140821 15:14:18 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7684618754799 140821 15:14:18 [Note] Plugin 'FEEDBACK' is disabled. 140821 15:14:18 server_audit: server_audit_excl_users set to 'root'. 140821 15:14:18 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140821 15:14:18 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140821 15:14:18 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140821 15:14:19 [Note] Starting crash recovery... 140821 15:14:19 [Note] Crash recovery finished. 140821 15:14:19 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-21 15:14:19 7f58c87b2760 InnoDB: Error: table `mysql_tmp`.`#sql5e43_1b19_1c` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140821 15:14:19 [Note] Event Scheduler: Loaded 0 events 140821 15:14:19 [Note] Event Scheduler: scheduler thread started with id 2 140821 15:14:19 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140821 15:30:51 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=30 max_threads=1002 thread_count=26 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f56f6a4f008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f58c868bde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c6186] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f56f4e7c020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 5 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140821 15:30:52 mysqld_safe Number of processes running now: 0 140821 15:30:52 mysqld_safe mysqld restarted 140821 15:30:53 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140821 15:30:53 [Note] InnoDB: The InnoDB memory heap is disabled 140821 15:30:53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140821 15:30:53 [Note] InnoDB: Compressed tables use zlib 1.2.3 140821 15:30:53 [Note] InnoDB: Using Linux native AIO 140821 15:30:53 [Note] InnoDB: Not using CPU crc32 instructions 140821 15:30:53 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140821 15:30:53 [Note] InnoDB: Completed initialization of buffer pool 140821 15:30:54 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140821 15:30:54 [Note] InnoDB: Highest supported file format is Barracuda. 140821 15:30:54 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7684722469700 140821 15:30:54 [Note] InnoDB: Database was not shutdown normally! 140821 15:30:54 [Note] InnoDB: Starting crash recovery. 140821 15:30:54 [Note] InnoDB: Reading tablespace information from the .ibd files... 140821 15:30:56 [Note] InnoDB: Restoring possible half-written data pages 140821 15:30:56 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7684727712256 InnoDB: Doing recovery: scanned up to log sequence number 7684732955136 InnoDB: Doing recovery: scanned up to log sequence number 7684738198016 InnoDB: Doing recovery: scanned up to log sequence number 7684743440896 InnoDB: Doing recovery: scanned up to log sequence number 7684747069172 140821 15:31:02 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 40896337, file name /mysql_logs/mysql_products_3104/mysql-bin.013412 140821 15:31:06 [Note] InnoDB: 128 rollback segment(s) are active. 140821 15:31:06 [Note] InnoDB: Waiting for purge to start 140821 15:31:06 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7684747069172 140821 15:31:06 [Note] Plugin 'FEEDBACK' is disabled. 140821 15:31:06 server_audit: server_audit_excl_users set to 'root'. 140821 15:31:06 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140821 15:31:06 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140821 15:31:06 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140821 15:31:06 [Note] Starting crash recovery... 140821 15:31:06 [Note] Crash recovery finished. 140821 15:31:06 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-21 15:31:06 7f9f19fd2760 InnoDB: Error: table `mysql_tmp`.`#sql19f0_5_17` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140821 15:31:07 [Note] Event Scheduler: Loaded 0 events 140821 15:31:07 [Note] Event Scheduler: scheduler thread started with id 2 140821 15:31:07 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140821 16:52:11 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=56 max_threads=1002 thread_count=49 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f9d464f1008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9d2b248de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c6186] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9d2ca2c020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 502 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140821 16:52:15 mysqld_safe Number of processes running now: 0 140821 16:52:15 mysqld_safe mysqld restarted 140821 16:52:16 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140821 16:52:16 [Note] InnoDB: The InnoDB memory heap is disabled 140821 16:52:16 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140821 16:52:16 [Note] InnoDB: Compressed tables use zlib 1.2.3 140821 16:52:16 [Note] InnoDB: Using Linux native AIO 140821 16:52:16 [Note] InnoDB: Not using CPU crc32 instructions 140821 16:52:16 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140821 16:52:16 [Note] InnoDB: Completed initialization of buffer pool 140821 16:52:17 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140821 16:52:17 [Note] InnoDB: Highest supported file format is Barracuda. 140821 16:52:17 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7685729254867 140821 16:52:17 [Note] InnoDB: Database was not shutdown normally! 140821 16:52:17 [Note] InnoDB: Starting crash recovery. 140821 16:52:17 [Note] InnoDB: Reading tablespace information from the .ibd files... 140821 16:52:23 [Note] InnoDB: Restoring possible half-written data pages 140821 16:52:23 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7685734497280 InnoDB: Doing recovery: scanned up to log sequence number 7685739740160 InnoDB: Doing recovery: scanned up to log sequence number 7685744983040 InnoDB: Doing recovery: scanned up to log sequence number 7685750225920 InnoDB: Doing recovery: scanned up to log sequence number 7685755468800 InnoDB: Doing recovery: scanned up to log sequence number 7685760711680 InnoDB: Doing recovery: scanned up to log sequence number 7685765954560 InnoDB: Doing recovery: scanned up to log sequence number 7685771197440 InnoDB: Doing recovery: scanned up to log sequence number 7685776440320 InnoDB: Doing recovery: scanned up to log sequence number 7685781683200 InnoDB: Doing recovery: scanned up to log sequence number 7685782027567 InnoDB: Transaction 918728066 was in the XA prepared state. InnoDB: 3 transaction(s) which must be rolled back or cleaned up InnoDB: in total 285 row operations to undo InnoDB: Trx id counter is 918728448 140821 16:52:39 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 183650833, file name /mysql_logs/mysql_products_3104/mysql-bin.013414 140821 16:52:50 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-21 16:52:50 7fcc9abff700 InnoDB: Rolling back trx with id 918728014, 75 rows to undo 140821 16:52:51 [Note] InnoDB: Rollback of trx with id 918728014 completed 2014-08-21 16:52:51 7fcc9abff700 InnoDB: Rolling back trx with id 918727966, 210 rows to undo 140821 16:52:57 [Note] InnoDB: Rollback of trx with id 918727966 completed 2014-08-21 16:52:57 7fcc9abff700 InnoDB: Rollback of non-prepared transactions completed 140821 16:53:04 [Note] InnoDB: Waiting for purge to start 140821 16:53:04 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7685782027567 140821 16:53:04 [Note] Plugin 'FEEDBACK' is disabled. 140821 16:53:04 server_audit: server_audit_excl_users set to 'root'. 140821 16:53:04 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140821 16:53:04 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140821 16:53:04 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140821 16:53:14 [Note] Starting crash recovery... 2014-08-21 16:53:14 7fce73110760 InnoDB: Starting recovery for XA transactions... 2014-08-21 16:53:14 7fce73110760 InnoDB: Transaction 918728066 in prepared state after recovery 2014-08-21 16:53:14 7fce73110760 InnoDB: Transaction contains changes to 14 rows 2014-08-21 16:53:14 7fce73110760 InnoDB: 1 transactions in prepared state after recovery 140821 16:53:14 [Note] Found 1 prepared transaction(s) in InnoDB 140821 16:53:14 [Note] Crash recovery finished. 140821 16:53:14 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-21 16:53:14 7fce73110760 InnoDB: Error: table `mysql_tmp`.`#sql38c7_1f6_2fb` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140821 16:53:14 [Note] Event Scheduler: Loaded 0 events 140821 16:53:14 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140821 16:53:14 [Note] Event Scheduler: scheduler thread started with id 2 140821 17:03:15 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=29 max_threads=1002 thread_count=25 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7fcca06a1008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fce71dfdde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c6186] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fcc9d78e020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 110 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140821 17:03:16 mysqld_safe Number of processes running now: 0 140821 17:03:16 mysqld_safe mysqld restarted 140821 17:03:17 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140821 17:03:17 [Note] InnoDB: The InnoDB memory heap is disabled 140821 17:03:17 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140821 17:03:17 [Note] InnoDB: Compressed tables use zlib 1.2.3 140821 17:03:17 [Note] InnoDB: Using Linux native AIO 140821 17:03:17 [Note] InnoDB: Not using CPU crc32 instructions 140821 17:03:17 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140821 17:03:18 [Note] InnoDB: Completed initialization of buffer pool 140821 17:03:18 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140821 17:03:18 [Note] InnoDB: Highest supported file format is Barracuda. 140821 17:03:18 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7685941619471 140821 17:03:18 [Note] InnoDB: Database was not shutdown normally! 140821 17:03:18 [Note] InnoDB: Starting crash recovery. 140821 17:03:18 [Note] InnoDB: Reading tablespace information from the .ibd files... 140821 17:03:21 [Note] InnoDB: Restoring possible half-written data pages 140821 17:03:21 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7685946862080 InnoDB: Doing recovery: scanned up to log sequence number 7685952104960 InnoDB: Doing recovery: scanned up to log sequence number 7685957347840 InnoDB: Doing recovery: scanned up to log sequence number 7685962590720 InnoDB: Doing recovery: scanned up to log sequence number 7685967833600 InnoDB: Doing recovery: scanned up to log sequence number 7685973076480 InnoDB: Doing recovery: scanned up to log sequence number 7685978319360 InnoDB: Doing recovery: scanned up to log sequence number 7685983562240 InnoDB: Doing recovery: scanned up to log sequence number 7685988805120 InnoDB: Doing recovery: scanned up to log sequence number 7685989582915 InnoDB: 2 transaction(s) which must be rolled back or cleaned up InnoDB: in total 416 row operations to undo InnoDB: Trx id counter is 918795008 140821 17:03:33 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 87599675, file name /mysql_logs/mysql_products_3104/mysql-bin.013415 140821 17:03:44 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-21 17:03:44 7f88febff700 InnoDB: Rolling back trx with id 918794682, 82 rows to undo 140821 17:03:44 [Note] InnoDB: Rollback of trx with id 918794682 completed 2014-08-21 17:03:44 7f88febff700 InnoDB: Rolling back trx with id 918794648, 334 rows to undo 140821 17:03:48 [Note] InnoDB: Rollback of trx with id 918794648 completed 2014-08-21 17:03:48 7f88febff700 InnoDB: Rollback of non-prepared transactions completed 140821 17:03:51 [Note] InnoDB: Waiting for purge to start 140821 17:03:51 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7685989582915 140821 17:03:51 [Note] Plugin 'FEEDBACK' is disabled. 140821 17:03:51 server_audit: server_audit_excl_users set to 'root'. 140821 17:03:51 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140821 17:03:51 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140821 17:03:51 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140821 17:03:52 [Note] Starting crash recovery... 140821 17:03:52 [Note] Crash recovery finished. 140821 17:03:52 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-21 17:03:52 7f8ad73e4760 InnoDB: Error: table `mysql_tmp`.`#sql49bc_6e_3e5` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140821 17:03:52 [Note] Event Scheduler: Loaded 0 events 140821 17:03:52 [Note] Event Scheduler: scheduler thread started with id 2 140821 17:03:52 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140821 17:19:18 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=29 max_threads=1002 thread_count=27 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f8904fdf008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f8ab385cde0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c61a1] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f88fac3f020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 38 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140821 17:19:19 mysqld_safe Number of processes running now: 0 140821 17:19:19 mysqld_safe mysqld restarted 140821 17:19:20 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140821 17:19:20 [Note] InnoDB: The InnoDB memory heap is disabled 140821 17:19:20 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140821 17:19:20 [Note] InnoDB: Compressed tables use zlib 1.2.3 140821 17:19:20 [Note] InnoDB: Using Linux native AIO 140821 17:19:20 [Note] InnoDB: Not using CPU crc32 instructions 140821 17:19:20 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140821 17:19:21 [Note] InnoDB: Completed initialization of buffer pool 140821 17:19:21 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140821 17:19:21 [Note] InnoDB: Highest supported file format is Barracuda. 140821 17:19:21 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7686206061405 140821 17:19:21 [Note] InnoDB: Database was not shutdown normally! 140821 17:19:21 [Note] InnoDB: Starting crash recovery. 140821 17:19:21 [Note] InnoDB: Reading tablespace information from the .ibd files... 140821 17:19:24 [Note] InnoDB: Restoring possible half-written data pages 140821 17:19:24 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7686211303936 InnoDB: Doing recovery: scanned up to log sequence number 7686216546816 InnoDB: Doing recovery: scanned up to log sequence number 7686221789696 InnoDB: Doing recovery: scanned up to log sequence number 7686227032576 InnoDB: Doing recovery: scanned up to log sequence number 7686232275456 InnoDB: Doing recovery: scanned up to log sequence number 7686237518336 InnoDB: Doing recovery: scanned up to log sequence number 7686242761216 InnoDB: Doing recovery: scanned up to log sequence number 7686248004096 InnoDB: Doing recovery: scanned up to log sequence number 7686253246976 InnoDB: Doing recovery: scanned up to log sequence number 7686258489856 InnoDB: Doing recovery: scanned up to log sequence number 7686259499949 InnoDB: 2 transaction(s) which must be rolled back or cleaned up InnoDB: in total 643 row operations to undo InnoDB: Trx id counter is 918812928 140821 17:19:30 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 145324933, file name /mysql_logs/mysql_products_3104/mysql-bin.013416 140821 17:20:05 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2014-08-21 17:20:05 7f39f6bff700 InnoDB: Rolling back trx with id 918812430, 428 rows to undo 140821 17:20:06 [Note] InnoDB: Rollback of trx with id 918812430 completed 2014-08-21 17:20:06 7f39f6bff700 InnoDB: Rolling back trx with id 918812420, 215 rows to undo 140821 17:20:10 [Note] InnoDB: Rollback of trx with id 918812420 completed 2014-08-21 17:20:10 7f39f6bff700 InnoDB: Rollback of non-prepared transactions completed 140821 17:20:31 [Note] InnoDB: Waiting for purge to start 140821 17:20:31 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7686259499949 140821 17:20:31 [Note] Plugin 'FEEDBACK' is disabled. 140821 17:20:31 server_audit: server_audit_excl_users set to 'root'. 140821 17:20:31 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140821 17:20:31 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140821 17:20:31 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140821 17:20:32 [Note] Starting crash recovery... 140821 17:20:32 [Note] Crash recovery finished. 140821 17:20:33 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-21 17:20:33 7f3bcf078760 InnoDB: Error: table `mysql_tmp`.`#sql6075_26_13d5` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140821 17:20:33 [Note] Event Scheduler: Loaded 0 events 140821 17:20:33 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140821 17:20:33 [Note] Event Scheduler: scheduler thread started with id 2 140821 18:10:35 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=33 max_threads=1002 thread_count=25 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f39eac4d008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3ba0706de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c6186] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f39de6398d0): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MktVenIds ttmvIds ON mv.marketVenueId=ttmvIds.ttv_MarketVenueId WHERE mi.marketSequence = 1 AND p.rowStatusId < NAME_CONST('compareStatus',2) AND pt.productTypeCode = 'DBT' AND i.rowStatusId < NAME_CONST('compareStatus',2) AND mkt.rowStatusId < NAME_CONST('compareStatus',2) AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= IF ( NAME_CONST('IncludeInActive',_utf8'N' COLLATE 'utf8_general_ci')='Y', 0, CONVERT(CURDATE(),UNSIGNED)) GROUP BY mv.marketId Connection ID (thread ID): 323 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140821 18:10:37 mysqld_safe Number of processes running now: 0 140821 18:10:37 mysqld_safe mysqld restarted 140821 18:10:38 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140821 18:10:38 [Note] InnoDB: The InnoDB memory heap is disabled 140821 18:10:38 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140821 18:10:38 [Note] InnoDB: Compressed tables use zlib 1.2.3 140821 18:10:38 [Note] InnoDB: Using Linux native AIO 140821 18:10:38 [Note] InnoDB: Not using CPU crc32 instructions 140821 18:10:38 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140821 18:10:39 [Note] InnoDB: Completed initialization of buffer pool 140821 18:10:39 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140821 18:10:39 [Note] InnoDB: Highest supported file format is Barracuda. 140821 18:10:39 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7686825797566 140821 18:10:39 [Note] InnoDB: Database was not shutdown normally! 140821 18:10:39 [Note] InnoDB: Starting crash recovery. 140821 18:10:39 [Note] InnoDB: Reading tablespace information from the .ibd files... 140821 18:10:42 [Note] InnoDB: Restoring possible half-written data pages 140821 18:10:42 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7686826169981 140821 18:10:51 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 146864658, file name /mysql_logs/mysql_products_3104/mysql-bin.013418 140821 18:10:54 [Note] InnoDB: 128 rollback segment(s) are active. 140821 18:10:54 [Note] InnoDB: Waiting for purge to start 140821 18:10:54 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7686826169981 140821 18:10:54 [Note] Plugin 'FEEDBACK' is disabled. 140821 18:10:54 server_audit: server_audit_excl_users set to 'root'. 140821 18:10:54 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140821 18:10:54 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140821 18:10:54 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140821 18:10:55 [Note] Starting crash recovery... 140821 18:10:55 [Note] Crash recovery finished. 140821 18:10:56 [Note] Server socket created on IP: '0.0.0.0'. 140821 18:10:56 [Note] Event Scheduler: Loaded 0 events 140821 18:10:56 [Note] Event Scheduler: scheduler thread started with id 2 140821 18:10:56 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server 140821 18:15:05 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.12-MariaDB-log key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=23 max_threads=1002 thread_count=22 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 823012 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f1e1264e008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1fe2d36de0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbb674e] sql/signal_handler.cc:153(handle_fatal_signal)[0x7152ec] /lib64/libpthread.so.0[0x30ece0f4a0] sql/item_sum.cc:3428(Item_func_group_concat::fix_fields(THD*, Item**))[0x7c6186] sql/sql_base.cc:7778(setup_fields(THD*, Item**, List&, enum_mark_columns, List*, bool))[0x54a459] sql/sql_select.cc:781(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x5c4386] sql/sql_select.cc:3281(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5e738a] sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5ea71d] sql/sql_parse.cc:3529(mysql_execute_command(THD*))[0x59a5e5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sp_head.cc:3191(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x83b56e] sql/sp_head.cc:2959(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x83b70c] sql/sp_head.cc:3108(sp_instr_stmt::execute(THD*, unsigned int*))[0x841902] sql/sp_head.cc:1366(sp_head::execute(THD*, bool))[0x83f71c] sql/sp_head.cc:2156(sp_head::execute_procedure(THD*, List*))[0x840ef3] sql/sql_parse.cc:4690(mysql_execute_command(THD*))[0x5954f5] sql/sql_parse.cc:6409(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59b2da] sql/sql_parse.cc:1311(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x59d087] sql/sql_parse.cc:1006(do_command(THD*))[0x59d7a5] sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x664874] sql/sql_connect.cc:1295(handle_one_connection)[0x6649b2] perfschema/pfs.cc:1863(pfs_spawn_thread)[0x8b6739] /lib64/libpthread.so.0[0x30ece077f1] /lib64/libc.so.6(clone+0x6d)[0x30ec6e5ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f1e0e510020): INSERT INTO tempdb.tt_markets SELECT mv.marketId, NULL, GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.symbolCode,v.symbolCode), v.symbolCode) ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(v.`permissionsCode` ORDER BY 1 SEPARATOR ',' ), GROUP_CONCAT(IF (tp.overridesExchangeCode, IFNULL(tp.tradingPlatformName, v.venueAcronym), v.venueAcronym) ORDER BY 1 SEPARATOR ',' ) FROM marketvenues mv JOIN market mkt ON mv.marketId = mkt.marketId JOIN markettemplate mt ON mkt.marketTemplateId = mt.marketTemplateId JOIN marketinstruments mi ON mkt.marketId = mi.marketId JOIN instrument i ON i.instrumentId = mi.instrumentId JOIN debtinstrument di ON i.instrumentId = di.instrumentId JOIN product p ON i.productId = p.productId JOIN producttype pt ON p.productTypeId = pt.productTypeId JOIN venue v ON mv.venueId = v.venueId LEFT JOIN tradingplatform tp ON tp.tradingPlatformId = v.tradingPlatformId JOIN tempdb.tt_MarketVenueIds ttmvIds ON mv.marketVenueId=ttmvIds.marketvenueId WHERE mi.marketSequence = 1 AND pt.productTypeCode = 'DBT' AND p.rowStatusId = 1 AND i.rowStatusId = 1 AND mkt.rowStatusId = 1 AND mt.marketTemplateCode = "OUT" AND di.maturityDate >= CONVERT(CURDATE(),UNSIGNED) GROUP BY mv.marketId Connection ID (thread ID): 5 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140821 18:15:07 mysqld_safe Number of processes running now: 0 140821 18:15:07 mysqld_safe mysqld restarted 140821 18:15:07 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140821 18:15:07 [Note] InnoDB: The InnoDB memory heap is disabled 140821 18:15:07 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140821 18:15:07 [Note] InnoDB: Compressed tables use zlib 1.2.3 140821 18:15:07 [Note] InnoDB: Using Linux native AIO 140821 18:15:07 [Note] InnoDB: Not using CPU crc32 instructions 140821 18:15:07 [Note] InnoDB: Initializing buffer pool, size = 6.0G 140821 18:15:08 [Note] InnoDB: Completed initialization of buffer pool 140821 18:15:09 [ERROR] InnoDB: Space id in fsp header 2112613,but in the page header 0 140821 18:15:09 [Note] InnoDB: Highest supported file format is Barracuda. 140821 18:15:09 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7686827294332 140821 18:15:09 [Note] InnoDB: Database was not shutdown normally! 140821 18:15:09 [Note] InnoDB: Starting crash recovery. 140821 18:15:09 [Note] InnoDB: Reading tablespace information from the .ibd files... 140821 18:15:11 [Note] InnoDB: Restoring possible half-written data pages 140821 18:15:11 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 7686827361647 140821 18:15:16 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 20944, file name /mysql_logs/mysql_products_3104/mysql-bin.013419 140821 18:15:19 [Note] InnoDB: 128 rollback segment(s) are active. 140821 18:15:19 [Note] InnoDB: Waiting for purge to start 140821 18:15:19 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 7686827361647 140821 18:15:19 [Note] Plugin 'FEEDBACK' is disabled. 140821 18:15:19 server_audit: server_audit_excl_users set to 'root'. 140821 18:15:19 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED. 140821 18:15:19 server_audit: logging started to the file /data/mysql/logs/server_audit_3104.log. 140821 18:15:19 [Note] Recovering after a crash using /mysql_logs/mysql_products_3104/mysql-bin 140821 18:15:19 [Note] Starting crash recovery... 140821 18:15:19 [Note] Crash recovery finished. 140821 18:15:19 [Note] Server socket created on IP: '0.0.0.0'. 2014-08-21 18:15:19 7fde32107760 InnoDB: Error: table `mysql_tmp`.`#sql62e6_5_13` does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 140821 18:15:19 [Note] Event Scheduler: Loaded 0 events 140821 18:15:19 [Note] Event Scheduler: scheduler thread started with id 2 140821 18:15:19 [Note] /data/mysql/bin/mysqld: ready for connections. Version: '10.0.12-MariaDB-log' socket: '/data/mysql/socket/products_3104.sock' port: 3104 MariaDB Server