Crash on querying I_S.xtradb_rseg when innodb_rollback_segments/innodb_undo_logs lower then default

Bug #1260259 reported by Przemek
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
High
Laurynas Biveinis
5.1
Invalid
Undecided
Unassigned
5.5
Invalid
Undecided
Unassigned
5.6
Fix Released
High
Laurynas Biveinis

Bug Description

Since MySQL 5.6 or Percona Server 5.6, it is possible to initialize innodb system table space with lower then default (128) number of rollback segments. Unfortunately this leads to server crash when selecting from information_schema.xtradb_rseg.
If the number of rollback_segments is default 128, no crash happens. Applies only to PS 5.6.x as there seems to be no way to initialize lower then default number in MySQL/PS 5.5.x

How to quickly reproduce using mysqlsandbox:

* edit MySQL configuration (my.sandbox.cnf) and add:
innodb_undo_logs=32

* re-initalize InnoDB tablespaces, from sandbox directory run:
./clear

* start Percona Server - InnoDB table spaces and logs are re-created:
./start

* check related variables and status:

mysql [localhost] {msandbox} ((none)) > select @@innodb_rollback_segments;
+----------------------------+
| @@innodb_rollback_segments |
+----------------------------+
| 32 |
+----------------------------+
1 row in set (0.00 sec)

mysql [localhost] {msandbox} ((none)) > select @@innodb_undo_logs;
+--------------------+
| @@innodb_undo_logs |
+--------------------+
| 32 |
+--------------------+
1 row in set (0.01 sec)

mysql [localhost] {msandbox} ((none)) > show status like 'Innodb_available_undo_logs'; +----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| Innodb_available_undo_logs | 32 |
+----------------------------+-------+
1 row in set (0.00 sec)

* select from xtradb_rseg system view

mysql [localhost] {msandbox} ((none)) > select * from information_schema.XTRADB_RSEG;
ERROR 2013 (HY000): Lost connection to MySQL server during query

Error log:

2013-12-12 10:58:07 24485 [Note] /home/sandbox/tarballs/percona5.6.14/bin/mysqld: ready for connections.
Version: '5.6.14-rel62.0' socket: '/tmp/mysql_sandbox25614.sock' port: 25614 Percona Server with XtraDB (GPL), Release rel62.0, Revision 483

10:12:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69143 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x3362c10
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc3c3a42d58 thread_stack 0x40000
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(my_print_stacktrace+0x35)[0x8dc375]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(handle_fatal_signal+0x4c4)[0x662d64]
/lib64/libpthread.so.0[0x3e1060f000]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld[0x919920]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f1)[0x710681]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_ZN4JOIN14prepare_resultEPP4ListI4ItemE+0x9d)[0x704b6d]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_ZN4JOIN4execEv+0xdf)[0x6c31df]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x250)[0x705660]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x187)[0x705ee7]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld[0x6e100d]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_Z21mysql_execute_commandP3THD+0x191b)[0x6e3d4b]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x608)[0x6e7978]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x146f)[0x6e8e4f]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x16f)[0x6b5d5f]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(handle_one_connection+0x47)[0x6b5e37]
/home/sandbox/tarballs/percona5.6.14/bin/mysqld(pfs_spawn_thread+0x12a)[0xacde7a]
/lib64/libpthread.so.0[0x3e10607d15]
/lib64/libc.so.6(clone+0x6d)[0x3e102f253d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fc3b0004ff0): select * from information_schema.XTRADB_RSEG
Connection ID (thread ID): 1
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
131212 11:12:02 mysqld_safe Number of processes running now: 0
131212 11:12:02 mysqld_safe mysqld restarted
2013-12-12 11:12:02 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2013-12-12 11:12:02 24706 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)

2013-12-12 11:12:02 24706 [Warning] Buffered warning: Changed limits: table_cache: 431 (requested 2000)

2013-12-12 11:12:02 24706 [Note] Plugin 'FEDERATED' is disabled.
2013-12-12 11:12:02 24706 [Note] InnoDB: The InnoDB memory heap is disabled
2013-12-12 11:12:02 24706 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2013-12-12 11:12:02 24706 [Note] InnoDB: Compressed tables use zlib 1.2.3
2013-12-12 11:12:02 24706 [Note] InnoDB: Using Linux native AIO
2013-12-12 11:12:02 24706 [Note] InnoDB: Using CPU crc32 instructions
2013-12-12 11:12:02 24706 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2013-12-12 11:12:02 24706 [Note] InnoDB: Completed initialization of buffer pool
2013-12-12 11:12:02 24706 [Note] InnoDB: Highest supported file format is Barracuda.
2013-12-12 11:12:02 24706 [Note] InnoDB: The log sequence numbers 0 and 0 in ibdata files do not match the log sequence number 445136 in the ib_logfiles!
2013-12-12 11:12:02 24706 [Note] InnoDB: Database was not shutdown normally!
2013-12-12 11:12:02 24706 [Note] InnoDB: Starting crash recovery.
2013-12-12 11:12:02 24706 [Note] InnoDB: Reading tablespace information from the .ibd files...
2013-12-12 11:12:02 24706 [Note] InnoDB: Restoring possible half-written data pages
2013-12-12 11:12:02 24706 [Note] InnoDB: from the doublewrite buffer...
2013-12-12 11:12:03 24706 [Note] InnoDB: 32 rollback segment(s) are active.
2013-12-12 11:12:03 24706 [Note] InnoDB: Waiting for purge to start
2013-12-12 11:12:03 24706 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.14-rel62.0 started; log sequence number 445136
2013-12-12 11:12:03 24706 [Note] RSA private key file not found: /home/przemek/sandboxes/percona5.6.14/data//private_key.pem. Some authentication plugins will not work.
2013-12-12 11:12:03 24706 [Note] RSA public key file not found: /home/przemek/sandboxes/percona5.6.14/data//public_key.pem. Some authentication plugins will not work.
2013-12-12 11:12:03 24706 [Note] Server hostname (bind-address): '127.0.0.1'; port: 25614
2013-12-12 11:12:03 24706 [Note] - '127.0.0.1' resolves to '127.0.0.1';
2013-12-12 11:12:03 24706 [Note] Server socket created on IP: '127.0.0.1'.
2013-12-12 11:12:03 24706 [Note] Event Scheduler: Loaded 0 events
2013-12-12 11:12:03 24706 [Note] /home/sandbox/tarballs/percona5.6.14/bin/mysqld: ready for connections.
Version: '5.6.14-rel62.0' socket: '/tmp/mysql_sandbox25614.sock' port: 25614 Percona Server with XtraDB (GPL), Release rel62.0, Revision 483

Related branches

Przemek (pmalkowski)
tags: added: i37704
Revision history for this message
Valerii Kravchuk (valerii-kravchuk) wrote :
Download full text (5.1 KiB)

This is easy to confirm:

openxs@ao756:~/dbs/p5.6$ bin/mysql --no-defaults -uroot test
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.14-debug MySQL Community Server (GPL)

Copyright (c) 2009-2013 Percona LLC and/or its affiliates
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like 'innodb_undo%';
+-------------------------+----------------------------+
| Variable_name | Value |
+-------------------------+----------------------------+
| innodb_undo_directory | /home/openxs/dbs/p5.6/undo |
| innodb_undo_logs | 32 |
| innodb_undo_tablespaces | 1 |
+-------------------------+----------------------------+
3 rows in set (0,00 sec)

mysql> select * from information_schema.XTRADB_RSEG;
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 131212 12:26:36 mysqld_safe Number of processes running now: 0
131212 12:26:36 mysqld_safe mysqld restarted

mysql> exit
Bye
openxs@ao756:~/dbs/p5.6$ tail -100 data/ao756.err 2013-12-12 12:25:41 31028 [Note] InnoDB: 32 rollback segment(s) are active.
2013-12-12 12:25:41 31028 [Note] InnoDB: Waiting for purge to start
2013-12-12 12:25:41 31028 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.14-60.3 started; log sequence number 432817
2013-12-12 12:25:41 31028 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: c06f915e-6317-11e3-8907-b888e3aea471.
2013-12-12 12:25:41 31028 [Note] Server hostname (bind-address): '*'; port: 3306
2013-12-12 12:25:41 31028 [Note] IPv6 is available.
2013-12-12 12:25:41 31028 [Note] - '::' resolves to '::';
2013-12-12 12:25:41 31028 [Note] Server socket created on IP: '::'.
2013-12-12 12:25:41 31028 [Note] Event Scheduler: Loaded 0 events
2013-12-12 12:25:41 31028 [Note] /home/openxs/dbs/p5.6/bin/mysqld: ready for connections.
Version: '5.6.14-debug' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Server (GPL)
10:26:36 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69189 K bytes of memory
Hope that's ok; if not, decrease s...

Read more...

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

It is an 5.6 port merge regression. The rseg_list was changed to rseg_array and the canonical way to iterate through it in the rest of the server includes checking each member != NULL, which does not happen in i_s_xtradb_rseg_fill() in xtradb_i_s.cc.

tags: added: low-hanging-fruit merge-regression
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-751

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.