mysqld can't startup when a plugin was installed

Bug #1011047 reported by yinfeng
58
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
High
Sergei Glushchenko
5.1
Fix Released
High
Sergei Glushchenko
5.5
Fix Released
High
Sergei Glushchenko
5.6
Fix Released
High
Sergei Glushchenko

Bug Description

i had installed daemon_example, then shutdown the server. i tried to restart mysqld, but failed eveytime. bellow is the backtrace in error log:
stack_bottom = 0x7fffbbbc5490 thread_stack 0x40000
/u01/mysql/bin/mysqld(my_print_stacktrace+0x39)[0x7b1c99]
/u01/mysql/bin/mysqld(handle_segfault+0x43c)[0x4fa49c]
/lib64/libpthread.so.0[0x3e00e0f4a0]
/u01/mysql/bin/mysqld(my_hash_insert+0x331)[0x79a4f1]
/u01/mysql/bin/mysqld(_ZN7handler25update_global_table_statsEv+0x296)[0x6845b6]
/u01/mysql/bin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xa7)[0x53acf7]
/u01/mysql/bin/mysqld(_Z19close_thread_tablesP3THD+0x1ab)[0x53b1fb]
/u01/mysql/bin/mysqld(_Z18close_mysql_tablesP3THD+0x11)[0x53b281]
/u01/mysql/bin/mysqld[0x582c84]
/u01/mysql/bin/mysqld(_Z11plugin_initPiPPci+0x89b)[0x584e5b]
/u01/mysql/bin/mysqld[0x4ffb76]
/u01/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x563)[0x503333]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3e0061ecdd]
/u01/mysql/bin/mysqld[0x4f8b49]

To our surprise, in other machine everything is normal

Related branches

Revision history for this message
yinfeng (yinfeng-zwx) wrote :

version: percona server 5.5.18

Revision history for this message
yinfeng (yinfeng-zwx) wrote :

i just found i had set userstat in configure file. may be there's some bug related to userstat.

now i am able to repeat this bug every time:

in my.cnf

     userstat = 1

install plugin
     install plugin daemon_example soname 'libdaemon_example.so';

shutdown mysqld and restart it.

Revision history for this message
yinfeng (yinfeng-zwx) wrote :

a simple patch :
init_global_table_stats should be called before plugin_init

--- a/sql/mysqld.cc 2011-12-20 06:38:58.000000000 +0800
+++ b/sql/mysqld.cc 2012-06-10 12:31:49.000000000 +0800
@@ -3850,7 +3850,7 @@
 #ifdef HAVE_REPLICATION
   init_slave_list();
 #endif
-
+ init_global_table_stats();
   /* Setup logs */

   /*
@@ -4064,7 +4064,6 @@
   /* We have to initialize the storage engines before CSV logging */
   TC_init();

- init_global_table_stats();
   init_global_index_stats();

   if (ha_init())

Stewart Smith (stewart)
tags: added: userstats
summary: - mysqld can't startup when a plugin had being installed
+ mysqld can't startup when a plugin was installed
tags: added: contribution
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :
Download full text (3.5 KiB)

Verified the bug. The crash happens because global_table_stats is not initialized when my_hash_insert is invoked on it resulting in segfault.

Complete backtrace:

#0 0x00007ffff6185fa5 in raise () from /lib/libc.so.6
#1 0x00007ffff6187428 in abort () from /lib/libc.so.6
#2 0x00007ffff617f002 in __assert_fail_base () from /lib/libc.so.6
#3 0x00007ffff617f0b2 in __assert_fail () from /lib/libc.so.6
#4 0x00000000008c5f36 in my_realloc (oldpoint=0x0, size=0, my_flags=my_flags@entry=80) at /media/Vone/percona-server/raghu/Percona-Server/mysys/my_malloc.c:81
#5 0x000000000089ee95 in alloc_dynamic (array=array@entry=0x1b8c168 <global_table_stats+40>) at /media/Vone/percona-server/raghu/Percona-Server/mysys/array.c:150
#6 0x00000000008a485c in my_hash_insert (info=0x1b8c140 <global_table_stats>, record=0x1e22a90 "mysql.plugin") at /media/Vone/percona-server/raghu/Percona-Server/mysys/hash.c:382
#7 0x00000000007451cc in handler::update_global_table_stats (this=0x1e20490) at /media/Vone/percona-server/raghu/Percona-Server/sql/handler.cc:3871
#8 0x0000000000571623 in close_thread_table (thd=thd@entry=0x7fffffffb100, table_ptr=table_ptr@entry=0x7fffffffb1a0) at /media/Vone/percona-server/raghu/Percona-Server/sql/sql_base.cc:1598
#9 0x0000000000571de6 in close_open_tables (thd=0x7fffffffb100) at /media/Vone/percona-server/raghu/Percona-Server/sql/sql_base.cc:1362
#10 close_thread_tables (thd=0x7fffffffb100) at /media/Vone/percona-server/raghu/Percona-Server/sql/sql_base.cc:1569
#11 0x00000000005815dd in close_mysql_tables (thd=0x7fffffffb100) at /media/Vone/percona-server/raghu/Percona-Server/sql/sql_base.cc:9326
#12 0x00000000005dd16d in plugin_load (tmp_root=tmp_root@entry=0x7fffffffe310, argc=argc@entry=0x1a9915c <remaining_argc>, argv=argv@entry=0x1ddf700) at /media/Vone/percona-server/raghu/Percona-Server/sql/sql_plugin.cc:1526
#13 0x00000000005deee6 in plugin_init (argc=0x1a9915c <remaining_argc>, argv=0x1ddf700, flags=0) at /media/Vone/percona-server/raghu/Percona-Server/sql/sql_plugin.cc:1367
#14 0x0000000000529487 in init_server_components () at /media/Vone/percona-server/raghu/Percona-Server/sql/mysqld.cc:3952
#15 0x000000000052ab7a in mysqld_main (argc=22, argv=0x1ddf700) at /media/Vone/percona-server/raghu/Percona-Server/sql/mysqld.cc:4672
#16 0x000000000051d9dd in main (argc=<optimized out>, argv=<optimized out>) at /media/Vone/percona-server/raghu/Percona-Server/sql/main.cc:25

====================

The fix provided by yinfeng is good, however, need to move the other init upwards too:

=== modified file 'Percona-Server/sql/mysqld.cc'
--- Percona-Server/sql/mysqld.cc 2012-08-07 06:10:00 +0000
+++ Percona-Server/sql/mysqld.cc 2012-08-23 17:54:46 +0000
@@ -3947,6 +3947,9 @@
     sql_print_warning("ignore-builtin-innodb is deprecated "
                       "and will be removed in future releases.");

+ init_global_table_stats();
+ init_global_index_stats();
+
   if (plugin_init(&remaining_argc, remaining_argv,
                   (opt_noacl ? PLUGIN_INIT_SKIP_PLUGIN_TABLE : 0) |
                   (opt_help ? PLUGIN_INIT_SKIP_INITIALIZATION : 0)))
@@ -4007,8 +4010,6 @@
   /* We have to initialize the storage engines b...

Read more...

Revision history for this message
Geoffrey Anderson (geoff-geoffreyanderson) wrote :

I experienced the same issue when trying to enable auth_pam.so in Percona Server 5.5.28

tags: added: userstat
removed: userstats
Revision history for this message
Valerii Kravchuk (valerii-kravchuk) wrote :

Does not apply to PS 5.6.10 directly (yet), IMHO, as userstat is not there.

Revision history for this message
Valerii Kravchuk (valerii-kravchuk) wrote :

Confirmed on PS 5.1.68 also.

Revision history for this message
Valerii Kravchuk (valerii-kravchuk) wrote :

Sorry, current PS 5.6 from lp:

[openxs@centos percona-5.6]$ bzr version-info
revision-id: <email address hidden>
date: 2013-05-28 10:15:40 +0300
build-date: 2013-05-28 18:49:23 +0300
revno: 358
branch-nick: percona-5.6

is also affected, based on code review.

Revision history for this message
kernel (kernins) wrote :

Still affects 5.5.31 & 32 with SphinxSE plugin

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-133

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.