Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault in table_fuzzer.py when killing/reviving servers #4972

Closed
Tryneus opened this issue Oct 20, 2015 · 10 comments
Closed

Segmentation fault in table_fuzzer.py when killing/reviving servers #4972

Tryneus opened this issue Oct 20, 2015 · 10 comments
Assignees
@mlucy
Labels
st:review tp:bug
Milestone
2.2

Comments

@Tryneus
Copy link
Member

Tryneus commented Oct 20, 2015

I was testing the new --kill option for table_fuzzer.py, and ran into this crash with the command line table_fuzzer.py --servers 7 --threads 48 --changefeeds --kill

Error in ../src/arch/runtime/thread_pool.cc at line 359:
Segmentation fault from reading the address (nil).
Backtrace:
Tue Oct 20 16:00:50 2015

1 [0x1d1fbf8]: rethinkdb_backtrace(void**, int) at rethinkdb_backtrace.cc:101
2 [0x19085c1]: backtrace_t::backtrace_t() at backtrace.cc:203
3 [0x1908c2f]: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at backtrace.cc:283
4 [0x190850f]: format_backtrace(bool) at backtrace.cc:198
5 [0x18fc3ce]: report_fatal_error(char const*, int, char const*, ...) at errors.cc:83
6 [0x172eac8]: linux_thread_pool_t::fatal_signal_handler(int, siginfo*, void*) at thread_pool.cc:359
7 [0x7fa66d524cb0]: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fa66d524cb0] at 0x7fa66d524cb0 (/lib/x86_64-linux-gnu/libpthread.so.0)
8 [0x18aeb9e]: /home/ssd2/grey/rethinkdb/build/debug/rethinkdb() [0x18aeb9e] at 0x18aeb9e ()
9 [0x18ae346]: mailbox_manager_t::on_local_message(connectivity_cluster_t::connection_t*, auto_drainer_t::lock_t, std::vector<char, std::allocator<char> >&&) at mailbox.cc:218
10 [0x183daa4]: connectivity_cluster_t::send_message(connectivity_cluster_t::connection_t*, auto_drainer_t::lock_t, unsigned char, cluster_send_message_write_callback_t*) at cluster.cc:1343
11 [0x18adda6]: send(mailbox_manager_t*, raw_mailbox_t::address_t, mailbox_write_callback_t*) at mailbox.cc:129
12 [0x1b11a4f]: void send<ql::changefeed::stamped_msg_t>(mailbox_manager_t*, mailbox_t<void (ql::changefeed::stamped_msg_t)>::address_t, ql::changefeed::stamped_msg_t const&) at typed.hpp:207
13 [0x1af6094]: ql::changefeed::server_t::send_all(ql::changefeed::msg_t const&, store_key_t const&, rwlock_in_line_t*, auto_drainer_t::lock_t const&) at changefeed.cc:536
14 [0x1a0469c]: rdb_modification_report_cb_t::on_mod_report(rdb_modification_report_t const&, bool, new_mutex_in_line_t*, rwlock_in_line_t*) at btree.cc:1335
15 [0x19fed42]: do_a_replace_from_batched_replace(auto_drainer_t::lock_t, fifo_enforcer_sink_t*, fifo_enforcer_write_token_t const&, btree_loc_info_t const&, one_replace_t, ql::configured_limits_t const&, promise_t<superblock_t*>*, rdb_modification_report_cb_t*, bool, ql::datum_t*, profile::trace_t*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*) at btree.cc:366
16 [0x1a18e39]: void std::_Bind<void (*(auto_drainer_t::lock_t, fifo_enforcer_sink_t*, fifo_enforcer_write_token_t, btree_loc_info_t, one_replace_t, ql::configured_limits_t, promise_t<superblock_t*>*, rdb_modification_report_cb_t*, bool, ql::datum_t*, profile::trace_t*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*))(auto_drainer_t::lock_t, fifo_enforcer_sink_t*, fifo_enforcer_write_token_t const&, btree_loc_info_t const&, one_replace_t, ql::configured_limits_t const&, promise_t<superblock_t*>*, rdb_modification_report_cb_t*, bool, ql::datum_t*, profile::trace_t*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*)>::__call<void, , 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11>(std::tuple<>&&, std::_Index_tuple<0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11>) at functional:1146
17 [0x1a15392]: void std::_Bind<void (*(auto_drainer_t::lock_t, fifo_enforcer_sink_t*, fifo_enforcer_write_token_t, btree_loc_info_t, one_replace_t, ql::configured_limits_t, promise_t<superblock_t*>*, rdb_modification_report_cb_t*, bool, ql::datum_t*, profile::trace_t*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*))(auto_drainer_t::lock_t, fifo_enforcer_sink_t*, fifo_enforcer_write_token_t const&, btree_loc_info_t const&, one_replace_t, ql::configured_limits_t const&, promise_t<superblock_t*>*, rdb_modification_report_cb_t*, bool, ql::datum_t*, profile::trace_t*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*)>::operator()<, void>() at functional:1206
18 [0x1a11a98]: std::_Function_handler<void (), std::_Bind<void (*(auto_drainer_t::lock_t, fifo_enforcer_sink_t*, fifo_enforcer_write_token_t, btree_loc_info_t, one_replace_t, ql::configured_limits_t, promise_t<superblock_t*>*, rdb_modification_report_cb_t*, bool, ql::datum_t*, profile::trace_t*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*))(auto_drainer_t::lock_t, fifo_enforcer_sink_t*, fifo_enforcer_write_token_t const&, btree_loc_info_t const&, one_replace_t, ql::configured_limits_t const&, promise_t<superblock_t*>*, rdb_modification_report_cb_t*, bool, ql::datum_t*, profile::trace_t*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*)> >::_M_invoke(std::_Any_data const&) at functional:1780
19 [0x1718380]: std::function<void ()>::operator()() const at functional:2162
20 [0x19fee74]: /home/ssd2/grey/rethinkdb/build/debug/rethinkdb() [0x19fee74] at 0x19fee74 ()
21 [0x1a14e7b]: coro_pool_t<std::function<void ()> >::worker_run(std::function<void ()>, auto_drainer_t::lock_t) at coro_pool.hpp:68
22 [0x1a2242a]: std::_Mem_fn<void (coro_pool_t<std::function<void ()> >::*)(std::function<void ()>, auto_drainer_t::lock_t)>::operator()(coro_pool_t<std::function<void ()> >*, std::function<void ()>, auto_drainer_t::lock_t) const at functional:551
23 [0x1a21e7d]: void std::_Bind<std::_Mem_fn<void (coro_pool_t<std::function<void ()> >::*)(std::function<void ()>, auto_drainer_t::lock_t)> (coro_pool_t<std::function<void ()> >*, std::function<void ()>, auto_drainer_t::lock_t)>::__call<void, , 0, 1, 2>(std::tuple<>&&, std::_Index_tuple<0, 1, 2>) at functional:1146
24 [0x1a21ce8]: void std::_Bind<std::_Mem_fn<void (coro_pool_t<std::function<void ()> >::*)(std::function<void ()>, auto_drainer_t::lock_t)> (coro_pool_t<std::function<void ()> >*, std::function<void ()>, auto_drainer_t::lock_t)>::operator()<, void>() at functional:1206
25 [0x1a21bde]: callable_action_instance_t<std::_Bind<std::_Mem_fn<void (coro_pool_t<std::function<void ()> >::*)(std::function<void ()>, auto_drainer_t::lock_t)> (coro_pool_t<std::function<void ()> >*, std::function<void ()>, auto_drainer_t::lock_t)> >::run_action() at callable_action.hpp:31
26 [0x1729928]: callable_action_wrapper_t::run() at runtime_utils.cc:43
27 [0x1731b2f]: coro_t::run() at coroutines.cc:207
28 [0x1a15161]: coro_t* coro_t::spawn_sometime<std::_Bind<std::_Mem_fn<void (coro_pool_t<std::function<void ()> >::*)(std::function<void ()>, auto_drainer_t::lock_t)> (coro_pool_t<std::function<void ()> >*, std::function<void ()>, auto_drainer_t::lock_t)> >(std::_Bind<std::_Mem_fn<void (coro_pool_t<std::function<void ()> >::*)(std::function<void ()>, auto_drainer_t::lock_t)> (coro_pool_t<std::function<void ()> >*, std::function<void ()>, auto_drainer_t::lock_t)>&&) at coroutines.hpp:59
29 [0x1a116c9]: coro_pool_t<std::function<void ()> >::on_source_availability_changed() at coro_pool.hpp:86
30 [0x1739ef5]: availability_control_t::set_available(bool) at passive_producer.hpp:66
31 [0x1a0dd64]: unlimited_fifo_queue_t<std::function<void ()>, std::list<std::function<void ()>, std::allocator<std::function<void ()> > > >::push(std::function<void ()> const&) at unlimited_fifo.hpp:49
32 [0x19ff283]: rdb_batched_replace(btree_info_t const&, scoped_ptr_t<real_superblock_t>*, std::vector<store_key_t, std::allocator<store_key_t> > const&, btree_batched_replacer_t const*, rdb_modification_report_cb_t*, ql::configured_limits_t, profile::sampler_t*, profile::trace_t*) at btree.cc:427
33 [0x1c5aae8]: rdb_write_visitor_t::operator()(batched_insert_t const&) at store.cc:691
@Tryneus Tryneus added this to the 2.2 milestone Oct 20, 2015
@Tryneus
Copy link
Member Author

Tryneus commented Oct 20, 2015

Just got another crash, probably the same problem but a slightly different code path:

error: Error in ../src/arch/runtime/thread_pool.cc at line 359:
error: Segmentation fault from reading the address (nil).
error: Backtrace:
error: Tue Oct 20 16:32:42 2015

1 [0x1d1fbf8]: rethinkdb_backtrace(void**, int) at rethinkdb_backtrace.cc:101
2 [0x19085c1]: backtrace_t::backtrace_t() at backtrace.cc:203
3 [0x1908c2f]: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at backtrace.cc:283
4 [0x190850f]: format_backtrace(bool) at backtrace.cc:198
5 [0x18fc3ce]: report_fatal_error(char const*, int, char const*, ...) at errors.cc:83
6 [0x172eac8]: linux_thread_pool_t::fatal_signal_handler(int, siginfo*, void*) at thread_pool.cc:359
7 [0x7f2b3fd7acb0]: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f2b3fd7acb0] at 0x7f2b3fd7acb0 (/lib/x86_64-linux-gnu/libpthread.so.0)
8 [0x18aebd0]: /home/ssd2/grey/rethinkdb/build/debug/rethinkdb() [0x18aebd0] at 0x18aebd0 ()
9 [0x18ae5a4]: mailbox_manager_t::on_message(connectivity_cluster_t::connection_t*, auto_drainer_t::lock_t, read_stream_t*) at mailbox.cc:245
10 [0x183cb14]: connectivity_cluster_t::run_t::handle(keepalive_tcp_conn_stream_t*, boost::optional<peer_id_t>, boost::optional<peer_address_t>, auto_drainer_t::lock_t, bool*) at cluster.cc:1192
11 [0x183a850]: connectivity_cluster_t::run_t::connect_to_peer(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*) at cluster.cc:333
12 [0x1852f88]: std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)>::operator()(connectivity_cluster_t::run_t*, peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*) const at functional:551
13 [0x1850eab]: void std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>::__call_c<void, long&, 0, 1, 2, 3, 4, 5, 6>(std::tuple<long&>&&, std::_Index_tuple<0, 1, 2, 3, 4, 5, 6>) const at functional:1155
14 [0x184e267]: void std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>::operator()<long&, void>(long&) const at functional:1220
15 [0x184a611]: pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>, long>::operator()() at pmap.hpp:21
16 [0x1855c10]: callable_action_instance_t<pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>, long> >::run_action() at callable_action.hpp:31
17 [0x1729928]: callable_action_wrapper_t::run() at runtime_utils.cc:43
18 [0x1731b2f]: coro_t::run() at coroutines.cc:207
19 [0x184a5db]: void coro_t::spawn_now_dangerously<pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>, long> >(pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>, long>&&) at coroutines.hpp:53
20 [0x1846e03]: void pmap<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)> >(long, long, std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)> const&) at pmap.hpp:43
21 [0x1842da5]: void pmap<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)> >(long, std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)> const&) at pmap.hpp:54
22 [0x183ad0b]: connectivity_cluster_t::run_t::join_blocking(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t) at cluster.cc:376
23 [0x1856969]: std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)>::operator()(connectivity_cluster_t::run_t*, peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t) const at functional:551
24 [0x18563aa]: void std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)> (connectivity_cluster_t::run_t*, peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)>::__call<void, , 0, 1, 2, 3>(std::tuple<>&&, std::_Index_tuple<0, 1, 2, 3>) at functional:1146
25 [0x185604c]: void std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)> (connectivity_cluster_t::run_t*, peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)>::operator()<, void>() at functional:1206
26 [0x1855c2e]: callable_action_instance_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)> (connectivity_cluster_t::run_t*, peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)> >::run_action() at callable_action.hpp:31
27 [0x1729928]: callable_action_wrapper_t::run() at runtime_utils.cc:43
28 [0x1731b2f]: coro_t::run() at coroutines.cc:207

@Tryneus
Copy link
Member Author

Tryneus commented Oct 20, 2015

gdb analysis of the core file shows a completely different backtrace:

#3  0x0000000001b0b8c0 in operator() (this=0x7f2b0c7f1700, sub=0x7f2b28187c00) at ../src/rdb_protocol/changefeed.cc:2308
#4  0x0000000001b26f20 in std::_Function_handler<void(ql::changefeed::range_sub_t*), ql::changefeed::msg_visitor_t::operator()(const ql::changefeed::msg_t::change_t&) const::<lambda(ql::changefeed::range_sub_t*)> >::_M_invoke(const std::_Any_data &, ql::changefeed::range_sub_t *) (__functor=..., __args#0=0x7f2b28187c00) at /usr/include/c++/4.6/functional:1778
#5  0x0000000001b3a071 in std::function<void(ql::changefeed::range_sub_t*)>::operator()(ql::changefeed::range_sub_t *) const (this=0x7f2b0ab04e90, __args#0=0x7f2b28187c00)
    at /usr/include/c++/4.6/functional:2161
#6  0x0000000001b28ca8 in ql::changefeed::feed_t::each_sub_in_vec_cb<ql::changefeed::range_sub_t>(const std::function<void(ql::changefeed::range_sub_t*)> &, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, const std::vector<int, std::allocator<int> > &, int) (this=0x7f2b0d7ff200, f=...,
    vec=std::vector of length 25, capacity 25 = {...}, subscription_threads=std::vector of length 1, capacity 1 = {...}, i=0) at ../src/rdb_protocol/changefeed.cc:2933
#7  0x0000000001b5ec3b in std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>::operator()(ql::changefeed::feed_t *, const std::function<void(ql::changefeed::range_sub_t*)> &, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, const std::vector<int, std::allocator<int> > &, int) const (this=0x7f2b0ab04800,
    __object=0x7f2b0d7ff200, __args#0=..., __args#1=std::vector of length 25, capacity 25 = {...}, __args#2=std::vector of length 1, capacity 1 = {...}, __args#3=0)
    at /usr/include/c++/4.6/functional:551
#8  0x0000000001b58a07 in std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>::__call_c<void, long&, 0, 1, 2, 3, 4>(std::tuple<long&> &, std::_Index_tuple<0, 1, 2, 3, 4>) const (this=0x7f2b0ab04800,
    __args=empty std::tuple) at /usr/include/c++/4.6/functional:1155
#9  0x0000000001b512c5 in std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>::operator()<long int&, void>(void) const (this=0x7f2b0ab04800) at /usr/include/c++/4.6/functional:1219
#10 0x0000000001b46f09 in pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>, long int>::operator()(void) (this=0x7f2b0ab04750) at ../src/concurrency/pmap.hpp:20
#11 0x0000000001b3a333 in pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*> >&, const std::vector<int>&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*> > >, std::reference_wrapper<const std::vector<int> >, std::_Placeholder<1>)> >(int64_t, int64_t, const std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)> &) (begin=0,
    end=1, c=...) at ../src/concurrency/pmap.hpp:47
#12 0x0000000001b28e00 in pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*> >&, const std::vector<int>&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*> > >, std::reference_wrapper<const std::vector<int> >, std::_Placeholder<1>)> >(int64_t, const std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)> &) (count=1, c=...)
    at ../src/concurrency/pmap.hpp:53
#13 0x0000000001b1a01a in ql::changefeed::feed_t::each_sub_in_vec<ql::changefeed::range_sub_t>(const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, rwlock_in_line_t *, const auto_drainer_t::lock_t &, const std::function<void(ql::changefeed::range_sub_t*)> &) (this=0x7f2b0d7ff200,
    vec=std::vector of length 25, capacity 25 = {...}, spot=0x7f2b0ab048f0, lock=..., f=...) at ../src/rdb_protocol/changefeed.cc:2916
#14 0x0000000001afd95c in ql::changefeed::feed_t::each_range_sub(const auto_drainer_t::lock_t &, const std::function<void(ql::changefeed::range_sub_t*)> &) (this=0x7f2b0d7ff200, lock=...,
    f=...) at ../src/rdb_protocol/changefeed.cc:2942

This looks a little more useful (although it's hard to say if it's more correct than the other backtraces). Inspecting sub shows an almost certainly corrupted (or already freed) object. The binary, core file, and server folders are available on newton in ~grey/issue_4972.

@mlucy
Copy link
Member

mlucy commented Oct 20, 2015

Are these crashes on next or 2.1.x?

@Tryneus
Copy link
Member Author

Tryneus commented Oct 21, 2015

These were on next (153a96a). The test framework was recently reworked a bit, and it appears the version in v2.1.x deadlocks with this version of the table fuzzer. Still, I ran the test against v2.1.x and could not reproduce the crash. I'll keep trying, as I've run into other problems.

@Tryneus
Copy link
Member Author

Tryneus commented Oct 21, 2015

I got one of these on the v2.1.x build last night (a007331), core file and server files available on newton in ~grey/issue_4972_2/.

Backtrace from gdb:

#0  0x0000000001604ba5 in linux_thread_pool_t::fatal_signal_handler (signum=11, info=0x7fb00903e570, data=0x7fb00903e440)
        at ../src/arch/runtime/thread_pool.cc:359
#1  <signal handler called>
#2  0x00000000019017b5 in operator() (this=0x7fafe9d63860, sub=0x7fb009217800)
        at ../src/rdb_protocol/changefeed.cc:2306
#3  0x000000000191deae in std::_Function_handler<void(ql::changefeed::range_sub_t*), ql::changefeed::msg_visitor_t::operator()(const ql::changefeed::msg_t::change_t&) const::<lambda(ql::changefeed::range_sub_t*)> >::_M_invoke(const std::_Any_data &, ql::changefeed::range_sub_t *) (__functor=..., __args#0=0x7fb009217800)
        at /usr/include/c++/4.6/functional:1778
#4  0x0000000001931cfb in std::function<void(ql::changefeed::range_sub_t*)>::operator()(ql::changefeed::range_sub_t *) const (this=0x7faff6d84e90, __args#0=0x7fb009217800)
        at /usr/include/c++/4.6/functional:2161
#5  0x000000000191fc36 in ql::changefeed::feed_t::each_sub_in_vec_cb<ql::changefeed::range_sub_t>(const std::function<void(ql::changefeed::range_sub_t*)> &, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, const std::vector<int, std::allocator<int> > &, int) (this=0x7fafe2bfcf00, f=..., vec=std::vector of length 25, capacity 25 = {...}, subscription_threads=std::vector of length 1, capacity 1 = {...}, i=0)
        at ../src/rdb_protocol/changefeed.cc:2932
#6  0x00000000019555bb in std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>::operator()(ql::changefeed::feed_t *, const std::function<void(ql::changefeed::range_sub_t*)> &, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, const std::vector<int, std::allocator<int> > &, int) const (this=0x7faff6d84800, __object=0x7fafe2bfcf00, __args#0=..., __args#1=std::vector of length 25, capacity 25 = {...}, __args#2=std::vector of length 1, capacity 1 = {...}, __args#3=0)
        at /usr/include/c++/4.6/functional:551
#7  0x000000000194fb4b in std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>::__call_c<void, long&, 0, 1, 2, 3, 4>(std::tuple<long&> &, std::_Index_tuple<0, 1, 2, 3, 4>) const (this=0x7faff6d84800, __args=empty std::tuple)
        at /usr/include/c++/4.6/functional:1155
#8  0x0000000001948a71 in std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>::operator()<long int&, void>(void) const (this=0x7faff6d84800)
        at /usr/include/c++/4.6/functional:1219
#9  0x000000000193f2e3 in pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>, long int>::operator()(void) (this=0x7faff6d84750)
        at ../src/concurrency/pmap.hpp:20
#10 0x0000000001931fbd in pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*> >&, const std::vector<int>&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*> > >, std::reference_wrapper<const std::vector<int> >, std::_Placeholder<1>)> >(int64_t, int64_t, const std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)> &) (begin=0, end=1, c=...)
        at ../src/concurrency/pmap.hpp:47
#11 0x000000000191fd8e in pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*> >&, const std::vector<int>&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*> > >, std::reference_wrapper<const std::vector<int> >, std::_Placeholder<1>)> >(int64_t, const std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)> &) (count=1, c=...)
        at ../src/concurrency/pmap.hpp:53
#12 0x00000000019106f0 in ql::changefeed::feed_t::each_sub_in_vec<ql::changefeed::range_sub_t>(const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, rwlock_in_line_t *, const auto_drainer_t::lock_t &, const std::function<void(ql::changefeed::range_sub_t*)> &) (this=0x7fafe2bfcf00, vec=std::vector of length 25, capacity 25 = {...}, spot=0x7faff6d848f0, lock=..., f=...)
        at ../src/rdb_protocol/changefeed.cc:2915
#13 0x00000000018f386e in ql::changefeed::feed_t::each_range_sub(const auto_drainer_t::lock_t &, const std::function<void(ql::changefeed::range_sub_t*)> &) (this=0x7fafe2bfcf00, lock=..., f=...)
        at ../src/rdb_protocol/changefeed.cc:2941
#14 0x0000000001902d5d in ql::changefeed::msg_visitor_t::operator() (this=0x7faff6d85380, change=...)
        at ../src/rdb_protocol/changefeed.cc:2392
#15 0x0000000001958f9b in boost::detail::variant::invoke_visitor<ql::changefeed::msg_visitor_t>::internal_visit<ql::changefeed::msg_t::change_t const> (this=0x7faff6d85120, operand=...)
        at /usr/local/include/boost/variant/variant.hpp:858
...

This is on the same sub->active() call that I got from the last backtrace on next, and inspection of the sub object once again shows corruption of some kind. The segfault is probably due to following the corrupted vtable pointer.

@Tryneus Tryneus modified the milestones: 2.1.x, 2.2 Oct 21, 2015
@danielmewes
Copy link
Member

Assigning @mlucy .
Since as far as we know nobody has run into this in production and it might require rapid restarting of servers, I'm going to move the issue to the 2.2 milestone though.
We should still probably backport the fix to v2.1.x if it's not too hard.

@danielmewes danielmewes modified the milestones: 2.2, 2.1.x Oct 21, 2015
@mlucy
Copy link
Member

mlucy commented Oct 23, 2015

This looks like #4878 to me, so I guess that probably wasn't fixed by the changes in the patch.

@mlucy
Copy link
Member

mlucy commented Oct 23, 2015

I think I found it (and the fuzzer doesn't seem to be producing this crash for me any more). This is in CR 3296 by @danielmewes .

@mlucy
Copy link
Member

mlucy commented Oct 26, 2015

A fix for this is in next and v2.1.x.

@mlucy mlucy closed this as completed Oct 26, 2015
@mlucy
Copy link
Member

mlucy commented Oct 26, 2015

(Or what I believe is a fix. @Tryneus, let me know if you encounter this again.)

@danielmewes danielmewes mentioned this issue Nov 6, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mlucy mlucy

Labels
st:review tp:bug
Projects
None yet
Milestone
2.2
Development

No branches or pull requests
3 participants
@danielmewes @Tryneus @mlucy