Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash in msg_visitor_t #4878

Closed
danielmewes opened this issue Sep 23, 2015 · 7 comments
Closed

Crash in msg_visitor_t #4878

danielmewes opened this issue Sep 23, 2015 · 7 comments
Assignees
@mlucy
Labels
pr:high tp:bug
Milestone
2.1.5

Comments

@danielmewes
Copy link
Member

Here's the log from a Compose RethinkDB instance, reported by @weshoke

 pure virtual method called
 pure virtual method called
 pure virtual method called
 pure virtual method called
 Version: rethinkdb 2.1.3~0trusty (GCC 4.8.2)
 Version: rethinkdb 2.1.3~0trusty (GCC 4.8.2)
 pure virtual method called
 Version: rethinkdb 2.1.3~0trusty (GCC 4.8.2)
 Version: rethinkdb 2.1.3~0trusty (GCC 4.8.2)
 error: Error in src/errors.cc at line 131:
 error: Error in src/errors.cc at line 131:
 Version: rethinkdb 2.1.3~0trusty (GCC 4.8.2)
 error: std::terminate() called without any exception.
 error: Error in src/errors.cc at line 131:
 error: Backtrace:
 error: std::terminate() called without any exception.
 error: Error in src/errors.cc at line 131:
 error: Backtrace:
 error: std::terminate() called without any exception.
 error: Backtrace:
 error: Error in src/errors.cc at line 131:
 error: std::terminate() called without any exception.
 error: std::terminate() called without any exception.
 error: Backtrace:
 error: Backtrace:
 pure virtual method called
 Version: rethinkdb 2.1.3~0trusty (GCC 4.8.2)
 error: Error in src/errors.cc at line 131:
 error: std::terminate() called without any exception.
 error: Backtrace:
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 addr2line: 'rethinkdb': No such file
 error: Tue Sep 22 18:04:18 2015

        1: backtrace_t::backtrace_t() at 0xa009e0 (rethinkdb)
        2: format_backtrace(bool) at 0xa00d73 (rethinkdb)
        3: report_fatal_error(char const*, int, char const*, ...) at 0xc39d15 (rethinkdb)
        4: terminate_handler() at 0xc39f25 (rethinkdb)
        5: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e6d6) [0x7feb6921c6d6] at 0x7feb6921c6d6 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6)
        6: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e703) [0x7feb6921c703] at 0x7feb6921c703 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6)
        7: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5f1bf) [0x7feb6921d1bf] at 0x7feb6921d1bf (/usr/lib/x86_64-linux-gnu/libstdc++.so.6)
        8: ql::changefeed::msg_visitor_t::operator()(ql::changefeed::msg_t::change_t const&) const::{lambda(ql::changefeed::range_sub_t*)#1}::operator()(ql::changefeed::range_sub_t*) const at 0x853ddf (rethinkdb)
        9: void ql::changefeed::feed_t::each_sub_in_vec_cb<ql::changefeed::range_sub_t>(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int) at 0x845381 (rethinkdb)
        10: pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int)> (ql::changefeed::feed_t*, std::reference_wrapper<std::function<void (ql::changefeed::range_sub_t*)> const>, std::reference_wrapper<std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const>, std::reference_wrapper<std::vector<int, std::allocator<int> > const>, std::_Placeholder<1>)>, long>::operator()() at 0x8550b5 (rethinkdb)
        11: void pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int)> (ql::changefeed::feed_t*, std::reference_wrapper<std::function<void (ql::changefeed::range_sub_t*)> const>, std::reference_wrapper<std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const>, std::reference_wrapper<std::vector<int, std::allocator<int> > const>, std::_Placeholder<1>)> >(long, long, std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int)> (ql::changefeed::feed_t*, std::reference_wrapper<std::function<void (ql::changefeed::range_sub_t*)> const>, std::reference_wrapper<std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const>, std::reference_wrapper<std::vector<int, std::allocator<int> > const>, std::_Placeholder<1>)> const&) at 0x8551a3 (rethinkdb)
        12: void ql::changefeed::feed_t::each_sub_in_vec<ql::changefeed::range_sub_t>(std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, rwlock_in_line_t*, auto_drainer_t::lock_t const&, std::function<void (ql::changefeed::range_sub_t*)> const&) at 0x85536f (rethinkdb)
        13: ql::changefeed::feed_t::each_range_sub(auto_drainer_t::lock_t const&, std::function<void (ql::changefeed::range_sub_t*)> const&) at 0x83ad57 (rethinkdb)
        14: ql::changefeed::msg_visitor_t::operator()(ql::changefeed::msg_t::change_t const&) const at 0x855a1e (rethinkdb)
        15: ql::changefeed::real_feed_t::mailbox_cb(signal_t*, ql::changefeed::stamped_msg_t) at 0x843808 (rethinkdb)
        16: void std::_Mem_fn<void (ql::changefeed::real_feed_t::*)(signal_t*, ql::changefeed::stamped_msg_t)>::operator()<signal_t*, ql::changefeed::stamped_msg_t, void>(ql::changefeed::real_feed_t*, signal_t*&&, ql::changefeed::stamped_msg_t&&) const at 0x8570c9 (rethinkdb)
        17: std::_Function_handler<void (signal_t*, ql::changefeed::stamped_msg_t), std::_Bind<std::_Mem_fn<void (ql::changefeed::real_feed_t::*)(signal_t*, ql::changefeed::stamped_msg_t)> (ql::changefeed::real_feed_t*, std::_Placeholder<1>, std::_Placeholder<2>)> >::_M_invoke(std::_Any_data const&, signal_t*, ql::changefeed::stamped_msg_t) at 0x85734d (rethinkdb)
        18: mailbox_t<void (ql::changefeed::stamped_msg_t)>::read_impl_t::read(read_stream_t*, signal_t*) at 0x864782 (rethinkdb)
        19: rethinkdb() [0x944a02] at 0x944a02 ()
        20: coro_t::run() at 0x924728 (rethinkdb)
 error: Tue Sep 22 18:04:18 2015

        1: backtrace_t::backtrace_t() at 0xa009e0 (rethinkdb)
        2: format_backtrace(bool) at 0xa00d73 (rethinkdb)
        3: report_fatal_error(char const*, int, char const*, ...) at 0xc39d15 (rethinkdb)
        4: terminate_handler() at 0xc39f25 (rethinkdb)
        5: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e6d6) [0x7feb6921c6d6] at 0x7feb6921c6d6 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6)
        6: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e703) [0x7feb6921c703] at 0x7feb6921c703 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6)
        7: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5f1bf) [0x7feb6921d1bf] at 0x7feb6921d1bf (/usr/lib/x86_64-linux-gnu/libstdc++.so.6)
        8: ql::changefeed::msg_visitor_t::operator()(ql::changefeed::msg_t::change_t const&) const::{lambda(ql::changefeed::range_sub_t*)#1}::operator()(ql::changefeed::range_sub_t*) const at 0x853ddf (rethinkdb)
        9: void ql::changefeed::feed_t::each_sub_in_vec_cb<ql::changefeed::range_sub_t>(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int) at 0x845381 (rethinkdb)
        10: callable_action_instance_t<pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int)> (ql::changefeed::feed_t*, std::reference_wrapper<std::function<void (ql::changefeed::range_sub_t*)> const>, std::reference_wrapper<std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const>, std::reference_wrapper<std::vector<int, std::allocator<int> > const>, std::_Placeholder<1>)>, long> >::run_action() at 0x84681b (rethinkdb)
        11: coro_t::run() at 0x924728 (rethinkdb)
 error: Exiting.
 error: Exiting.
 /bin/bash: line 1:    18 Trace/breakpoint trap   (core dumped) rethinkdb --config-file /app/instance1.conf

@mlucy any ideas?
@danielmewes danielmewes added this to the 2.1.x milestone Sep 23, 2015
@danielmewes
Copy link
Member Author

An important piece of information: The server had lost the connection to another server in the cluster just before this happened.

@mlucy
Copy link
Member

mlucy commented Sep 23, 2015

@weshoke -- is this reproducible?

I think I see how this might happen, actually. We have an auto_drainer_t on a class subscription_t that has some pure virtual functions. So when we acquire a lock on that drainer, it stops the base class from being destructed, but not the subclasses. I think we need to have the drainer on each of the most-derived classes instead, and access it with a virtual function.

@mlucy
Copy link
Member

mlucy commented Sep 23, 2015

The expected fix is in CR 3244 by @danielmewes.

@danielmewes
Copy link
Member Author

@mlucy As far as I know this happened only once

@mlucy
Copy link
Member

mlucy commented Sep 24, 2015

This is in next and 2.1.x.

@mlucy mlucy closed this as completed Sep 24, 2015
@mlucy mlucy self-assigned this Sep 24, 2015
@fasiha
Copy link

fasiha commented Oct 1, 2015

I just built 2.1.4 from source for Debian Jessie on a Scaleway.com armhf processor. It segfaults as soon as I start it with the same addr2line message—but not at the msg_visitor_t function. I know ARM support is experimental, but could this be related to this issue?

$ rethinkdb
Running rethinkdb 2.1.4 (GCC 4.9.2)...
Running on Linux 4.1.6-249 armv7l
Loading data from directory /home/deploy/software/rethinkdb-2.1.4/rethinkdb_data
Listening for intracluster connections on port 29015
Listening for client driver connections on port 28015
Listening for administrative HTTP connections on port 8080
Listening on addresses: 127.0.0.1, ::1
To fully expose RethinkDB on the network, bind to all addresses by running rethinkdb with the `--bind all` command line option.
Server ready, "unlock01_6rt" 3fc9dac4-831b-4bee-8d52-47314967caee
Version: rethinkdb 2.1.4 (GCC 4.9.2)
error: Error in src/errors.cc at line 104:
error: Segmentation fault.
error: Backtrace:
addr2line: 'rethinkdb': No such file
addr2line: 'rethinkdb': No such file
error: Thu Oct  1 13:41:51 2015

       1: backtrace_t::backtrace_t() at 0x317f80 (rethinkdb)
       2: format_backtrace(bool) at 0x3181dc (rethinkdb)
       3: report_fatal_error(char const*, int, char const*, ...) at 0x65d38a (rethinkdb)
       4: generic_crash_handler(int) at 0x65d4c2 (rethinkdb)
       5: /lib/arm-linux-gnueabihf/libc.so.6(+0x26ac0) [0xb5309ac0] at 0xb5309ac0 (/lib/arm-linux-gnueabihf/libc.so.6)
       6: void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::RAllocator> >::ParseValue<16u,
rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::RAllocator>
, rapidjson::RAllocator> >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllo
cator<rapidjson::RAllocator>, rapidjson::RAllocator>&) at 0x367278 (rethinkdb)
       7: json_to_datum(std::string const&, ql::configured_limits_t const&, reql_version_t, attach_json_to_error_t, http_result_t*) at 0x622618 (rethinkdb)
       8: perform_http(http_opts_t*, http_result_t*) at 0x623032 (rethinkdb)
       9: http_job_t::worker_fn(read_stream_t*, write_stream_t*) at 0x623548 (rethinkdb)
       10: spawner_run_t::main_loop() at 0x61f77e (rethinkdb)
       11: extproc_spawner_t::fork_spawner() at 0x61f408 (rethinkdb)
       12: extproc_spawner_t::extproc_spawner_t() at 0x61f468 (rethinkdb)
       13: main_rethinkdb_porcelain(int, char**) at 0x593342 (rethinkdb)
       14: main+0x153 at 0x2c3da4 (rethinkdb)
       15: __libc_start_main+0x99 at 0xb52fa632 (/lib/arm-linux-gnueabihf/libc.so.6)
error: Exiting.
warn: Problem when checking for new versions of RethinkDB: HTTP request to update.rethinkdb.com failed.

@danielmewes
Copy link
Member Author

Hi @fasiha, that is an unrelated problem which is indeed ARM specific.
See #4839 (comment) for a work-around.
I also recommend building the v2.1.x branch rather than 2.1.4. Only v2.1.x has the bug fix for this issue.

@danielmewes danielmewes modified the milestones: 2.1.x, 2.1.5 Oct 1, 2015
@mlucy mlucy mentioned this issue Oct 23, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mlucy mlucy

Labels
pr:high tp:bug
Projects
None yet
Milestone
2.1.5
Development

No branches or pull requests
3 participants
@fasiha @danielmewes @mlucy