Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error during table fuzzing, ReqlInternalError: Unexpected exception #4758

Closed
Tryneus opened this issue Aug 25, 2015 · 3 comments
Closed

Error during table fuzzing, ReqlInternalError: Unexpected exception #4758

Tryneus opened this issue Aug 25, 2015 · 3 comments
Assignees
Labels
Milestone

Comments

@Tryneus
Copy link
Member

Tryneus commented Aug 25, 2015

The full error is:

ReqlInternalError("Unexpected exception: The attempt to read or "
    "modify the table's configuration failed because none of the "
    "servers were accessible.  If it was an attempt to modify, the "
    "modification did not take place.")

The source of this is a failed_table_op_exc_t, but contrary to the error text, the cluster was fully available during this time, and no disconnections were seen in the logs. A ReqlInternalError should not be returned to users during normal operation, so either the error needs to be changed or we need more error handling somewhere. This exception can be thrown from 5 places:

clustering/administration/tables/generate_config.cc:290 - table_generate_config(...) when generating a config for a table that shows as disconnected through the table_meta_client_t.

clustering/administration/tables/split_points.cc:109 - fetch_distribution(...) when the distribution_read_t throws a cannot_perform_query_exc_t, but the table_meta_client_t does not indicate that the table has been deleted.

clustering/table_manager/table_meta_client.cc:561 - create_or_emergency_repair(...) when no business cards could be found for any of the servers involved with the new config for the table

clustering/table_manager/table_meta_client.cc:772 - retry(...) just rethrows failed_table_op_exc_t after all attempts failed, ignore this one.

clustering/table_manager/table_meta_client.cc:789 - throw_appropriate_exception(...), which is used in several places to throw a failed_table_op_exc_t if the table is known to exist, and a no_such_table_exc_t otherwise. Mostly after a get_status call fails, and once in set_config if no leader can be found for the table being changed.

@Tryneus Tryneus added this to the 2.1.x milestone Aug 25, 2015
@Tryneus
Copy link
Member Author

Tryneus commented Aug 25, 2015

It seems that there may just be a place missing a CATCH_OP_ERRORS, which would convert the failed_table_op_exc_t to a ReqlAvailabilityError.

Here is an example query that can result in the ReqlInternalError, currently running the fuzzer to see if it finds any more.

r.db(db).table(table).config().update( { 'shards': [...] } )

@VeXocide VeXocide self-assigned this Aug 25, 2015
@VeXocide
Copy link
Member

I've found two places where we forgot to CATCH_OP_ERRORS and one where we use it with empty error messages.

@VeXocide
Copy link
Member

Merged into branched next and v2.1.x via commits 7252119 and c986f65 respectively.

@danielmewes danielmewes modified the milestones: 2.1.x, 2.1.3 Sep 4, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants