New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Server crash after upgrading from 2.0.4 to 2.2.0 #5085
Comments
Do you know if there was a particular query that triggered this? Does it happen repeatedly? |
@danielmewes Yes it occurs repeatly, But I don't know which query cause this |
@jilen -- Thanks for the bug report! Would you mind answering a few more questions to help us track this down?
|
|
Ah, sorry, I didn't see #5083 before. Is it possible that you have an index which hasn't been upgraded since before RethinkDB 1.16? Looking at the code, that's the only thing that I think could cause this. (Although that would be very strange, because we should be refusing to do reads on pre-1.16 indexes in this version. Right @danielmewes?) Sorry you've run into two problems with the new release! The good news is that hopefully they're related. |
Pre 1.16 indexes would fail very differently (on startup), and I think @jilen was running 2.0 before? Could this happen if some hash shards aren't ready when a read is performed? |
@danielmewes -- I don't think so, I think that should produce an error in the unshard visitor and we should never get to the guarantee. We should test it though. We could also get to that error if the clustering code is in a weird state and silently routes the read to a subset of the shards it needs to go to rather than producing an error because some of the shards it needs to go to are unavailable. |
@danielmewes I am upgrading from 2.0.4 |
Still exists after I migrate sencondary indexes |
I upgraded my test environment from 2.1.5-2 to 2.2.0 and can easily reproduce crashing the server by running Hope this helps. |
Hi @hueniverse ! Thanks, that should help. What platform are you on? |
I'm on Ubuntu 14.04.3. If you click on the travis badge you can see the issue as well. |
@hueniverse How could I downgrade the version to 2.1.5-2 on a production server ? |
Alright, a fix for this problem is up in CR 3338 by @danielmewes . We should put out 2.2.1 soon with the fix. Thanks for helping us track this down! |
@jilen -- you should be able to use the package manager on your system to uninstall RethinkDB 2.2 and install the old package. I've never used CentOS before, so I don't know the exact command. |
@jilen The problem with downgrading to 2.1.5 is that you won't be able to use the data files anymore, since RethinkDB 2.2 will have migrated them to a new format. |
We'll try to release an official update (2.2.1) with this on Monday or Tuesday. |
@jilen You can download the hot fix build for CentOS 6 from here: |
@danielmewes Thanks very much. |
Is there an Ubuntu hotfix? |
What is CR 3338? -- "a fix for this problem is up in CR 3338 by @danielmewes" I want to build from source, but don't know what source to build. |
The branch to build from is |
mglukhov in slack pointed me to the branch just now too, and I've launched a build from source. It's about half way done. If that works, I'll use it; otherwise, I'll really need an Ubuntu 15.04 package. I'll post back here either way in a few minutes. |
I successfully built RethinkDB from source on my Ubuntu 15.04 system. I of course also have the RethinkDB official 2.2 system-wide package installed. The result of the build from source seems to be a new rethinkdb binary. Would it be fine for me to just do only: |
Yes, that should work. Once the new package comes out, it will automatically overwrite the file. |
OK, everything seems to be working perfectly now. I'll report any other problems I see. Incidentally, the site and web servers worked fine with RethinkDB servers randomly failing every 30 minutes, e.g., the clients properly recovered in all cases. So this bug was also a good test of autofailover. |
This fix is in next and 2.2.x. |
@jilen @williamstein @hueniverse Packages for RethinkDB 2.2.1 are now available with this fix. |
Confirmed. Thanks. |
Thanks. |
The text was updated successfully, but these errors were encountered: