New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data disappeared overnight #3717
Comments
@nuin: sorry you ran into this, we'll do our best to sort it out. Can you describe a few details about your server, namely:
@danielmewes may ask for other details that will help us understand what's going on. |
Hi @mglukhovsky, our server is small and it has 4Gb of memory, only a couple of tables are mainly active, with maybe 10-20 queries day. The output of ls won't help you at the moment. I had to update to the newer version in order to be able to load data from a backup as we still don't have a backup hardware system. But I remember listing the contents of the directory and seeing the same files as I had before the data disappearance, but all files had a smaller size than before. I updated to version rethinkdb 1.16.0+1~0wheezy, including the Python module. Data load was fine and quick, but there's an issue on the Dashboard, as all tables appear empty. Checking on the data explorer, data seems fine. |
Hi @nuin , your log file lists the last restart for 2015-02-03 09:22. Was that the one after which your data had disappeared? When the data wasn't there, did the tables still exist and were simply empty? Or were the tables gone as well? With the old data directory being lost, it's going to be pretty difficult to figure out how this happened exactly. We're going to think through any possibilities though. |
Hi @danielmewes The chronology of the events:
When I checked this morning, all databases and tables were there, but all of them reported 0 documents, I tried the data explorer on the web server and nothing was returned from past history command. No secondary indexes were available at the time I checked. On one of the tables, I had created a secondary index. Sorry, that I overwrote the old data directory. But as this is our only DB server, I couldn't just point the queries to another box. I am lobbying for new hardware, so we have some redundancy. I will keep daily dumps, and in my case I can recreate the data from scripts, the only problem is that it takes some hours from scratch data. I reported the error just in case someone runs into a similar situation, as it might help them and you to figure out. I use RethinkDB in my production apps, albeit all my programs run on a intranet, and I am quite happy with the performance and user-friendly features. |
Thanks a lot for your additional info. Beginning with the restart in step 4, the observed behavior could stem from something deleting the table files (they have UUIDs as their file names) but not the metadata. After restarting, the RethinkDB server would have recreated the table files with empty data. Since secondary indexes are stored in the table files and not the metadata, they would have disappeared together with the data. One thing that doesn't quite fit with that explanation though is that the application reported DB errors before restarting the server. If some process had deleted the files, the server would still have had the old files open and shouldn't have generated any query failures until restarted. Maybe someone else has another idea (pinging @timmaxw @Tryneus ). |
@danielmewes There were no real errors on the queries, they were just returned empty. The user was expecting a value X for an absolute value, and the program was showing 0. Sorry if I wasn't clear in my explanation. |
@nuin Ah thanks for your clarification. That's probably still an indication that the tables had suddenly become empty in a running server isn't it? |
@danielmewes As I mentioned, we don't have many queries every day, and the last successful query (before the disappearance) was yesterday afternoon. |
@nuin -- one more question, which file system are you using? |
@coffeemug RethinkDB is on a Debian: 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux. I was running 1.15.3 before the problem, just updated to 1.16. |
@nuin can you post the output of |
@danielmewes Here it is
I have a couple of other Windows disks mounted on /mnt using cifs, but they are read only and not used for writing in this box |
Thanks @nuin . It looks like the data files would have been on ext4. That should be fine. We've had some issues with BTRFS, but I don't think we've had any lately with ext4. |
@nuin We just released RethinkDB 1.16.1, which now writes a log entry every time it deletes a file (as per #1780). I recommend updating, so that we get some additional data for the case this happens to you again. |
Unfortunately I don't think there's anything we can do here at the moment, since we don't have the data and no additional hints as to why this could have happened. I'm going to close the issue, but we'll keep an eye on it and re-open as soon as we get any new data on this (@nuin also please re-open if anything remotely close to this happens again). |
Thanks, I will keep an eye on my side and updating things this coming week. |
All data on all my tables disappeared overnight, without any action or problem on the server host. I am running a Debian box, with rethinkdb 1.15.3~0wheezy (GCC 4.7.2), that was working fine since January 13th, with a quick shutdown on January 27th. No table was created or modified lately, only actions were data retrieval. Here is the log I have since I updated to version 1.15.3 on Jan 13th
Any help appreciated.
The text was updated successfully, but these errors were encountered: