Large Memory Consumption on OSX - Yosemite 10.10.2 #3712
Comments
|
Georges-MacBook-Pro:parts marchon$ system_profiler SPMemoryDataType |
|
SATA/SATA Express: |
|
Georges-MacBook-Pro:Downloads marchon$ brew update && brew install rethinkdb and run This formula is keg-only, which means it was not symlinked into /usr/local. Mac OS X already provides this software and installing another version in Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries Generally there are no consequences of this for you. If you build your ==> Summary To force the link and overwrite all conflicting files: To list all files that would be deleted: Possible conflicting files are: |
|
/usr/local/bin/rethinkdb |
|
|
|
|
|
Hi @marchon. Sorry you ran into this, and thank you for your detailed report. Could you provide us some more information to help diagnose this problem?
Thanks! |
|
Here are the two different log_files - from two different runs. {
}
}
}
} On Mon, Feb 2, 2015 at 7:48 PM, Timothy Maxwell notifications@github.com
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
From the first execution {
}
}
}
} On Mon, Feb 2, 2015 at 8:21 PM, George Lambert marchon@gmail.com wrote:
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
Thanks for the stats dumps, @marchon. We'll try to reproduce the memory leak locally. |
|
Would you like the files to go with it? I can give you the datafile and the python script if that is useful. G. On Mon, Feb 2, 2015 at 9:27 PM, Timothy Maxwell notifications@github.com
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
@marchon Thanks a lot for your help in tracking this down. Your data files and the Python script would be very helpful. |
|
they have now been sent to you. On Tue, Feb 3, 2015 at 12:12 AM, Daniel Mewes notifications@github.com
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
@marchon, thanks for your help! We've transferred your data to our internal servers, so we can try to reproduce this. (cc @danielmewes) |
|
@marchon I think the memory issue might have to do with the secondary index construction that was going on at the same time as the inserts. If you want to give it another shot, I recommend one of these work-arounds:
We're working on rewriting how secondary indexes are constructed (see #2134) because of issues like this, but the new implementation will likely take a few months before it's ready and sufficiently tested for shipping. Another problem with the current index construction implementation is that it can sometimes slow other queries down significantly (e.g. #1955). This is also going to be solved with #2134, but still exists as of now. |
|
I offered this help in the interest of advancing your project which looks These changes are not going to have a direct positive result for me, as I Thank you for keeping me up to date and good luck with your project. G. On Tue, Feb 3, 2015 at 5:37 PM, Daniel Mewes notifications@github.com
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
@marchon I understand, thank you for your help again and sorry it didn't work out. I did some more research, and this is rather interesting. It seems that this problem has to do with how RethinkDB is compiled / which libraries we are using.
@AtnNn which configure / make flags are we using for our dmg built? configure output: |
|
Glad that I could help. That is what makes this type of collaboration helpful. G. On Wed, Feb 4, 2015 at 2:22 PM, Daniel Mewes notifications@github.com
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
Seems like is has to do with boost. This causes high memory consumption: This one works as expected (low memory consumption): I'll do some memory profiling next to find out what's using the memory exactly. |
|
Boost 1.55 appears to be good, but both Boost 1.56.0 and 1.57.0 exhibit the problem. |
|
I can also reproduce this on Linux with Boost 1.56.0 |
|
Valgrind: The number of lost bytes seems to grow the longer I'm running the inserts (this was after just ~4500 inserts). |
|
If you are doing a dimple memory allocation or object creation in boost - If they do behave the same - then the problem will be in your code. Hope that helps. G. On Thu, Feb 5, 2015 at 1:30 AM, Daniel Mewes notifications@github.com
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
@marchon Yeah I doubt it's that easy. We use boost::variant all over the place, but only this one is causing an issue. I did a git bisect between v1.15.x and v1.16.x and it seems that this commit introduced the memory leak that Valgrind reports: The parent 0f2b17a was still good. Also pinging @mlucy. |
|
debugging memory problems in C++ between your code base and boost is well I can do it in c - but without setting up a debugging environment and You might get me to switch back yet - if you have time for a quick google G. On Thu, Feb 5, 2015 at 2:54 AM, Daniel Mewes notifications@github.com
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
The Valgrind-reported leak was probably a red herring. We traced it down to what looks like a Clang bug in at least Clang 3.2. We don't know yet whether this specific bug even affects the OS X build, since we're using a more recent version there. In any case after working around the leak that Valgrind reported the memory consumption is still way higher with Boost 1.56.0 than with 1.55.0, both on Linux and on OS X. Valgrind doesn't report any leaks, so we're mostly back to square one. |
|
Found the problem. As a consequence of the move, we were never evicting pages from the cache, thereby filling up the memory. Going to put a fix up soon, so we can roll out a point release next week. |
|
The fix is implemented in branch daniel_3712_2 and currently in code review 2582. |
|
congratulations guys - quick fix G On Fri, Feb 6, 2015 at 11:33 PM, Daniel Mewes notifications@github.com
P THINK BEFORE PRINTING: is it really necessary? This e-mail and its attachments are confidential and solely for the |
|
Thanks @marchon , your original testing and script helped a lot. |
|
Still have to double-check the Valgrind reported leak with newer Clang. |
|
Just tested with CLANG 3.5 and the Clang-specific leak is gone. |
|
@marchon we just shipped RethinkDB 1.16.1 with the fix for this problem. You can download it through the usual channels http://rethinkdb.com/docs/install/osx/ |
https://cloudup.com/iDdsKmsLXWB
Attached is a screenshot of rethinkdb consuming all of my available ram and slowing my machine down to the point that I could barely type in a terminal window.
My database was a mere 6,000,000 records with 4 fields each. It was a small sample of that I was hoping to accomplish with rethinkdb.
.
I will admit is seems like a great idea, but there are probably some issues with memory usage that need to be addressed.
The text was updated successfully, but these errors were encountered: