Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limit number of subprocesses for rethinkdb export (or use threads) #4809

Closed
danielmewes opened this issue Sep 8, 2015 · 3 comments
Closed

Limit number of subprocesses for rethinkdb export (or use threads) #4809

danielmewes opened this issue Sep 8, 2015 · 3 comments
Assignees
@encryptio
Labels
Python
Milestone
2.2.1

Comments

@danielmewes
Copy link
Member

This was brought up by @s3tqr2w in #2371 (comment) .
Apparently we currently fork one subprocess for each table in rethinkdb export, even if the number of clients is set to a lower number.
@danielmewes danielmewes added this to the 2.2-polish milestone Sep 8, 2015
@danielmewes danielmewes mentioned this issue Sep 8, 2015
Open
@srevenant
Copy link

Thanks guys. If you need any help I can assist. I think the simplest approach would be to have a -safe option or some such that tells it not to fork off for each table. More complex would be to manage and throttle the sub-processes somehow.

Note: for context, when backups run on our DB it consumes almost 2GB of memory. Our db is 346Mb in size.

@danielmewes danielmewes mentioned this issue Nov 12, 2015
Closed
@encryptio encryptio assigned encryptio and unassigned Tryneus Nov 14, 2015
@danielmewes danielmewes modified the milestones: 2.2.x, 2.2-polish Nov 16, 2015
@encryptio
Copy link
Contributor

In review with Marc Hesse, CR 3343.

As measured in my Ubuntu 15.04 64-bit VM, memory usage is around 15 + (17+27)*clients megabytes across 1 + 2*clients processes, regardless of number of tables with my change. Before, it was around 15 + 17*table_count + 27*clients megabytes of ram across 1 + table_count + clients processes.

In my specific test case, that meant several gigabytes of ram down to a hundred megs.

@danielmewes danielmewes modified the milestones: 2.2.1, 2.2.x Nov 17, 2015
encryptio added a commit that referenced this issue Nov 17, 2015
@encryptio
rethinkdb export now limits the number of processes to 1 + 2*clients
f479e9a 
Fixes #4809
@danielmewes
Copy link
Member Author

The fix shipped as part of the updated Python driver 2.2.0.post1 today.
We've also notified Compose about the update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@encryptio encryptio

Labels
Python
Projects
None yet
Milestone
2.2.1
Development

No branches or pull requests
4 participants
@encryptio @danielmewes @Tryneus @srevenant