Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make streams not be terribly inefficient on sharded tables #4343

Closed
danielmewes opened this issue Jun 4, 2015 · 2 comments
Closed

Make streams not be terribly inefficient on sharded tables #4343

danielmewes opened this issue Jun 4, 2015 · 2 comments
Assignees
Milestone

Comments

@danielmewes
Copy link
Member

Consider the query r.table("t").update({a: "foo"}) on a sharded table "t".

Currently we request one batch of keys from each shard. Then we discard all but the one that is lowest in the key space. Then we send out updates to only that first shard.
We then repeat this until no keys are left to be processed in the first shard, and a similar thing happens for the remaining shards.

This has two problems:
a) It wastes a lot of resources on the shards because the same batch is requested and then discarded over and over.
b) It makes the query slower because each batch of updates only goes to a single shard, rather than being spread across multiple shards (this was noticed by @ha1331).

The first part of the problem I think applies to all kinds of sorted and unsorted streams in RethinkDB, not just to writes.

Assigning @mlucy

@danielmewes danielmewes added this to the 2.1-polish milestone Jun 4, 2015
@danielmewes danielmewes modified the milestones: 2.2, 2.1-polish Jul 27, 2015
@danielmewes danielmewes modified the milestones: 2.1.x, 2.2 Aug 31, 2015
@mlucy mlucy modified the milestones: 2.2, 2.1.x Sep 8, 2015
@mlucy
Copy link
Member

mlucy commented Sep 8, 2015

Bumping this to 2.2. It's about 70% done, and the logic is already sufficiently different that I don't think I'd be comfortable putting this into a point release. (A lot of the lazy datum stream logic ended up completely rewritten, and that's been a source of bugs in the past.)

@mlucy
Copy link
Member

mlucy commented Nov 10, 2015

This is in next.

@mlucy mlucy closed this as completed Nov 10, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants