Here's a reconstructed timeline of what happened:

1. Upgraded to 2.1.3 from 2.0.4 on Friday 9pm
2. Followed the rebuild secondary index instructions http://www.rethinkdb.com/docs/troubleshooting/#my-secondary-index-is-outdated Sunday morning
3. Noticed the Unable to retrieve the start stamps. Did you just reshard? errors soon after
4. The errors persisted until the next day.
5. Will soon reconfigure the table and see if that solves the error

Note to self: an easy patch for this would probably be to just remove the entry in the changefeed_client_t for a particular table whenever we get this response for that table, since we know it's in an invalid state.

Since this has happened again to another user, putting in that work-around sounds like a good idea to me if we can't find out why this is actually happening.

I think we know roughly why it's happening. It would still be good to try and reproduce it though.

(I'm the other user mentioned above.) For me this error message appears every time I do any resharding; fortunately I've always been able to resolve it by restarting every database server process (restarting clients has no effect). This bug means that any resharding of the tables in my rethinkdb cluster involves downtime, which is annoying for a rapidly growing operational website. Thanks for looking into this!

I'm having some trouble reproducing this, which is weird since @williamstein said it happens every time he does resharding. I'm trying to reproduce it by making a two-node cluster, opening a few hundred changefeed subscriptions per second, and then either resharding or kill -9ing one of the servers.

A few possibilities off the top of my head:

• It's timing-related, and I need either heavier load or more network latency to catch it.
• It only happens with a particular type of changefeed, or when changefeeds are used in conjunction with some other feature.
• It only shows up for some platforms/configuraitons. (@williamstein, you're running on Linux, right?)

Tracking down the root source of this will probably take a while, so in the meantime I think we should just do the quick patch so that people experiencing the issue can still reshard or restart a single node under load.

mlucy -- send me an email at wstein@sagemath.com and I could set things up so you can replicate this on a clone of the VM I was using. I was using Ubuntu 15.04, three rethinkdb processes (all on the same machine), etc.

I've been looking into this a little bit more, and I think I found the problem (by code inspection). Currently putting together a fix for the problem + the patch we discussed to automatically reset the feed if this error occurs again.

@williamstein -- a testing environment would be great, thanks! I'll send you an email.

A fix to what I think is the problem + the retrying patch mentioned above is in CR 3256 by @danielmewes .

The changes that I think fix this are in next and 2.1.x.

@williamstein tested a pre-release binary with the fix and was able to confirm that the issue no longer occurred. Closing.