/cc @danielmewes

Daniel, what do you think about this?

A quick clarification ahead: RethinkDB uses one raft cluster per table. That means that for each table, a different server can be the Raft leader.

How about exposing a raft_leader field in table_status, which gets set to the server that's currently the Raft leader (or null if there is no known leader)?
I quite like that idea.

Also pinging @VeXocide for any thoughts on this.

@danielmewes ah right, I did fail to notice that this is per-table! Other than that, raft_leader sounds like a great idea +1

In the meantime, I've ended up with this reql to watch for leadership changes on a given table (r.db('test').table('changing_thing') in this case). The query currently takes ~350ms on my localhost, can you think of a way to speed it up? Unfortunately it doesn't seem that the logs provide any relevant index for me to key on, and the table id is a string I need to match for.

r.db('test').table('changing_thing').config()
  .do(function(config) {
    return r.db('rethinkdb').table('logs')
      .filter(function(log) {
        return log('message').match(config('id')).and(log('message').match('Raft leader'));
      });
  })
  .changes();

P.S. what penalty do I incur by attaching a change feed to such a slow query?

strangely enough if I change the match line from:

return log('message').match(config('id')).and(log('message').match('Raft leader'));

to

return log('message').match(config('id').add(': This server is Raft leader'));

I get a slightly slower query (but this is all based on Data Explorer, so I imagine the numbers are close enough). The query profile for the results are 44ms server time, and 54ms server time respectively

I think reading the full log table might be what's slow here. It's not parsed in the most efficient way at the moment. You can make it a bit faster by stopping once you've found the first match:

r.db('rethinkdb').table('logs')
      .orderBy(r.desc({index: "id"}))
      .filter(function(log) {
        return log('message').match(config('id')).and(log('message').match('Raft leader'));
      }).limit(1)

I don't think we actually support changefeeds on such queries yet, since it involves fetching data from multiple tables.

You can however get a changefeed on the logs table, and that should be efficient. It's probably easiest if you first retrieve the table ID manually in your client, and then do something like this:

r.db('rethinkdb').table('logs')
  .filter(function(log) {
      return log('message').match(tableId).and(log('message').match('Raft leader'));
    })
  .changes()

@danielmewes thanks for the quick response. It looks like the plain filter on logs takes about 43ms server time (with a cached id), but adding the orderBy and limit reduced that by half. You had a typo with r.desc, so here's the filter for future readers:

r.db('rethinkdb').table('logs')
  .orderBy({ index: r.desc("id") })
  .filter(function(log) {
    return log('message').match(cachedId).and(log('message').match('Raft leader'));
  })
  .limit(1);

@danielmewes so are you saying that even if raft_leader were available on a given table's configuration I wouldn't be able to set up a changefeed for it?

additionally it seems that orderyBy.limit doesn't work on system tables, bummer!

@mbroadst No that's not what I meant. I just meant that since your query uses both the table configuration as well as the logs table, it doesn't work with changefeeds at the moment.

The changefeed query I mentioned above on the logs table works fine for me.
Similarly, once we have the raft_leader in the table configuration, you can listen to changes by simply doing:

r.db('rethinkdb').table('table_status').filter({id: tableId})('raft_leader').changes()

You just need to retrieve the table ID before.

Eventually you'll also be able to use the more efficient

r.table(...).status()('raft_leader').changes()

but this doesn't yet work because our changefeeds on single documents (rather than sequences) don't allow transformations (here: selecting the 'raft_leader' field) yet. We hope to add that functionality soon.

@mbroadst Regarding orderBy.limit on system tables: Could you post the exact query you're running?

Something like the mentioned

r.db('rethinkdb').table('logs')
  .orderBy({ index: r.desc("id") })
  .filter(function(log) {
    return log('message').match(cachedId).and(log('message').match('Raft leader'));
  })
  .limit(1);

works fine for me.

It's changefeed specific, I get this error:

Unhandled rejection ReqlOpFailedError: System tables don't support changefeeds on `.limit()` in:
r.db("rethinkdb").table("logs").orderBy({
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    index: r.desc("id")
    ^^^^^^^^^^^^^^^^^^^
}).filter(function(var_2) {
^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return var_2("message").match("02aa72e5-cfbd-407b-86ff-870461a4c82b").and(var_2("message").match("Raft leader"))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
}).limit(1).changes()
^^^^^^^^^^^^^^^^^^^^^

Ah yes, that's true for changefeeds. You can just register a changefeed without the orderBy and limit though, and that should give you the updates whenever a new message matching the filter appears in the logs.

What I recommend doing is something like this:

1. Register a changefeed with the correct filter on the logs table (without any orderBy or limit).
2. Run the query with the orderBy and limit(1) to get the current raft leader
3. If you have seen any notifications on the changefeed, use the raft leader from the changefeed. If not, use the one from the manual one-time query.

RethinkDB 2.2 will have an include_initial option that allows you to get both the initial value and a changefeed together. We might be able to simplify this a bit once that option is available.

Yep that's what I've got now! I am definitely looking forward to all these new features though 😄

@danielmewes is there possibly a way to forcibly change the raft leader for a table from e.g. reql? This would be incredibly useful for testing on our side

@mbroadst Right now the only way is to take the server down or to interrupt the network connection (using iptables or similar).

We had a proposal for a debug interface to forcibly change the Raft leader, but it hasn't been implemented yet. I think we should eventually expose a system table API for all this (changing the Raft leader, subscribing to changes on it, etc.)

@danielmewes and I spoke on IRC earlier and discussed the possibility of also adding a raft_status alongside raft_leader so someone could theoretically make decisions based on whether e.g. an election is currently in process, or failed, etc.

Merged into next via commit 0f71d4b.