A third relatively easy option is to reduce the amount of stored information by making the following optimizations:

• Don't store recency for blob blocks. They can just use the recency of the btree leaf node that they are attached to. Internally we would probably have two LBA lists in the serializer, one for blocks that have a recency and one for blocks that don't. We would partition the space of block ids into something like two halves, to identify which block id belongs where (and the cache would assign block ids from the different pools accordingly).
• ser_block_size doesn't need to be 32 bit. 16 bit would probably be enough, maybe 24 bit.
• flagged_off64_t also doesn't need the full 64 bit.

That would easily cut the overhead in half.

This would work well, because whenever we modify a blob, we just rewrite it completely anyway.

Not that this may not be the case forever, so I wouldn't count on this functionality. There is a lot of demand for incremental document updates (i.e. incrementing a counter in a large doc shouldn't affect the whole doc). We can get away with things as they are for now, but I don't think we'll be able to get away with it forever.

@coffeemug: That's ok though. We can still update the recency of the leaf node whenever we (partially) modify a blob. We just couldn't selectively backfill only a part of the blob.

(Edit: My initial reply missed the point. Replaced by something better)

Working on this now.
I'm cutting out the replication timestamp for blob blocks, which saves 16 bytes per such block (8 bytes in the in-memory LBA, and 8 bytes in the buffer cache).
I'm also reducing the block size from 32 to 16 bit. We currently only use block sizes up to 4 KB, so this is still plenty.

I'm keeping the on-disk format unchanged, so that we can easily revert any of these changes later if for example we want to support blocks of more than 64 KB at some point.

These changes should get the worst-case overhead from ~5% down to ~2%.

I have a working version in branch daniel_1951.
The branch also includes a few unrelated improvements to the cache's memory usage accounting, that make the actually used cache memory remain significantly closer to the specified cache size (though it's still not quite identical).

This change has a slightly strange interaction with existing tables:
With the change, the in-memory copy of the LBA now has two arrays from block ID's to block infos. One for regular blocks (IDs starting at 0), and one for blob blocks (IDs now starting at 2^63). The array for blob blocks uses the more compact structure that omits the recency timestamp.
Every newly written blob is going to use block IDs from the new ID range for blobs, and is going to end up in the blob array. As more and more values of a table get rewritten, most of the block IDs used for blobs are going to be moved over into the blob LBA array.
The weird side effect of this is that the regular blob array doesn't actually shrink, as long as there's at least one block ID close to the end of the block range that's still in use by a regular block. So even if values in an existing table are getting rewritten, this change is not actually going to save as much memory as one might expect. (Though it will still save roughly 8 bytes per block due to an additional optimization.)
The change is therefore going to be most effective for new tables, or for tables that are still growing in size due to inserts.

In CR 3241 by @Tryneus .

Merged into next cfca276