go/oasis-node: Previously synced node starts lagging after enabling aggressive pruning

[martintomazic]

When doing experiments with aggressive pruning (state from the genesis -> keep only 1000 latest blocks), I managed to overwhelm already synced node, making latest_height be 2h behind the actual latest block as observed on the block explorer.

To my surprise, the status was still marked as ready:

 "consensus": {
    "status": "ready",
    ....
}

Worst, we have parts of the runtime code that never assume a consensus node may fall behind once marked as ready. E.g. https://github.com/oasisprotocol/oasis-core/blob/49023d0c895ef6f1eead7292dbe44d5e19a23dea/go/worker/storage/committee/worker.go#L1197:L1199.

[martintomazic]

Explanation:
Consensus state pruning starts with few minutes lag, but once it does the pruning is extremely fast (e.g. 6 million blocks in 8 minutes when no runtime configured, else prune handlers slow things down...). Then there is also cometbft blockstore pruning that lags behind a bit with pruning. Both processes effectively starve the progress of syncing... So if you had synced node with lots of state, stop it and configure pruning to keep only few blocks, and finally re-start immediately, this is quite possible.

As can be seen state pruning is not rate limited. Same applies for the blockstore, where pruning of the blocks is not rate limited.

In essence this is reverse of the issue #6334, where pruning is rate limited too much.

TODO: Also note that once the cometbft catches up via the block sync reactor, it is switched to consensus reactor. I would have to check but maybe this one is slower so once the node falls behind and was switched here it has trubles catching up again.

[martintomazic]

This will be mitigated via #6341 and documented as part of the oasisprotocol/docs#1469.