go/oasis-node/cmd/storage: Add command that flattens consensus dbs

[martintomazic]

Part of #6309.

This command has been thoroughly tested and is able to immediately reclaim all the disk space even if enabling pruning late.

Unfortunately, there is a known corner case where disk space used during the peak of the compaction may be double the original size - see. It is for this reason that I suggest to release this as experimental command.

Performance:
Currently it takes less then 1h to compact all the consensus databases (with all the state/blocks from the genesis) on my 2TB playground.

I haven't observed that increasing number of the Flatten workers would increase the overall disk usage during the peak of the compaction (as warned by the badger cli documentation). How about adding this as an optional parameter?

This way users with spare disk space could e.g. configure it to use 8 workers, and thus speed-up compaction significantly. 3-5 times as from my experiments.

We may also explore the impact of increasing BlockCacheSize...

Follow-up
Given that I was testing this only on the consensus DB instances, I suggest to add compaction of the runtime DB instances as a follow-up. The testing there is also complicated as runtime state pruning is very slow, making it hard to prepare realistic samples.

I suggest the following order of follow-ups:

1. go/oasis-node/cmd/storage: Add command for offline pruning #6341 (only consensus)
2. Enable compaction for the runtime databases.
3. E2e test for the compact command.
4. Enable offline pruning for the runtime databases.

[netlify]

✅ Deploy Preview for oasisprotocol-oasis-core canceled.

Name	Link
🔨 Latest commit	ac4c227
🔍 Latest deploy log	https://app.netlify.com/projects/oasisprotocol-oasis-core/deploys/68f5f3ed16873800088d8e3d

[martintomazic]

Commands above don't initialize this, thus all the logs are lost as logger is not initialized. Note this redirects logs to stdout, which depending on the configuration may produce a lot of logs on the operator terminal.

Should this be actually written to stderr by default?

update: related to #6311 (comment)

[martintomazic]

Added a docs section as requested yesterday.

[codecov]

Codecov Report

❌ Patch coverage is 14.77273% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.19%. Comparing base (8e2ca0e) to head (c4616bb).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
go/oasis-node/cmd/storage/storage.go	1.61%	61 Missing ⚠️
go/storage/mkvs/db/badger/badger.go	25.00%	6 Missing ⚠️
go/storage/mkvs/db/pathbadger/pathbadger.go	25.00%	6 Missing ⚠️
go/storage/mkvs/db/api/api.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6311      +/-   ##
==========================================
- Coverage   64.82%   64.19%   -0.63%     
==========================================
  Files         698      698              
  Lines       67826    67908      +82     
==========================================
- Hits        43968    43594     -374     
- Misses      18860    19297     +437     
- Partials     4998     5017      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.