core/txpool/blobpool: migrate billy to new slot size #31966

MariusVanDerWijden · 2025-06-04T16:43:38Z

Implements a migration path for the blobpool slotter

core/txpool/blobpool/blobpool.go

lightclient · 2025-07-08T21:09:47Z

core/txpool/blobpool/blobpool.go

+	// txBlobOverhead is an approximation of the overhead that an additional blob
+	// has on transaction size. This is added to the slotter to avoid
+	// tiny overflows causing all txs to move a shelf higher, wasting disk space.
+	txBlobOverhead = 2048


Just curious, is this a cautious estimate or based on a specific computation?

This is an estimation based on specific computation :D
I computed the size of a bunch of blob transactions and found that the overhead is ~1300 bytes iirc. So 2048 kinda gives us a round estimate over that

I think we should update this value.

The size of original proof is 48 bytes, while the new version cell proof is 128 * 48 = 6144, already beyond this constant.

Okay I bumped it to 6144, plus 64 of wiggle room.

lightclient · 2025-07-08T21:15:43Z

core/txpool/blobpool/blobpool.go

+	if p.chain.Config().IsOsaka(head.Number, head.Time) {
+		// Check if we need to migrate our blob db to the new slotter
+		oldSlotSize := 135168
+		if os.Stat(filepath.Join(queuedir, fmt.Sprintf("bkt_%08d.bag", oldSlotSize))); err == nil {


These seems to be taking advantage of billy internals -- can we just open the db and use Infos() to get the current slot size?

We could, but that would mean we need to open the DB. I agree that this is not a super clean solution...

core/txpool/blobpool/slotter.go

lightclient · 2025-07-14T22:21:32Z

@MariusVanDerWijden let me know what you think about this last commit. I think the best thing to do is probably make a PR to billy to improve the migration flow, but given the current interface it seems like this is the best we can do downstream without hard coding any billy-specific behavior.

core/txpool/blobpool/slotter.go

core/txpool/blobpool/blobpool.go

lightclient · 2025-09-04T17:26:23Z

Okay I have updated the PR with a conversion step to convert v0 sidecars to v1 on startup. However, I wonder how useful this is since the code only runs on start up after Osaka has passed. So by the time this runs, I would guess most of the blobs actually do not have v0 sidecars.

@rjl493456442 since you made the initial comment, wdyt? Some other approaches we could take:

migrate the DB once Osaka is scheduled, but don't convert sidecars
do 1), but also convert the sidecars online after the fork passes (I shied away from this because of the comment in Clear() about how inefficient it is to iterate through all tx metas, but it could be done in another thread while yielding back locks throughout to not stop all progress.
continue to migrate DB after Osaka passes on startup, but separately do online conversion like mentioned in 2).

Right now I'm leaning towards 2).

Co-authored-by: lightclient <[email protected]>

…ng billy

…gration

… operation

…ctive

lightclient · 2025-09-11T19:48:54Z

Updated in line with 2) as we discussed.

MariusVanDerWijden · 2025-09-11T20:17:57Z

LGTM

core/txpool/blobpool/blobpool.go

core/txpool/blobpool/limbo.go

core/txpool/blobpool/slotter.go

rjl493456442 · 2025-09-15T04:04:11Z

[[ OLD GETH ]]
gary@dev:~/mount/eth-mainnet/geth/blobpool/limbo$ ls
bkt_00004096.bag  bkt_00266240.bag  bkt_00528384.bag  bkt_00790528.bag  bkt_01052672.bag  bkt_01314816.bag  bkt_01576960.bag  bkt_01839104.bag  bkt_02101248.bag
bkt_00135168.bag  bkt_00397312.bag  bkt_00659456.bag  bkt_00921600.bag  bkt_01183744.bag  bkt_01445888.bag  bkt_01708032.bag  bkt_01970176.bag  bkt_02232320.bag
gary@dev:~/mount/eth-mainnet/geth/blobpool/limbo$ ^C
gary@dev:~/mount/eth-mainnet/geth/blobpool/limbo$ ls

[[ AFTER MIGRATION ]]
bkt_00000008.bag  bkt_00141376.bag  bkt_00415936.bag  bkt_00690496.bag  bkt_00965056.bag  bkt_01239616.bag  bkt_01514176.bag  bkt_01788736.bag  bkt_02063296.bag  bkt_02337856.bag
bkt_00004096.bag  bkt_00278656.bag  bkt_00553216.bag  bkt_00827776.bag  bkt_01102336.bag  bkt_01376896.bag  bkt_01651456.bag  bkt_01926016.bag  bkt_02200576.bag
gary@dev:~/mount/eth-mainnet/geth/blobpool/limbo$ ls ../queue/
bkt_00000008.bag  bkt_00141376.bag  bkt_00415936.bag  bkt_00690496.bag  bkt_00965056.bag  bkt_01239616.bag  bkt_01514176.bag  bkt_01788736.bag  bkt_02063296.bag  bkt_02337856.bag
bkt_00004096.bag  bkt_00278656.bag  bkt_00553216.bag  bkt_00827776.bag  bkt_01102336.bag  bkt_01376896.bag  bkt_01651456.bag  bkt_01926016.bag  bkt_02200576.bag

As a consequence of moving blob sidecar version migration code around, we ended up building blocks with a mix of v0 and v1 blob transactions (different proof encoding in the sidecar). This PR makes sure we are not building illegal blocks after Osaka. Blob migration is left for another PR. Related issues and PRs: - #31791 - #32347 - #31966 - #32235 --------- Signed-off-by: Csaba Kiraly <[email protected]>

…32577) As a consequence of moving blob sidecar version migration code around, we ended up building blocks with a mix of v0 and v1 blob transactions (different proof encoding in the sidecar). This PR makes sure we are not building illegal blocks after Osaka. Blob migration is left for another PR. Related issues and PRs: - ethereum#31791 - ethereum#32347 - ethereum#31966 - ethereum#32235 --------- Signed-off-by: Csaba Kiraly <[email protected]>

Implements a migration path for the blobpool slotter --------- Co-authored-by: lightclient <[email protected]> Co-authored-by: lightclient <[email protected]> Co-authored-by: Gary Rong <[email protected]>

MariusVanDerWijden requested review from fjl, gballet, jwasinger, lightclient and rjl493456442 as code owners June 4, 2025 16:43

MariusVanDerWijden force-pushed the peerdas-slotter branch from 2503da0 to 7539ab9 Compare June 4, 2025 16:43

MariusVanDerWijden mentioned this pull request Jun 4, 2025

eth/catalyst: implement getBlobsV2 #31791

Merged

fjl changed the title ~~WIP: Peerdas slotter~~ core/txpool/blobpool: peerdas slotter Jun 17, 2025

fjl assigned lightclient Jun 17, 2025

MariusVanDerWijden commented Jun 20, 2025

View reviewed changes

core/txpool/blobpool/blobpool.go Outdated Show resolved Hide resolved

MariusVanDerWijden changed the title ~~core/txpool/blobpool: peerdas slotter~~ core/txpool/blobpool: migrate billy for new slot size Jul 7, 2025

MariusVanDerWijden changed the title ~~core/txpool/blobpool: migrate billy for new slot size~~ core/txpool/blobpool: migrate billy to new slot size Jul 7, 2025

lightclient reviewed Jul 8, 2025

View reviewed changes

fjl added the osaka label Jul 10, 2025

rjl493456442 self-assigned this Jul 22, 2025

rjl493456442 reviewed Jul 25, 2025

View reviewed changes

core/txpool/blobpool/slotter.go Outdated Show resolved Hide resolved

rjl493456442 reviewed Jul 25, 2025

View reviewed changes

core/txpool/blobpool/blobpool.go Outdated Show resolved Hide resolved

rjl493456442 reviewed Jul 25, 2025

View reviewed changes

core/txpool/blobpool/blobpool.go Outdated Show resolved Hide resolved

lightclient force-pushed the peerdas-slotter branch from e561a53 to 83a27ca Compare August 25, 2025 15:19

lightclient force-pushed the peerdas-slotter branch 2 times, most recently from 76146a5 to 90502a0 Compare September 4, 2025 17:12

lightclient added the status:triage label Sep 9, 2025

cskiraly mentioned this pull request Sep 10, 2025

core/txpool/blobpool: Do not use blobs with wrong sidecar version while building block after Osaka #32577

Merged

fjl removed the status:triage label Sep 11, 2025

lightclient force-pushed the peerdas-slotter branch from 3b3805f to ebb2a43 Compare September 11, 2025 19:40

MariusVanDerWijden added 3 commits September 11, 2025 13:43

core/txpool/blobpool: adjust slot size for cell proof transactions

f246a99

core/txpool/blobpool: adjust slot size for cell proof transactions

ae2cd38

core/txpool/blobpool: factor out peerdas slotter

f599be1

lightclient and others added 9 commits September 11, 2025 13:44

go.mod: run go mod tidy

72c3a94

Update core/txpool/blobpool/slotter.go

fc8d926

Co-authored-by: lightclient <[email protected]>

core/txpool/blobpool: correctly assign error from os.Stat when checki…

0147ee1

…ng billy

core/txpool/blobpool: read slotter version from billy before doing mi…

b8340ac

…gration

core/txpool/blobpool: fix slotter slot size

89f21d2

core/txpool/blobpool: update blobpool test for new sidecar version

dd99cf0

core/txpool/blobpool: bump per blob overhead

0a061be

core/txpool/blobpool: remove check for version byte in normal slotter…

30832e2

… operation

core/txpool/blobpool: migrate billy if osaka is scheduled, not just a…

38ed51c

…ctive

lightclient force-pushed the peerdas-slotter branch from 2275e46 to 38ed51c Compare September 11, 2025 19:44

lightclient previously approved these changes Sep 11, 2025

View reviewed changes

rjl493456442 reviewed Sep 12, 2025

View reviewed changes

core/txpool/blobpool/blobpool.go Outdated Show resolved Hide resolved

rjl493456442 reviewed Sep 12, 2025

View reviewed changes

core/txpool/blobpool/blobpool.go Outdated Show resolved Hide resolved

core/txpool/blobpool: fix overhead constant

681716a

MariusVanDerWijden dismissed lightclient’s stale review via 681716a September 12, 2025 08:14

MariusVanDerWijden added 2 commits September 12, 2025 14:09

core/txpool/blobpool: fix overhead constant

ace873c

core/txpool/blobpool: also migrate limbo

e6ea290

rjl493456442 reviewed Sep 15, 2025

View reviewed changes

core/txpool/blobpool/limbo.go Show resolved Hide resolved

rjl493456442 reviewed Sep 15, 2025

View reviewed changes

core/txpool/blobpool/slotter.go Show resolved Hide resolved

core/txpool/blobpool: correct blobpool datadir

2e37dd8

rjl493456442 approved these changes Sep 15, 2025

View reviewed changes

rjl493456442 added this to the 1.16.4 milestone Sep 15, 2025

rjl493456442 merged commit df0bd89 into ethereum:master Sep 15, 2025
5 of 6 checks passed

core/txpool/blobpool: migrate billy to new slot size #31966

core/txpool/blobpool: migrate billy to new slot size #31966

Uh oh!

Conversation

MariusVanDerWijden commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lightclient Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

MariusVanDerWijden Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

rjl493456442 Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightclient Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

lightclient Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

MariusVanDerWijden Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lightclient commented Jul 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightclient commented Sep 4, 2025

Uh oh!

lightclient commented Sep 11, 2025

Uh oh!

MariusVanDerWijden commented Sep 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rjl493456442 commented Sep 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MariusVanDerWijden commented Jun 4, 2025 •

edited

Loading

rjl493456442 Jul 25, 2025 •

edited

Loading