panic in provide system near deque.(*Deque[...]).PushBack

[lidel]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

built from source

Version

0.39.0-dev-ae8667296 (`master` branch)

Config

defaults + `SweepEnabled=true`

Description

I was running smoke-test with latest master (ae86672) where I started pinning bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze (/ipns/en.wikipedia-on-ipfs.org/, ~350GiB, >20M CIDs) and in the morning I've found node dead:

2025-11-02T08:13:17.029+0100    INFO    dht/provider    provider/provider.go:1003       sent provider records to peers in 5.462364831s, errors 13/42
panic: runtime error: index out of range [16] with length 16

goroutine 283 [running]:
github.com/gammazero/deque.(*Deque[...]).PushBack(0x3f64c47abc, {0x3ede1c7050?, 0x23, 0x30})
        github.com/gammazero/deque@v1.1.0/deque.go:66 +0xf0
github.com/ipfs/go-dsqueue.(*DSQueue).worker(0x3f64c40150, {0x239f7d8, 0x3f64c037c0}, 0x4000, 0x0?, 0xdf8475800)
        github.com/ipfs/go-dsqueue@v0.1.0/dsqueue.go:305 +0x854
created by github.com/ipfs/go-dsqueue.New in goroutine 1

After restarting daemon and restarting pinning it crashed again:

[..]
2025-11-02T15:13:53.155+0100    DEBUG   autoconf        autoconf/fetch.go:118   GetCached: returning cached config
Daemon is ready
[..]
2025-11-02T15:16:35.354+0100    INFO    dht/provider    provider/provider.go:1003       sent provider records to peers in 5.020575289s, errors 13/33
panic: runtime error: index out of range [128] with length 128

goroutine 270 [running]:
github.com/gammazero/deque.(*Deque[...]).PushBack(0x3f4d68dabc, {0x3ee6ab7ec0?, 0x23, 0x30})
        github.com/gammazero/deque@v1.1.0/deque.go:66 +0xf0
github.com/ipfs/go-dsqueue.(*DSQueue).worker(0x3f4cbd4af0, {0x239f7d8, 0x3f4cbd3130}, 0x4000, 0x0?, 0xdf8475800)
        github.com/ipfs/go-dsqueue@v0.1.0/dsqueue.go:305 +0x854
created by github.com/ipfs/go-dsqueue.New in goroutine 1
        github.com/ipfs/go-dsqueue@v0.1.0/dsqueue.go:69 +0x378

After restarting and NOT resuming pinning, the node runs fine and does not crash, which suggests the problem is in code paths related to providing newly retrieved blocks.

I did third test and restarted and resumed pinning and it seemed to work fine, but panic occurred again after ~49min.

[lidel]

@guillaumemichel i know not everything landed in master yet, is this by chance something you've already seen, or a new thing that me and @gammazero need to investigate once everything lands?

[guillaumemichel]

I haven't seen this error before, so the problem unlikely to be addressed in open PRs.

Just glancing through the message, the problem seems to come from go-dsqueue. The only kad-dht user is the buffered provider.

It is possible that go-dsqueue is used incorrectly in the buffered provider, or the problem comes from go-dsqueue directly. @gammazero could you have a look?

[gammazero]

• This could not be reproduced on multiple systems, even when recursively pinning all of wikipedia.
• Added a stress test that did reproduce the issue on the original risc-v system, but it also cannot reproduce the problem on any other system.
• Updated deque package to fix a small inefficiency when using a non-default base capacity.