How to find out what caused a corruption in JetStream Cluster?

[KenLR]

Summary

We experienced persistent RAFT corruption in our 3-node JetStream cluster (NATS v2.9.8) that caused streams to lose quorum and JetStream to shut down with Critical write error: malformed or corrupt message errors. Even after deleting and recreating individual PVCs, the corruption spread back from other replicas. We're trying to understand what could have caused this and how to prevent it.

Environment

NATS Server:

• Version: nats:2.9.8-alpine
• Deployment: Kubernetes StatefulSet with 3 replicas
• Storage: AWS EBS gp3 volumes (200GB per pod)

JetStream Configuration:

• Replicas: 3
• Storage: File-backed
• Retention: Work queue
• Max message size: 1MB

Kubernetes:

• StatefulSet managing 3 pods
• Each pod has dedicated PVC
• TLS enabled with mTLS authentication

Server-Side Errors

NATS logs on nats-jetstream-0:

[WRN] RAFT [xY6fvr60 - C-R3F-atNRG1rP] Wrong index, ae is &{leader:VNsg0pXW term:7 commit:3258048 ...}, index stored was 3258025, n.pindex is 3258048
[ERR] RAFT [xY6fvr60 - C-R3F-atNRG1rP] Critical write error: malformed or corrupt message
[ERR] JetStream out of resources, will be DISABLED
[WRN] Failed to install snapshot for '$G > IDX_TRADE_P005 > WORK_TRADE' [C-R3F-atNRG1rP]: malformed or corrupt message
[INF] Initiating JetStream Shutdown...
[INF] JetStream Shutdown

also seeing this:

[8] 2025/10/22 19:45:48.179026 [WRN] RAFT [xY6fvr60 - C-R3F-q6QNvUn0] Wrong index, ae is &{leader:VNsg0pXW term:65 commit:108751 pterm:65 pindex:108751 entries: 1}, index stored was 5066558170728645, n.pindex is 108751
[8] 2025/10/22 19:45:48.179040 [WRN] RAFT [xY6fvr60 - C-R3F-q6QNvUn0] Expected first catchup entry to be a snapshot and peerstate, will retry
[8] 2025/10/22 19:45:49.179115 [WRN] RAFT [xY6fvr60 - C-R3F-q6QNvUn0] Wrong index, ae is &{leader:VNsg0pXW term:65 commit:108751 pterm:65 pindex:108751 entries: 1}, index stored was 5066558170728646, n.pindex is 108751
[8] 2025/10/22 19:45:49.179129 [WRN] RAFT [xY6fvr60 - C-R3F-q6QNvUn0] Error storing entry to WAL: raft: could not store entry to WAL

Other pods in the cluster reported:

[WRN] JetStream cluster stream '$G > IDX_TRADE_P005' has NO quorum, stalled

Additional Context

• The cluster had been running for 20 days before the corruption appeared
• Storage usage was only 6-7% of 200GB capacity
• Memory usage was 1.3GB out of 2GB limit

Recovery Attempts

Attempt 1: Restart Pod

• Action: Deleted nats-jetstream-0 pod
• Result: Pod restarted, but immediately hit the same corruption on PVC reload

Attempt 2: Delete Stream & Pod

• Action:
  • Deleted stream CRD idx-trade-p000
  • ArgoCD auto-recreated it
  • Restarted pod
• Result: Same corruption errors

Attempt 3: Delete PVC for Pod 0

• Action:
  • Deleted nats-jetstream-0 pod
  • Deleted its PVC nats-jetstream-js-pvc-nats-jetstream-0
  • Let pod recreate with fresh PVC
• Result: Pod started syncing from healthy replicas but then:

[WRN] RAFT [xY6fvr60 - C-R3F-zA7ksrsX] Wrong index, ae is &{leader:eEF5hQCc term:7 commit:87695 ...}
[ERR] RAFT [xY6fvr60 - C-R3F-zA7ksrsX] Critical write error: malformed or corrupt message
[ERR] JetStream out of resources, will be DISABLED

Now when I restart I don't even see the line "restored" log for the affected stream like this one:

[7] 2025/10/22 19:52:18.430999 [INF] Restored 1,859 messages for stream '$G > IDX_TRADE_P001'

Any suggestions on how to debug, fix or mitigate such issues going forward would be very welcome.

[ripienaar]

2.9.x is now very old, unsupported and 100s of bug fixes behind, we have invested a lot of time on improving the areas you mention.

You need to upgrade to 2.12.x, we can’t help with such old versions.

[KenLR]

Thanks! I should have checked before posting.. upgrading did fix the issue:

[7] 2025/10/22 23:08:17.651214 [INF]   Restored 998,291 messages for stream '$G > IDX_TRADE_P001' in 13.513s