SIGSEGV when I try to view the stream content

[db-ks]

Observed behavior

I am connected through a k8s-hosted NATS server (2.10.29-alpine) to a leaf-node server (2.11.1) and trying to view data from a JetStream on that leaf-node using NATS CLI (0.2.3).

I get the following:

./nats --timeout 5m -s localhost:4223 --creds production.creds --js-api-prefix='$JS.Realm.RemoteEndpoint.API' stream view StreamName
Subject:  Reply: 

  Status: 503
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11875cd]

goroutine 1 [running]:
github.com/nats-io/jsm%2ego.(*MsgInfo).Stream(...)
	/home/runner/go/pkg/mod/github.com/nats-io/jsm.go@v0.2.3/msginfo.go:40
github.com/nats-io/natscli/cli.(*streamCmd).viewAction(0xc000036408, 0x40e3df?)
	/home/runner/work/natscli/natscli/cli/stream_command.go:1165 +0x88d
github.com/choria-io/fisk.(*actionMixin).applyActions(...)
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/actions.go:28
github.com/choria-io/fisk.(*Application).applyActions(0xc000415a40?, 0xc000402120)
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:827 +0xd0
github.com/choria-io/fisk.(*Application).execute(0xc000415a40, 0xc000402120, {0xc0002d85c0, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:613 +0x65
github.com/choria-io/fisk.(*Application).Parse(0xc000415a40, {0xc000150010?, 0x2c?, 0xc0004a2870?})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:276 +0x17f
github.com/choria-io/fisk.(*Application).MustParseWithUsage(0xc000415a40, {0xc000150010, 0xa, 0xa})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:893 +0x45
main.main()
	/home/runner/work/natscli/natscli/nats/main.go:84 +0x2e08

Expected behavior

No crash

Server and client version

• NATS server - 2.10.29-alpine
• NATS leaf-node server - 2.11.1
• NATS CLI - 0.2.3

Host environment

• k8s v1.31.10
• Ubuntu 24.04.3 LTS

Steps to reproduce

No response

[db-ks]

I also tried with the newest available NATS CLI (0.2.4), I get the same:

  Status: 503
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x117d14e]

goroutine 1 [running]:
github.com/nats-io/jsm%2ego.(*MsgInfo).Stream(...)
	/home/runner/go/pkg/mod/github.com/nats-io/jsm.go@v0.2.4/msginfo.go:40
github.com/nats-io/natscli/cli.(*streamCmd).viewAction(0xc000148408, 0x41617f?)
	/home/runner/work/natscli/natscli/cli/stream_command.go:1177 +0x8ae
github.com/choria-io/fisk.(*actionMixin).applyActions(...)
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/actions.go:28
github.com/choria-io/fisk.(*Application).applyActions(0xc0007263c0?, 0xc000160090)
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:827 +0xd0
github.com/choria-io/fisk.(*Application).execute(0xc0007263c0, 0xc000160090, {0xc00069f400, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:613 +0x65
github.com/choria-io/fisk.(*Application).Parse(0xc0007263c0, {0xc0000320c0?, 0x2c?, 0xc0000f10e0?})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:276 +0x17f
github.com/choria-io/fisk.(*Application).MustParseWithUsage(0xc0007263c0, {0xc0000320c0, 0xa, 0xa})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:893 +0x45
main.main()
	/home/runner/work/natscli/natscli/nats/main.go:84 +0x2dc8

[db-ks]

./nats --timeout 5m -s localhost:4223 --creds production.creds --js-api-prefix='$JS.Realm.RemoteEndpoint.API' stream info StreamName yields correct output

[ripienaar]

Can you please show the output from stream view using --trace?

Seems there is an unhandled error from the server

[db-ks]

11:13:55 >>> $JS.Realm.RemoteEndpoint.API.STREAM.INFO.Executions


11:13:55 >>> Connected to nats://localhost:4223 (127.0.0.1:4223)
11:13:55 <<< $JS.Realm.RemoteEndpoint.API.STREAM.INFO.Executions
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"Executions","subjects":["Prefix.RemoteEndpoint.Suffix.*","Prefix.RemoteEndpoint.Suffix.*.Parameters","Prefix.RemoteEndpoint.Suffix.*.Job.*","Prefix.RemoteEndpoint.Suffix.*.Job.*.Parameters","Prefix.RemoteEndpoint.Suffix.*.Job.*.Result.*","Prefix.RemoteEndpoint.Suffix.*.Logs","Prefix.RemoteEndpoint.Suffix.*.Artifact.*"],"retention":"interest","max_consumers":-1,"max_msgs":-1,"max_bytes":1000000000,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"new","storage":"file","num_replicas":1,"duplicate_window":120000000000,"compression":"none","allow_direct":true,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false,"consumer_limits":{},"allow_msg_ttl":false,"metadata":{"_nats.level":"1","_nats.req.level":"0","_nats.ver":"2.11.1"}},"created":"2025-07-10T21:24:59.547731299Z","state":{"messages":0,"bytes":0,"first_seq":510542,"first_ts":"1970-01-01T00:00:00Z","last_seq":510541,"last_ts":"2025-09-11T09:13:49.851309412Z","consumer_count":1},"domain":"RemoteEndpoint","cluster":{"name":"RemoteEndpoint","leader":"RemoteEndpoint"},"ts":"2025-09-11T09:13:55.522288716Z"}

11:13:55 >>> $JS.Realm.RemoteEndpoint.API.STREAM.INFO.Executions
{"offset":0}

11:13:55 <<< $JS.Realm.RemoteEndpoint.API.STREAM.INFO.Executions
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"Executions","subjects":["Prefix.RemoteEndpoint.Suffix.*","Prefix.RemoteEndpoint.Suffix.*.Parameters","Prefix.RemoteEndpoint.Suffix.*.Job.*","Prefix.RemoteEndpoint.Suffix.*.Job.*.Parameters","Prefix.RemoteEndpoint.Suffix.*.Job.*.Result.*","Prefix.RemoteEndpoint.Suffix.*.Logs","Prefix.RemoteEndpoint.Suffix.*.Artifact.*"],"retention":"interest","max_consumers":-1,"max_msgs":-1,"max_bytes":1000000000,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"new","storage":"file","num_replicas":1,"duplicate_window":120000000000,"compression":"none","allow_direct":true,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false,"consumer_limits":{},"allow_msg_ttl":false,"metadata":{"_nats.level":"1","_nats.req.level":"0","_nats.ver":"2.11.1"}},"created":"2025-07-10T21:24:59.547731299Z","state":{"messages":0,"bytes":0,"first_seq":510542,"first_ts":"1970-01-01T00:00:00Z","last_seq":510541,"last_ts":"2025-09-11T09:13:49.851309412Z","consumer_count":1},"domain":"RemoteEndpoint","cluster":{"name":"RemoteEndpoint","leader":"RemoteEndpoint"},"ts":"2025-09-11T09:13:55.691552538Z"}

11:13:55 >>> $JS.API.DIRECT.GET.Executions
{"seq":510542,"next_by_subj":"\u003e"}


11:13:56 <<< (0) map[Status:[503]]

Subject:  Reply: 

  Status: 503
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x117d14e]

goroutine 1 [running]:
github.com/nats-io/jsm%2ego.(*MsgInfo).Stream(...)
	/home/runner/go/pkg/mod/github.com/nats-io/jsm.go@v0.2.4/msginfo.go:40
github.com/nats-io/natscli/cli.(*streamCmd).viewAction(0xc00043c408, 0x41617f?)
	/home/runner/work/natscli/natscli/cli/stream_command.go:1177 +0x8ae
github.com/choria-io/fisk.(*actionMixin).applyActions(...)
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/actions.go:28
github.com/choria-io/fisk.(*Application).applyActions(0xc00015e000?, 0xc00016c120)
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:827 +0xd0
github.com/choria-io/fisk.(*Application).execute(0xc00015e000, 0xc00016c120, {0xc0002639e0, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:613 +0x65
github.com/choria-io/fisk.(*Application).Parse(0xc00015e000, {0xc0000320d0?, 0x2c?, 0xc0003c9ee0?})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:276 +0x17f
github.com/choria-io/fisk.(*Application).MustParseWithUsage(0xc00015e000, {0xc0000320d0, 0xb, 0xb})
	/home/runner/go/pkg/mod/github.com/choria-io/fisk@v0.7.1/app.go:893 +0x45
main.main()
	/home/runner/work/natscli/natscli/nats/main.go:84 +0x2dc8

[ripienaar]

It's pretty weird to see just a bare 503 with no description, can you please do:

$ nats req '$JS.API.DIRECT.GET.Executions' '{"seq":510542,"next_by_subj":">"}'

(but adjust for whatever seq you see in the error net time)

And show me the output?

The 503 indicates the cluster is just unhealthy I think, all I can do in this case is exit cleanly without panic and maybe a better error - but want to be sure exactly what's happening there.