shim disconnected without error message

[ydp]

I replaced containerd-shim with this rust version, but it's not working, and just disconnected without any error, could you please help how to look into this? Thanks!

Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.167197484Z" level=info msg="CreateContainer within sandbox \"f1d615c78e3365992fa92ad36b24b05fb9e471688358dc2add58712ba22bf6db\" for &ContainerMetadata{Name:alpine,Attempt:5,} returns container id \"557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d\""
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.168317205Z" level=info msg="StartContainer for \"557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d\""
Jan 12 01:25:10 blade-5 containerd[1038855]: [INFO] Create request for CreateTaskRequest { id: "557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d", bundle: "/run/containerd/io.containerd.runtime.v2.task/k8s.io/557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d", rootfs: [Mount { type_: "overlay", source: "overlay", target: "", options: ["index=off", "workdir=/data0/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/18826/work", "upperdir=/data0/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/18826/fs", "lowerdir=/data0/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/28/fs:/data0/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/27/fs:/data0/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/26/fs:/data0/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/25/fs"], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }], terminal: false, stdin: "", stdout: "/run/containerd/io.containerd.grpc.v1.cri/containers/557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d/io/3215463519/557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d-stdout", stderr: "/run/containerd/io.containerd.grpc.v1.cri/containers/557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d/io/3215463519/557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d-stderr", checkpoint: "", parent_checkpoint: "", options: MessageField(Some(Any { type_url: "runtimeoptions.v1.Options", value: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } })), special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
Jan 12 01:25:10 blade-5 containerd[1038855]: [INFO] Create request for 557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d returns pid 1040500
Jan 12 01:25:10 blade-5 containerd[1038855]: [INFO] Connect request for ConnectRequest { id: "557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d", special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.527656970Z" level=info msg="shim disconnected" id=557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.527745732Z" level=warning msg="cleaning up after shim disconnected" id=557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d namespace=k8s.io
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.527765055Z" level=info msg="cleaning up dead shim"
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.527754174Z" level=error msg="failed to get task exit status for \"f1d615c78e3365992fa92ad36b24b05fb9e471688358dc2add58712ba22bf6db\"" error="ttrpc: closed: unknown"
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.527726014Z" level=error msg="Failed to pipe stdout of container \"557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d\"" error="read /proc/self/fd/45: file already closed"
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.527654842Z" level=info msg="shim disconnected" id=f1d615c78e3365992fa92ad36b24b05fb9e471688358dc2add58712ba22bf6db
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.527871643Z" level=error msg="Failed to pipe stderr of container \"557e3eb3129cea488bebe475d1f0734145c213e0e4c248dd613e630b8ea9df8d\"" error="read /proc/self/fd/46: file already closed"
Jan 12 01:25:10 blade-5 containerd[1038855]: time="2023-01-12T01:25:10.527900006Z" level=warning msg="cleaning up after shim disconnected" id=f1d615c78e3365992fa92ad36b24b05fb9e471688358dc2add58712ba22bf6db namespace=k8s.io

Containerd version:1.6.14
config change:

runtime_type = "io.containerd.runc.v2-rs"

[Burning1020]

Is the shim process not running? It seems it exited somehow before start task.

[ydp]

I can see the shim process is started running, then after a short period, it exits.

[Burning1020]

We can infer that the shim exited unexpectedly before the task start api without any concrete message, you can add some debug code then debug it.

BTW, I used to have a same problem before and it is caused by the old version of OCI spec json, you can check it if you have this file.