feat: Update /etc/sub*id files to support nested containers #232

tolusha · 2025-09-02T12:57:56Z

Reference issue

https://issues.redhat.com/browse/CRW-8320

How to test

launch 4.20 aws,no-spot
chectl update next && chectl server:deploy -p openshift
Start workspace from [1].
Check that podman build work, podman run doesn't work
Stop workspace devfile
Create SCC

cat << EOF | oc apply -f -
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: nested-podman-scc
priority: null
allowPrivilegeEscalation: true
allowedCapabilities:
- SETUID
- SETGID
fsGroup:
  type: MustRunAs
  ranges:
  - min: 1000
    max: 65534
runAsUser:
  type: MustRunAs
  uid: 1000
seLinuxContext:
  type: MustRunAs
  seLinuxOptions:
    type: container_engine_t
supplementalGroups:
  type: MustRunAs
  ranges:
  - min: 1000
    max: 65534
userNamespaceLevel: RequirePodLevel
EOF

Update CheCluster CR

oc patch checluster eclipse-che -n eclipse-che --patch '{"spec": {"devEnvironments": {"containerBuildConfiguration": {"openShiftSecurityContextConstraint": "nested-podman-scc"}}}}' --type=merge

Start workspace from [2]
Check that podman build and podman run works

podman build:

cd /projects
cat > Dockerfile <<EOF
FROM python:3.12
WORKDIR /usr/local/app
EOF

podman build .

podman run:

podman run -d --rm --name webserver -p 8080:80 quay.io/libpod/banner

[1] https://gist.githubusercontent.com/tolusha/f39a0ef91feadcd4adae832a5bc2b7fa/raw/59c5d4040b152d67db1e7f521aee8d2ddbc2a14b/devfile.yaml
[2] https://gist.githubusercontent.com/tolusha/5f92d30b7456f614cc793fc0c671c28e/raw/6f36816b260752814565ed0f1c29a7cd72641cc7/devfile.yaml

tolusha · 2025-09-02T13:05:30Z

cc @cgruver

github-actions · 2025-09-02T13:20:33Z

Pull Request images published ✨

UDI: quay.io/devfile/universal-developer-image:pr-232

dkwon17 · 2025-09-03T23:16:04Z

base/ubi9/entrypoint.sh

+        USER_NAME=$(whoami)
+        START_ID=$(( $(id -u) + 1 ))
+        END_ID=$(( 65536 - ${START_ID} ))
+        ID_RANGE="${USER_NAME}:${START_ID}:${END_ID}"


Is there a required length do we need the range to be?

dkwon17 · 2025-09-03T23:16:23Z

base/ubi9/entrypoint.sh

+  if [ -f /proc/self/uid_map ]; then
+    if ! grep -q '^\s*0\s\+0\s\+\([2-9]\|[1-9][0-9]\+\)' /proc/self/uid_map; then
+      echo "Running in a user namespace, user id: $(id -u)"
+      # By default, the valid UIDs/GIDs is the range 0-65535


IMHO the upstream https://github.com/devfile/developer-images should also work with CRI-O engine / OpenShift, can we support uids such as 1001250000?

We will use a dedicated SCC to enable container run capabilities [1]
So, the userid will always be lower than 65536

[1] https://github.com/cgruver/ocp-4-18-nested-container-tech-preview

github-actions · 2025-09-04T15:38:13Z

Pull Request images published ✨

UDI: quay.io/devfile/universal-developer-image:pr-232

base/ubi9/entrypoint.sh

github-actions · 2025-09-05T08:43:31Z

Pull Request images published ✨

UDI: quay.io/devfile/universal-developer-image:pr-232

dkwon17 · 2025-09-06T03:20:32Z

base/ubi9/entrypoint.sh

+      #         0          0 4294967295
+      # 2. When running in a user namespace:
+      #         0 1481179136      65536
+      # 3. When container is run in a user namespace:


3. When container is run in a user namespace:

Just so that I understand correctly, this is the nested container, is that right?

dkwon17 · 2025-09-06T03:41:15Z

base/ubi9/entrypoint.sh

+    # Why do we need to update /etc/sub*id files?
+    # From one hand, we are already in user namespace, so we don't have more than 65536 UIDs/GIDs available.
+    # See for more details:
+    # - https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#id-count-for-each-of-pods
+    # From the other hand, podman creates a new user namespace for a container being launched and maps the container's user
+    # and group IDs (UID/GID) to the corresponding user on the current namespace.
+    # For the mapping, podman refers to the /etc/sub*id files.
+    # See for more details:
+    # - https://man7.org/linux/man-pages/man5/subuid.5.html
+    # So, if user ID exceeds 65536 number, than it can be not be mapped, since it is not just available.
+    # As a result, podman commands fails.
+
+    # Even though, the range can be extended if needed by configuration, we can relay on the fact that IDs number at
+    # least 65536 are always available in a user namespace.
+    # See for more details:
+    # - https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-UserNamespaces


Suggested change

# Why do we need to update /etc/sub*id files?

# From one hand, we are already in user namespace, so we don't have more than 65536 UIDs/GIDs available.

# See for more details:

# - https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#id-count-for-each-of-pods

# From the other hand, podman creates a new user namespace for a container being launched and maps the container's user

# and group IDs (UID/GID) to the corresponding user on the current namespace.

# For the mapping, podman refers to the /etc/sub*id files.

# See for more details:

# - https://man7.org/linux/man-pages/man5/subuid.5.html

# So, if user ID exceeds 65536 number, than it can be not be mapped, since it is not just available.

# As a result, podman commands fails.

# Even though, the range can be extended if needed by configuration, we can relay on the fact that IDs number at

# least 65536 are always available in a user namespace.

# See for more details:

# - https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-UserNamespaces

# Why do we need to update /etc/sub*id files?

# We are already in the user namespace, so we know there are at least 65536 UIDs/GIDs available.

# For more details, see:

# - https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#id-count-for-each-of-pods

# Podman needs to create a new user namespace for any container being launched and map the container's user

# and group IDs (UID/GID) to the corresponding user on the current namespace.

# For the mapping, podman refers to the /etc/sub*id files.

# For more details, see:

# - https://man7.org/linux/man-pages/man5/subuid.5.html

# So if the user ID exceeds 65535, it cannot be mapped if only UIDs/GIDs from 0-65535 are available.

# If that's the case, podman commands would fail.

# Even though the range can be extended using configuration, we can rely on the fact that there are at least 65536 user IDs available in the user namespace.

# See for more details:

# - https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-UserNamespaces

@tolusha could you please check if I understood this correctly?

Everything is correct

Is there a technical/security reason as to why we need to set START_ID=$(( ${USER_ID} + 1 )) ? Or is it a best practice in general?

For example, technically if START_ID was hardcoded so that we have the same COUNT every time, it wouldn't cause problems with podman, is that right?

Is there a technical/security reason as to why we need to set START_ID=$(( ${USER_ID} + 1 )) ? Or is it a best practice in general?

There are technical reasons.

We can't include a current user id in the range:

$ id uid=1234(user) gid=0(root) groups=0(root),1234(user) $ cat /etc/subuid user:100:10000 $ podman pull quay.io/devfile/base-developer-image:ubi9-latest ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": invalid configuration: the specified mapping 100:10000 in "/etc/subuid" includes the user UID

We have to include as many as possible ids in the range to avoid the error potentially insufficient UIDs or GIDs available in user namespace while running podman command (for instance build or pull commands).
In my case

$ cat /etc/subuid user:10000:1000 $ podman pull quay.io/devfile/base-developer-image:ubi9-latest .... Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:05edb0cb5711ef3833206fbfe874a63095567d5de5a33d8a09d96b4b6981e532"/""/"sha256:299446d31e276f48e402c9d6b1224c3a406482d3ce8108faa12ced2b5f022808": ApplyLayer stdout: stderr: potentially insufficient UIDs or GIDs available in user namespace (requested 10001:0 for /etc/profile.d/udi_prompt.sh): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /etc/profile.d/udi_prompt.sh: invalid argument exit status 1

For example, technically if START_ID was hardcoded so that we have the same COUNT every time, it wouldn't cause problems with podman, is that right?

If we are taking about something like this by default:

$ cat /etc/subuid user:10002:55534

It is not possible for downstream where user will be added into a container by cri-o and we have to update /etc/sub*id files in an entrypoint.sh any way to set to correct user name.

We reduce the ids range by default. It might lead to the following issue:

enable persistUserHome

start wroskapce

build an image

check permissions, some files are owned by not a current user (I am not 100% sure how podman works here)

$ ls -l ~/.local/share/containers/storage/vfs/dir/<...>/etc ... -rw-r-----. 1 user 10041 373 Sep 8 21:55 gshadow -rw-r-----. 1 user 10041 364 Sep 8 00:00 gshadow- ...

restart a workspace

update /etc/sub*id files not to include user with 10041 id in the range

bump into the error

$ podman build . Error: creating build container: creating container: creating read-write layer with ID "ca29488a5e29a6e37b058032630d850e6ebda58f80b997fd76891f80582c2527": lchown /home/user/.local/share/containers/storage/vfs/dir/ca29488a5e29a6e37b058032630d850e6ebda58f80b997fd76891f80582c2527/var/cache/apt/archives/partial: operation not permitted

The solution is to clean up the dir rm -rf /home/user/.local/share/containers

@tolusha I see, thank you for the clarification

github-actions · 2025-09-08T07:32:09Z

Pull Request images published ✨

UDI: quay.io/devfile/universal-developer-image:pr-232

…er namespace Signed-off-by: Anatolii Bazko <[email protected]>

github-actions · 2025-09-08T08:01:15Z

Pull Request images published ✨

UDI: quay.io/devfile/universal-developer-image:pr-232

Signed-off-by: Anatolii Bazko <[email protected]>

github-actions · 2025-09-08T15:25:42Z

Pull Request images published ✨

UDI: quay.io/devfile/universal-developer-image:pr-232

openshift-ci · 2025-09-11T21:16:21Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dkwon17, rohanKanojia, tolusha

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tolusha requested review from SDawley, dkwon17 and svor as code owners September 2, 2025 12:57

tolusha requested a review from ibuziuk September 2, 2025 12:58

rohanKanojia approved these changes Sep 2, 2025

View reviewed changes

openshift-ci bot assigned rohanKanojia Sep 2, 2025

openshift-ci bot added the lgtm label Sep 2, 2025

rohanKanojia removed their assignment Sep 2, 2025

tolusha requested a review from l0rd September 3, 2025 06:53

dkwon17 reviewed Sep 3, 2025

View reviewed changes

tolusha marked this pull request as draft September 4, 2025 10:41

openshift-ci bot added do-not-merge/work-in-progress and removed lgtm labels Sep 4, 2025

dkwon17 reviewed Sep 4, 2025

View reviewed changes

base/ubi9/entrypoint.sh Outdated Show resolved Hide resolved

tolusha marked this pull request as ready for review September 5, 2025 11:37

openshift-ci bot removed the do-not-merge/work-in-progress label Sep 5, 2025

dkwon17 reviewed Sep 6, 2025

View reviewed changes

feat: Update /etc/sub*id files in order to run podman command in a us…

9dea356

…er namespace Signed-off-by: Anatolii Bazko <[email protected]>

tolusha force-pushed the usernamespace branch from 30c2765 to 9dea356 Compare September 8, 2025 07:36

tolusha changed the title ~~feat: Update /etc/sub*id files in order to run podman command in a us…~~ feat: Update /etc/sub*id files to support nested containers Sep 8, 2025

fixup

ddd75f4

Signed-off-by: Anatolii Bazko <[email protected]>

dkwon17 approved these changes Sep 11, 2025

View reviewed changes

openshift-ci bot assigned dkwon17 Sep 11, 2025

openshift-ci bot added the lgtm label Sep 11, 2025

tolusha merged commit ac63287 into main Sep 12, 2025
8 checks passed

tolusha deleted the usernamespace branch September 12, 2025 08:34

feat: Update /etc/sub*id files to support nested containers #232

feat: Update /etc/sub*id files to support nested containers #232

Uh oh!

Conversation

tolusha commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference issue

How to test

Uh oh!

tolusha commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 4, 2025

Uh oh!

Uh oh!

github-actions bot commented Sep 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkwon17 Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tolusha Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

openshift-ci bot commented Sep 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tolusha commented Sep 2, 2025 •

edited

Loading

dkwon17 Sep 6, 2025 •

edited

Loading

tolusha Sep 11, 2025 •

edited

Loading