🤔[question] Can you tell me if this setting in the experiment configuration file is not available in a k8s configured environment?

[ShiroZhang]

Describe your question

I configured the private repository authentication information in the configuration file

But it doesn't seem to be working.

[2025-06-13 01:24:50] || INFO: Scheduling Trial 53 (Experiment 53) (id: 53.edb64d32-0574-4130-9288-41eb49f8990c.1)
<info> [2025-06-13 01:24:51] 53.edb64 || INFO: Job det-59f549bc-exp-53-trial-53-attempt-1: Created pod: det-59f549bc-exp-53-trial-53-attempt-1-tcfbp
<info> [2025-06-13 01:24:51] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-1-tcfbp: Pod resources allocated.
<info> [2025-06-13 01:24:51] || INFO: Trial 53 (Experiment 53) was assigned to an agent
<info> [2025-06-13 01:24:56] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-1-tcfbp: Pulling image "192.168.100.200/xiatian/wangmou:latest"
<info> [2025-06-13 01:24:57] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-1-tcfbp: Failed to pull image "192.168.100.200/xiatian/wangmou:latest": Error response from daemon: unauthorized: unauthorized to access repository: xiatian/wangmou, action: pull: unauthorized to access repository: xiatian/wangmou, action: pull
<info> [2025-06-13 01:24:57] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-1-tcfbp: Error: ErrImagePull
<info> [2025-06-13 01:24:57] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-1-tcfbp: Back-off pulling image "192.168.100.200/xiatian/wangmou:latest"
<info> [2025-06-13 01:24:57] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-1-tcfbp: Error: ImagePullBackOff
<info> [2025-06-13 01:24:57] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-1-tcfbp: Back-off pulling image "192.168.100.200/xiatian/wangmou:latest"
<info> [2025-06-13 01:24:57] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-1-tcfbp: Error: ImagePullBackOff
<info> [2025-06-13 01:24:58] 53.edb64 || INFO: Job det-59f549bc-exp-53-trial-53-attempt-2: Created pod: det-59f549bc-exp-53-trial-53-attempt-2-rb97q
<info> [2025-06-13 01:24:58] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-2-rb97q: Pod resources allocated.
<error> [2025-06-13 01:24:58] 53.edb64 || ERROR: crashed: resources failed with non-zero exit code: job was stuck due to unrecoverable image pull errors
<error> [2025-06-13 01:24:58] || ERROR: Trial 53 (Experiment 53) was terminated: allocation failed: resources failed with non-zero exit code: job was stuck due to unrecoverable image pull errors
<info> [2025-06-13 01:24:58] || INFO: Scheduling Trial 53 (Experiment 53) (id: 53.edb64d32-0574-4130-9288-41eb49f8990c.2)
<info> [2025-06-13 01:24:58] || INFO: Trial 53 (Experiment 53) was assigned to an agent
<info> [2025-06-13 01:25:07] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-2-rb97q: Pulling image "192.168.100.200/xiatian/wangmou:latest"
<info> [2025-06-13 01:25:07] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-2-rb97q: Failed to pull image "192.168.100.200/xiatian/wangmou:latest": Error response from daemon: unauthorized: unauthorized to access repository: xiatian/wangmou, action: pull: unauthorized to access repository: xiatian/wangmou, action: pull
<info> [2025-06-13 01:25:07] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-2-rb97q: Error: ErrImagePull
<info> [2025-06-13 01:25:07] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-2-rb97q: Back-off pulling image "192.168.100.200/xiatian/wangmou:latest"
<info> [2025-06-13 01:25:07] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-2-rb97q: Error: ImagePullBackOff
<info> [2025-06-13 01:25:07] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-2-rb97q: Back-off pulling image "192.168.100.200/xiatian/wangmou:latest"
<info> [2025-06-13 01:25:07] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-2-rb97q: Error: ImagePullBackOff
<error> [2025-06-13 01:25:08] 53.edb64 || ERROR: crashed: resources failed with non-zero exit code: job was stuck due to unrecoverable image pull errors
<error> [2025-06-13 01:25:08] || ERROR: Trial 53 (Experiment 53) was terminated: allocation failed: resources failed with non-zero exit code: job was stuck due to unrecoverable image pull errors
<info> [2025-06-13 01:25:08] || INFO: Scheduling Trial 53 (Experiment 53) (id: 53.edb64d32-0574-4130-9288-41eb49f8990c.3)
<info> [2025-06-13 01:25:09] 53.edb64 || INFO: Job det-59f549bc-exp-53-trial-53-attempt-3: Created pod: det-59f549bc-exp-53-trial-53-attempt-3-wvtmz
<info> [2025-06-13 01:25:09] 53.edb64 || INFO: Pod det-59f549bc-exp-53-trial-53-attempt-3-wvtmz: Pod resources allocated.
<info> [2025-06-13 01:25:09] || INFO: Trial 53 (Experiment 53) was assigned to an agent
<info> [2025-06-13 01:25:23] || INFO: forcibly killing allocation's remaining resources (reason: user requested kill)
<info> [2025-06-13 01:25:23] 53.edb64 || INFO: killed: resources failed with non-zero exit code: killed
<info> [2025-06-13 01:25:24] || INFO: Trial 53 (Experiment 53) was terminated: allocation killed after all resources exited: resources were killed

Checklist

• Did you search the docs for a solution?
• Did you search github issues to find if somebody asked this question before?

[ioga]

on k8s, use the native pull secrets: https://docs.determined.ai/latest/setup-cluster/k8s/install-on-kubernetes.html#image-pull-secret-configuration

[ShiroZhang]

on k8s, use the native pull secrets: https://docs.determined.ai/latest/setup-cluster/k8s/install-on-kubernetes.html#image-pull-secret-configuration

I meant the pulling issue of image in the experiment, not in the deployment of the determined

[ioga]

ah, right, probably you have to do the pod spec override https://docs.determined.ai/latest/setup-cluster/k8s/custom-pod-specs.html#supported

[ShiroZhang]

ah, right, probably you have to do the pod spec override https://docs.determined.ai/latest/setup-cluster/k8s/custom-pod-specs.html#supported

ok, thanks for u reply