MLperf script fails to build proper container to run test, offline and Server scenario

[skm123]

Ran below script. Docker container built but failed to install dependency. There is no nvidia-ammo but it's mentioned in requirements.txt

https://docs.mlcommons.org/inference/benchmarks/language/llama2-70b/#__tabbed_1_2

Script
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
--model=llama2-70b-99
--implementation=nvidia
--framework=tensorrt
--category=datacenter
--scenario=Offline
--execution_mode=test
--device=cuda
--docker --quiet
--test_query_count=50
--tp_size=4
--nvidia_llama2_dataset_file_path=/home/nvidia/open_orca/open_orca_gpt4_tokenized_llama.sampled_24576.pkl --rerun

Error
Requirement already satisfied: torch<=2.2.0a in /home/mlcuser/.local/lib/python3.8/site-packages (from -r requirements.txt (line 16)) (2.1.0a0+git32f93b1)
Collecting nvidia-ammo~=0.7.0
Downloading nvidia-ammo-0.7.4.tar.gz (6.9 kB)
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-o3jj2u4v/nvidia-ammo/setup.py'"'"'; file='"'"'/tmp/pip-install-o3jj2u4v/nvidia-ammo/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-o3jj2u4v/nvidia-ammo/pip-egg-info
cwd: /tmp/pip-install-o3jj2u4v/nvidia-ammo/
Complete output (5 lines):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-o3jj2u4v/nvidia-ammo/setup.py", line 90, in
raise RuntimeError("Bad params")
RuntimeError: Bad params
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Traceback (most recent call last):
File "scripts/build_wheel.py", line 332, in
main(**vars(args))
File "scripts/build_wheel.py", line 68, in main
build_run(
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '"/usr/bin/python3" -m pip install -r requirements-dev.txt --extra-index-url https://pypi.ngc.nvidia.com' returned non-zero exit status 1.
make[1]: *** [Makefile.build:307: build_trt_llm] Error 1
make[1]: Leaving directory '/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_6b3f63c5/repo/closed/NVIDIA'
make: *** [/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_6b3f63c5/repo/closed/NVIDIA/Makefile.build:181: build] Error 2
Traceback (most recent call last):
File "/home/mlcuser/.local/bin/mlcr", line 8, in
sys.exit(mlcr())
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 87, in mlcr
main()
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 274, in main
res = method(run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1854, in _run
r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3362, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3554, in _run_deps
r = self.action_object.access(ii)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1638, in _run
r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3362, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3554, in _run_deps
r = self.action_object.access(ii)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 248, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = build-mlperf-inference-server-nvidia, return code = 256)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

#######################################################
Action

Edited requirements.txt (which is created in the container) to exclude #nvidia-ammo but script overwrites and fails again when ran the offline script

mlcuser@0a8c7a0385a8:$ cd /home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_6b3f63c5/repo/closed/NVIDIA/build/TRTLLM
mlcuser@0a8c7a0385a8:/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_6b3f63c5/repo/closed/NVIDIA/build/TRTLLM$ ls -al
total 108
drwxr-xr-x 14 mlcuser mlc 4096 Sep 10 18:03 .
drwxr-xr-x 5 mlcuser mlc 133 Sep 10 17:59 ..
-rw-r--r-- 1 mlcuser mlc 2356 Sep 10 18:02 .clang-format
-rw-r--r-- 1 mlcuser mlc 215 Sep 10 18:02 .dockerignore
drwxr-xr-x 10 mlcuser mlc 4096 Sep 10 18:03 .git
-rw-r--r-- 1 mlcuser mlc 40 Sep 10 18:02 .gitattributes
drwxr-xr-x 3 mlcuser mlc 36 Sep 10 18:02 .github
-rw-r--r-- 1 mlcuser mlc 313 Sep 10 18:02 .gitignore
-rw-r--r-- 1 mlcuser mlc 404 Sep 10 18:02 .gitmodules
-rw-r--r-- 1 mlcuser mlc 1494 Sep 10 18:02 .pre-commit-config.yaml
drwxr-xr-x 6 mlcuser mlc 80 Sep 10 18:02 3rdparty
-rw-r--r-- 1 mlcuser mlc 5646 Sep 10 18:02 CHANGELOG.md
-rw-r--r-- 1 mlcuser mlc 11358 Sep 10 18:00 LICENSE
-rw-r--r-- 1 mlcuser mlc 20362 Sep 10 18:02 README.md
drwxr-xr-x 4 mlcuser mlc 43 Sep 10 18:02 benchmarks
drwxr-xr-x 6 mlcuser mlc 113 Sep 10 18:02 cpp
drwxr-xr-x 3 mlcuser mlc 103 Sep 10 18:02 docker
drwxr-xr-x 3 mlcuser mlc 136 Sep 10 18:02 docs
drwxr-xr-x 33 mlcuser mlc 4096 Sep 10 18:02 examples
-rw-r--r-- 1 mlcuser mlc 261 Sep 10 18:02 requirements-dev-windows.txt
-rw-r--r-- 1 mlcuser mlc 211 Sep 10 18:02 requirements-dev.txt
-rw-r--r-- 1 mlcuser mlc 530 Sep 10 18:02 requirements-windows.txt
-rw-r--r-- 1 mlcuser mlc 447 Sep 10 18:03 requirements.txt
drwxr-xr-x 2 mlcuser mlc 59 Sep 10 18:02 scripts
-rw-r--r-- 1 mlcuser mlc 73 Sep 10 18:02 setup.cfg
-rw-r--r-- 1 mlcuser mlc 4067 Sep 10 18:02 setup.py
drwxr-xr-x 10 mlcuser mlc 4096 Sep 10 18:02 tensorrt_llm
drwxr-xr-x 12 mlcuser mlc 4096 Sep 10 18:02 tests
drwxr-xr-x 4 mlcuser mlc 125 Sep 10 18:02 windows
###############################################
mlcuser@0a8c7a0385a8:/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_6b3f63c5/repo/closed/NVIDIA/build/TRTLLM$ cat requirements.txt
--extra-index-url https://download.pytorch.org/whl/cu121
--extra-index-url https://pypi.nvidia.com
accelerate==0.25.0
build
colored
cuda-python # Do not override the custom version of cuda-python installed in the NGC PyTorch image.
diffusers==0.15.0
lark
mpi4py
numpy
onnx>=1.12.0
polygraphy
psutil
pynvml>=11.5.0
sentencepiece>=0.1.99
torch<=2.2.0a
#nvidia-ammo=0.7.0; platform_machine=="x86_64"
transformers==4.36.1
wheel
optimum
evaluate
janus

How to get the interactive docker ?

[anandhu-eng]

Hi @skm123 , there was an update to the documentation being rolled out yesterday. Could you please retry with the commands specified here.

[skm123]

Ran this script...almost ran 6-7 hours...Failed# also no container build #################

mlcr run-mlperf,inference,_find-performance,_full,_r5.1-dev
--model=llama2-70b-99
--implementation=nvidia
--framework=tensorrt
--category=datacenter
--scenario=Offline
--execution_mode=test
--device=cuda
--docker --quiet
--test_query_count=50
--tp_size=2 --rerun

ERROR: failed to build: failed to solve: process "/bin/bash -c apt-get update && apt-get install -y sudo && adduser ${USER_NAME} sudo && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && apt-get clean && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 12
make: *** [Makefile:122: devel_run] Error 1
make: Leaving directory '/home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322/repo/docker'
Traceback (most recent call last):
File "/home/nvidia/.local/bin/mlcr", line 7, in
sys.exit(mlcr())
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/main.py", line 87, in mlcr
main()
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/main.py", line 274, in main
res = method(run_args)
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/nvidia/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
File "/home/nvidia/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1778, in _run
r = customize_code.preprocess(ii)
File "/home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 285, in preprocess
r = mlc.access(ii)
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/script_action.py", line 290, in docker
return self.call_script_module_function("docker", run_args)
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/script_action.py", line 232, in call_script_module_function
result = automation_instance.docker(run_args) # Pass args to the run method
File "/home/nvidia/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 4573, in docker
return docker_run(self, i)
File "/home/nvidia/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py", line 346, in docker_run
r = self_module._run_deps(
File "/home/nvidia/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3552, in _run_deps
r = self.action_object.access(ii)
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
File "/home/nvidia/.local/lib/python3.10/site-packages/mlc/script_action.py", line 248, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-ml-model-llama2, return code = 256)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

[anandhu-eng]

ERROR: failed to build: failed to solve: process "/bin/bash -c apt-get update && apt-get install -y sudo && adduser ${USER_NAME} sudo && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && apt-get clean && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 12

Hi @skm123 , the actual error could be above this line, could u share a bit more log(logs before this line) to debug.

Also just to confirm whether you are using the latest mlperf-automation code, could you try to do mlc pull repo?

[skm123]

[2025-09-12 19:18:09,986 customize.py:403 INFO] - mlc pull repo && mlcr --tags=app,mlperf,inference,generic,_nvidia,_llama2-70b-99,_tensorrt,_cuda,_test,_r5.1-dev_default,_offline --quiet=true --env.MLC_QUIET=yes --env.MLC_HOST_PLATFORM_FLAVOR=x86_64 --env.MLC_HOST_OS_TYPE=linux --env.MLC_MLPERF_IMPLEMENTATION=nvidia --env.MLC_MLPERF_MODEL=llama2-70b-99 --env.MLC_MLPERF_DEVICE=cuda --env.MLC_MLPERF_LOADGEN_SCENARIO=Offline --env.MLC_MLPERF_RUN_STYLE=test --env.MLC_MLPERF_SKIP_SUBMISSION_GENERATION=False --env.MLC_DOCKER_PRIVILEGED_MODE=True --env.MLC_MLPERF_SUBMISSION_DIVISION=open --env.MLC_MLPERF_INFERENCE_TP_SIZE=1 --env.MLC_MLPERF_SUBMISSION_SYSTEM_TYPE=datacenter --env.MLC_MLPERF_USE_DOCKER=True --env.MLC_MLPERF_BACKEND=tensorrt --env.MLC_RERUN=True --env.MLC_TEST_QUERY_COUNT=50 --env.MLC_NVIDIA_TP_SIZE=2 --env.MLC_MLPERF_FIND_PERFORMANCE_MODE=yes --env.MLC_MLPERF_LOADGEN_ALL_MODES=no --env.MLC_MLPERF_LOADGEN_MODE=performance --env.MLC_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.MLC_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.MLC_MLPERF_INFERENCE_VERSION=5.1-dev --env.MLC_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r5.1-dev_default --env.MLC_MLPERF_SUBMISSION_CHECKER_VERSION=v5.1 --env.MLC_MLPERF_INFERENCE_SOURCE_VERSION=5.1.0 --env.MLC_MLPERF_LAST_RELEASE=v5.1 --env.MLC_MLPERF_INFERENCE_RESULTS_VERSION=r5.1-dev --env.MLC_MODEL=llama2-70b-99 --env.MLC_MLPERF_LOADGEN_COMPLIANCE=no --env.MLC_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.MLC_MLPERF_LOADGEN_SCENARIOS,=Offline --env.MLC_MLPERF_LOADGEN_MODES,=performance --env.MLC_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.coco2014-dataset.tags=_full --add_deps_recursive.igbh-dataset.tags=_full --add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r5.1-dev --add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r5.1-dev --add_deps_recursive.mlperf-inference-nvidia-scratch-space.tags=_version.r5.1-dev --print_env=False --print_deps=False --dump_version_info=True --quiet

[2025-09-12 19:18:09,986 customize.py:465 INFO] - Dockerfile written at /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/dockerfiles/nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v5.0-cuda12.8-pytorch25.01-ubuntu24.04-x86_64-release.Dockerfile
[2025-09-12 19:18:09,988 docker.py:220 INFO] - Dockerfile generated at /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/dockerfiles/nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v5.0-cuda12.8-pytorch25.01-ubuntu24.04-x86_64-release.Dockerfile
[2025-09-12 19:18:10,170 module.py:574 INFO] - * mlcr get,docker
[2025-09-12 19:18:10,171 module.py:1282 INFO] - ! load /home/nvidia/MLC/repos/local/cache/get-docker_d53da0e6/mlc-cached-state.json
[2025-09-12 19:18:10,190 module.py:574 INFO] - * mlcr get,mlperf,inference,submission,dir,local,_version.r5.1-dev
[2025-09-12 19:18:10,213 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-submission-dir/customize.py
[2025-09-12 19:18:10,232 module.py:2160 INFO] - - cache UID: efc10c92a7ce4aa6
[2025-09-12 19:18:10,262 module.py:574 INFO] - * mlcr get,ml-model,llama2-70b,_nvidia,_fp8,_v5.0,_tp-size.2,_pp-size.1
[2025-09-12 19:18:10,300 module.py:574 INFO] - * mlcr get,nvidia,scratch,space,_version.r5.1-dev
[2025-09-12 19:18:10,323 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_14420982
[2025-09-12 19:18:10,324 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-nvidia-scratch-space/run.sh from tmp-run.sh
[2025-09-12 19:18:10,341 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-nvidia-scratch-space/customize.py
[2025-09-12 19:18:10,359 module.py:2160 INFO] - - cache UID: 1442098214aa444d
[2025-09-12 19:18:10,373 module.py:574 INFO] - * mlcr get,git,repo,_repo.https://github.com/NVIDIA/TensorRT-LLM.git,_sha.2ea17cdad28bed0f30e80eea5b1380726a7c6493,_submodules.3rdparty/NVTX;3rdparty/cutlass;3rdparty/cxxopts;3rdparty/json;3rdparty/pybind11;3rdparty/ucxx;3rdparty/xgrammar
[2025-09-12 19:18:10,402 module.py:574 INFO] - * mlcr detect,os
[2025-09-12 19:18:10,407 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322
[2025-09-12 19:18:10,407 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-12 19:18:10,448 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-12 19:18:10,461 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322
[2025-09-12 19:18:10,461 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-git-repo/run.sh from tmp-run.sh
/home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322
rm -rf repo

---

Current directory: /home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322

Cloning TensorRT-LLM.git from https://github.com/NVIDIA/TensorRT-LLM.git

git clone --recurse-submodules https://github.com/NVIDIA/TensorRT-LLM.git repo

[2025-09-12 19:44:11,712 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-12 19:44:11,727 module.py:4198 INFO] - # Requested paths: /home/nvidia/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda
[2025-09-12 19:44:11,774 module.py:3947 INFO] - * /usr/bin/nvcc
[2025-09-12 19:44:11,775 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-cuda_18eb6070
[2025-09-12 19:44:11,775 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/run.sh from tmp-run.sh
[2025-09-12 19:44:11,789 module.py:5388 INFO] - ! call "detect_version" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/customize.py
[2025-09-12 19:44:11,794 customize.py:107 INFO] - Detected version: 11.5
[2025-09-12 19:44:11,794 module.py:3947 INFO] - * /usr/local/cuda/bin/nvcc
[2025-09-12 19:44:11,794 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-cuda_18eb6070
[2025-09-12 19:44:11,794 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/run.sh from tmp-run.sh
[2025-09-12 19:44:11,807 module.py:5388 INFO] - ! call "detect_version" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/customize.py
[2025-09-12 19:44:11,812 customize.py:107 INFO] - Detected version: 13.0
[2025-09-12 19:44:11,812 module.py:4016 INFO] - Selected 0: /usr/bin/nvcc
[2025-09-12 19:44:11,812 module.py:4263 INFO] - # Found artifact in /usr/bin/nvcc
[2025-09-12 19:44:11,813 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-cuda_18eb6070
[2025-09-12 19:44:11,813 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/run.sh from tmp-run.sh
[2025-09-12 19:44:11,828 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/customize.py
[2025-09-12 19:44:11,832 customize.py:107 INFO] - Detected version: 11.5
[2025-09-12 19:44:11,850 module.py:2160 INFO] - - cache UID: 18eb6070bcea431e
[2025-09-12 19:44:11,850 module.py:2235 INFO] - ENV[CUDA_HOME]: /usr
[2025-09-12 19:44:11,850 module.py:2235 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-12 19:44:11,850 module.py:2235 INFO] - ENV[MLC_CUDA_VERSION]: 11.5
[2025-09-12 19:44:11,850 module.py:2235 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu115
[2025-09-12 19:44:11,851 module.py:2235 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/bin/nvcc
[2025-09-12 19:44:11,863 module.py:574 INFO] - * mlcr get,cuda-devices,_with-pycuda
[2025-09-12 19:44:11,879 module.py:574 INFO] - * mlcr get,cuda,_toolkit
[2025-09-12 19:44:11,910 module.py:574 INFO] - * mlcr detect,os
[2025-09-12 19:44:11,915 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-cuda_8ab6faa9
[2025-09-12 19:44:11,916 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-12 19:44:11,958 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-12 19:44:11,973 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-cuda_8ab6faa9
[2025-09-12 19:44:11,973 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/run.sh from tmp-run.sh
[2025-09-12 19:44:11,987 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/customize.py
[2025-09-12 19:44:11,991 customize.py:107 INFO] - Detected version: 11.5
[2025-09-12 19:44:12,008 module.py:2160 INFO] - - cache UID: 8ab6faa938ee474a

2025-09-12 19:44:12,008 module.py:2235 INFO] - ENV[CUDA_HOME]: /usr
[2025-09-12 19:44:12,008 module.py:2235 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-12 19:44:12,008 module.py:2235 INFO] - ENV[MLC_CUDA_VERSION]: 11.5
[2025-09-12 19:44:12,008 module.py:2235 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu115
[2025-09-12 19:44:12,008 module.py:2235 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/bin/nvcc
[2025-09-12 19:44:12,019 module.py:574 INFO] - * mlcr get,python3
[2025-09-12 19:44:12,020 module.py:1282 INFO] - ! load /home/nvidia/MLC/repos/local/cache/get-python3_c845d4bd/mlc-cached-state.json
[2025-09-12 19:44:12,089 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-12 19:44:12,118 module.py:574 INFO] - * mlcr detect,os
[2025-09-12 19:44:12,123 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_7c45e27c
[2025-09-12 19:44:12,123 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-12 19:44:12,166 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-12 19:44:12,182 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-12 19:44:12,192 module.py:574 INFO] - * mlcr detect,os
[2025-09-12 19:44:12,197 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_7c45e27c
[2025-09-12 19:44:12,197 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-12 19:44:12,237 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-12 19:44:12,250 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_7c45e27c
[2025-09-12 19:44:12,250 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/run.sh from tmp-run.sh
[2025-09-12 19:44:12,603 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/customize.py
[2025-09-12 19:44:12,631 module.py:574 INFO] - * mlcr get,python3
[2025-09-12 19:44:12,633 module.py:1282 INFO] - ! load /home/nvidia/MLC/repos/local/cache/get-python3_c845d4bd/mlc-cached-state.json
[2025-09-12 19:44:12,742 module.py:574 INFO] - * mlcr get,generic-python-lib,_pip
[2025-09-12 19:44:12,771 module.py:574 INFO] - * mlcr detect,os
[2025-09-12 19:44:12,776 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_923b2d5b
[2025-09-12 19:44:12,776 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-12 19:44:12,818 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-12 19:44:12,836 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-12 19:44:12,845 module.py:574 INFO] - * mlcr detect,os
[2025-09-12 19:44:12,850 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_923b2d5b
[2025-09-12 19:44:12,850 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-12 19:44:12,894 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-12 19:44:12,908 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_923b2d5b
[2025-09-12 19:44:12,908 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/run.sh from tmp-run.sh
[2025-09-12 19:44:13,220 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/customize.py
[2025-09-12 19:44:13,249 module.py:574 INFO] - * mlcr get,python3
[2025-09-12 19:44:13,251 module.py:1282 INFO] - ! load /home/nvidia/MLC/repos/local/cache/get-python3_c845d4bd/mlc-cached-state

Downloading: bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) -d '/home/nvidia/MLC/repos/local/cache/download-file_50fe9635/llama-2-70b-chat-hf.uri' https://llama2.mlcommons-storage.org/metadata/llama-2-70b-chat-hf.uri
Checking if authentication is required...
Authentication required
Checking dependencies...
Logging in to Cloudflare Access at https://llama2.mlcommons-storage.org/metadata/llama-2-70b-chat-hf.uri...
Get tokens from edge false
A browser window should have opened at the following URL:

https://llama2.mlcommons-storage.org/cdn-cgi/access/cli?aud=2c232a285ef6d844dcaaa90dc7569f432fa890bfc956c2dc4cd2f3f0016768b7&edge_token_transfer=true&redirect_url=https%3A%2F%2Fllama2.mlcommons-storage.org%2Fmetadata%2Fllama-2-70b-chat-hf.uri%3Faud%3D2c232a285ef6d844dcaaa90dc7569f432fa890bfc956c2dc4cd2f3f0016768b7%26token%3DeWVqODZxOpsi6FujaX_JT1_TbzCu_ppvLWTpS4dlxRo%253D&send_org_token=true&token=eWVqODZxOpsi6FujaX_JT1_TbzCu_ppvLWTpS4dlxRo%3D

If the browser failed to open, please visit the URL above directly in your browser.

Authentication successful!
Checking token expiration time...
Cloudflare Access token will expire in 13 days, 23 hours
Downloading dataset information...
Downloading checksums file...
Preparing download...
Dataset: llama-2-70b-chat-hf
Download directory: /home/nvidia/MLC/repos/local/cache/download-file_50fe9635/llama-2-70b-chat-hf.uri
Creating download directory: /home/nvidia/MLC/repos/local/cache/download-file_50fe9635/llama-2-70b-chat-hf.uri
Preparing file list for download...
Starting download of dataset files...
--2025-09-12 19:46:56-- https://llama2.mlcommons-storage.org/data/Llama-2-70b-chat-hf/.git/info/exclude
Resolving llama2.mlcommons-storage.org (llama2.mlcommons-storage.org)... 104.26.0.198, 172.67.74.32, 104.26.1.198, ...
Connecting to llama2.mlcommons-storage.org (llama2.mlcommons-storage.org)|104.26.0.198|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 240 [application/octet-stream]
Saving to: ?/home/nvidia/MLC/repos/local/cache/download-file_50fe9635/llama-2-70b-chat-hf.uri/.git/info/exclude?

.git/info/exclude 100%[================================================================>] 240 --.-KB/s in 0s

2025-09-12 19:46:56 (209 MB/s) - ?/home/nvidia/MLC/repos/local/cache/download-file_50fe9635/llama-2-70b-chat-hf.uri/.git/info/exclude? saved [240/240]

--2025-09-12 19:46:56-- https://llama2.mlcommons-storage.org/data/Llama-2-70b-chat-hf/.git/lfs/objects/95/0a/950a4319f306b82b46903dcb5df2c4b5eb2d7047f789577f2af16f287ded196e
Reusing existing connection to llama2.mlcommons-storage.org:443.

.git/objects/34/dc203315838fc10bb6 100%[================================================================>] 127 --.-KB/s in 0s

2025-09-13 00:47:05 (84.8 MB/s) - ?/home/nvidia/MLC/repos/local/cache/download-file_50fe9635/llama-2-70b-chat-hf.uri/.git/objects/34/dc203315838fc10bb64376f3de2498c96c01b1? saved [127/127]

FINISHED --2025-09-13 00:47:05--
Total wall clock time: 5h 0m 9s
Downloaded: 198 files, 504G in 4h 58m 30s (28.8 MB/s)
Download completed successfully!
Verifying file integrity with MD5 checksums...
.git/info/exclude: OK
.git/lfs/objects/95/0a/950a4319f306b82b46903dcb5df2c4b5eb2d7047f789577f2af16f287ded196e: OK
pytorch_model-00009-of-00015.bin: OK
.git/objects/63/cb98879e8ed61f94a78e9ddab3a1bb3a82b38e: OK
.git/hooks/pre-applypatch.sample: OK
.git/objects/12/387d233f43391cc7ca6f332cc64c6673c300b8: OK
.git/objects/46/a11a504f7213b0fc67dfa967cbe5a5e7cd7f90: OK
.git/lfs/objects/eb/3a/eb3ab399414e25969934411cc77ef89e44248c26ef847b6a9cb391107ffe52a0: OK
model-00007-of-00015.safetensors: OK
.git/objects/6e/8a3ba959f3cd1d343d125caff8b0a3627288c5: OK
.git/lfs/objects/83/56/8356d772c4e6aede9b4102c338dcf9fb7c094a23c1bb8935ee13e9b8939e15c1: OK
pytorch_model-00002-of-00015.bin: OK
.git/objects/a6/e931b92caff4c79c5c56282f1e89569a0ae558: OK
.git/objects/ab/6d5f3786364cb2955fe139a1259fe567cf0620: OK
.git/objects/9e/72adb60a1e9065645bd61fb8fa22d3423037b5: OK
.git/objects/16/c1325554bed934d42c0a0f6adef4a55c98fff6: OK
.git/objects/a0/024735c8dd7afe47fe72792b2c4edaff63bd3b: OK
config.json: OK
.git/objects/70/94c658d83a4bc9f43ac12d2a800c8901443968: OK
.git/objects/ab/bcc199b2d1e4feb5d7e40c0bd67e1b0ce29e97: OK
.git/objects/47/2c0c528caff9210b3fe8bc4f895f691b92f4de: OK

odel-00015-of-00015.safetensors: OK
model.safetensors.index.json: OK
.git/objects/2c/1046f5680a9b53e5ca76dfbcbe54ee56954ee6: OK
.git/lfs/objects/a8/67/a867df6fd610652b39ed5d8b5e62d7b44ef26590caf76ecb7503e9768ba3a23c: OK
pytorch_model-00014-of-00015.bin: OK
.git/objects/34/dc203315838fc10bb64376f3de2498c96c01b1: OK
Checksum verification completed successfully!
All files have been downloaded and verified in: /home/nvidia/MLC/repos/local/cache/download-file_50fe9635/llama-2-70b-chat-hf.uri
[2025-09-13 01:00:56,280 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/download-file/customize.py
[2025-09-13 01:00:56,310 module.py:2160 INFO] - - cache UID: 50fe9635c4d14234
[2025-09-13 01:00:56,311 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/download-and-extract/customize.py
[2025-09-13 01:00:56,342 module.py:2160 INFO] - - cache UID: 1e93d823282347b2
[2025-09-13 01:00:56,344 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-ml-model-llama2_19bdd318
[2025-09-13 01:00:56,344 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-ml-model-llama2/run.sh from tmp-run.sh

[2025-09-13 01:00:56,353 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-ml-model-llama2/customize.py
[2025-09-13 01:00:56,381 module.py:2160 INFO] - - cache UID: 19bdd31869f44f1a
[2025-09-13 01:00:56,382 module.py:2235 INFO] - LLAMA2 checkpoint path: /home/nvidia/MLC/repos/local/cache/download-file_50fe9635/llama-2-70b-chat-hf.uri
[2025-09-13 01:00:56,403 module.py:574 INFO] - * mlcr get,nvidia,inference,common-code,_mlcommons,_v5.1-dev
[2025-09-13 01:00:56,451 module.py:574 INFO] - * mlcr get,mlperf,inference,results,official,_mlcommons,_code-only-for-v5.1,_v5.1-dev
[2025-09-13 01:00:56,498 module.py:574 INFO] - * mlcr get,git,repo,_branch.mlc-code-only-for-v5.1,_repo.https://github.com/mlcommons/inference_results_v5.0.git
[2025-09-13 01:00:56,527 module.py:574 INFO] - * mlcr detect,os
[2025-09-13 01:00:56,532 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_3fd84dac
[2025-09-13 01:00:56,533 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-13 01:00:56,571 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-13 01:00:56,586 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_3fd84dac
[2025-09-13 01:00:56,586 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-git-repo/run.sh from tmp-run.sh
/home/nvidia/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_3fd84dac
rm -rf repo

---

Current directory: /home/nvidia/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_3fd84dac

Cloning inference_results_v5.0.git from https://github.com/mlcommons/inference_results_v5.0.git

git clone -b mlc-code-only-for-v5.1 https://github.com/mlcommons/inference_results_v5.0.git --depth 5 repo

Cloning into 'repo'...
remote: Enumerating objects: 103123, done.
remote: Counting objects: 100% (103123/103123), done.
remote: Compressing objects: 100% (43490/43490), done.
remote: Total 103123 (delta 51212), reused 102887 (delta 51083), pack-reused 0 (from 0)
Receiving objects: 100% (103123/103123), 515.30 MiB | 39.07 MiB/s, done.
Resolving deltas: 100% (51212/51212), done.
Updating files: 100% (321239/321239), done.
git rev-parse HEAD >> ../tmp-mlc-git-hash.out
[2025-09-13 01:01:53,848 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-git-repo/customize.py
[2025-09-13 01:01:53,875 module.py:2160 INFO] - - cache UID: 3fd84daca6e2465a
[2025-09-13 01:01:53,875 module.py:2235 INFO] - MLC cache path to the Git repo: /home/nvidia/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_3fd84dac/repo
[2025-09-13 01:01:53,877 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-results/customize.py
[2025-09-13 01:01:53,904 module.py:2160 INFO] - - cache UID: 6f3db2206fb04938
[2025-09-13 01:01:53,912 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-nvidia-common-code/customize.py
[2025-09-13 01:01:53,938 module.py:2160 INFO] - - cache UID: 500523e410204006
[2025-09-13 01:01:53,964 module.py:574 INFO] - * mlcr get,preprocessed,dataset,openorca,_calibration,_mlc,_nvidia
[2025-09-13 01:01:54,006 module.py:574 INFO] - * mlcr get,python3
[2025-09-13 01:01:54,007 module.py:1282 INFO] - ! load /home/nvidia/MLC/repos/local/cache/get-python3_c845d4bd/mlc-cached-state.json
[2025-09-13 01:01:54,078 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pyarrow
[2025-09-13 01:01:54,106 module.py:574 INFO] - * mlcr detect,os
[2025-09-13 01:01:54,111 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_b55acf9f
[2025-09-13 01:01:54,111 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-13 01:01:54,152 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-13 01:01:54,180 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-13 01:01:54,189 module.py:574 INFO] - * mlcr detect,os
[2025-09-13 01:01:54,194 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_b55acf9f
[2025-09-13 01:01:54,194 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-09-13 01:01:54,233 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-09-13 01:01:54,247 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_b55acf9f
[2025-09-13 01:01:54,247 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/run.sh from tmp-run.sh
[2025-09-13 01:01:54,582 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/customize.py
[2025-09-13 01:01:54,610 module.py:574 INFO] - * mlcr get,python3
[2025-09-13 01:01:54,612 module.py:1282 INFO] - ! load /home/nvidia/MLC/repos/local/cache/get-python3_c845d4bd/mlc-cached-state.json
[2025-09-13 01:01:54,724 module.py:574 INFO] - * mlcr get,generic-python-lib,_pip

ons/script/get-generic-python-lib/customize.py
[2025-09-13 01:01:58,300 customize.py:152 INFO] - Detected version: 4.0.0
[2025-09-13 01:01:58,300 customize.py:117 INFO] - Extra PIP CMD: --break-system-packages
[2025-09-13 01:01:58,301 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_715fb39e
[2025-09-13 01:01:58,301 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh from tmp-run.sh

/usr/bin/python3 -m pip install "datasets<=2.19.2" --break-system-packages
Defaulting to user installation because normal site-packages is not writeable
Collecting datasets<=2.19.2
Downloading datasets-2.19.2-py3-none-any.whl.metadata (19 kB)
Requirement already satisfied: filelock in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (3.19.1)
Requirement already satisfied: numpy>=1.17 in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (1.24.4)
Requirement already satisfied: pyarrow>=12.0.0 in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (21.0.0)
Collecting pyarrow-hotfix (from datasets<=2.19.2)
Downloading pyarrow_hotfix-0.7-py3-none-any.whl.metadata (3.6 kB)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (0.3.8)
Requirement already satisfied: pandas in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (2.3.2)
Requirement already satisfied: requests>=2.32.1 in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (2.32.5)
Requirement already satisfied: tqdm>=4.62.1 in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (4.67.1)
Requirement already satisfied: xxhash in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (3.5.0)
Requirement already satisfied: multiprocess in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (0.70.16)
Collecting fsspec<=2024.3.1,>=2023.1.0 (from fsspec[http]<=2024.3.1,>=2023.1.0->datasets<=2.19.2)
Downloading fsspec-2024.3.1-py3-none-any.whl.metadata (6.8 kB)
Requirement already satisfied: aiohttp in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (3.12.15)
Requirement already satisfied: huggingface-hub>=0.21.2 in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (0.34.4)
Requirement already satisfied: packaging in /home/nvidia/.local/lib/python3.10/site-packages (from datasets<=2.19.2) (25.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3/dist-packages (from datasets<=2.19.2) (5.4.1)
Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /home/nvidia/.local/lib/python3.10/site-packages (from aiohttp->datasets<=2.19.2) (2.6.1)
Requirement already satisfied: aiosignal>=1.4.0 in /home/nvidia/.local/lib/python3.10/site-packages (from aiohttp->datasets<=2.19.2) (1.4.0)
Requirement already satisfied: async-timeout<6.0,>=4.0 in /home/nvidia/.local/lib/python3.10/site-packages (from aiohttp->datasets<=2.19.2) (5.0.1)
Requirement already satisfied: attrs>=17.3.0 in /home/nvidia/.local/lib/python3.10/site-packages (from aiohttp->datasets<=2.19.2) (25.3.0)
Requirement already satisfied: frozenlist>=1.1.1 in /home/nvidia/.local/lib/python3.10/site-packages (from aiohttp->datasets<=2.19.2) (1.7.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/nvidia/.local/lib/python3.10/site-packages (from aiohttp->datasets<=2.19.2) (6.6.4)
Requirement already satisfied: propcache>=0.2.0 in /home/nvidia/.local/lib/python3.10/site-packages (from aiohttp->datasets<=2.19.2) (0.3.2)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /home/nvidia/.local/lib/python3.10/site-packages (from aiohttp->datasets<=2.19.2) (1.20.1)
Requirement already satisfied: typing-extensions>=4.1.0 in /usr/local/lib/python3.10/dist-packages (from multidict<7.0,>=4.5->aiohttp->datasets<=2.19.2) (4.14.1)
Requirement already satisfied: idna>=2.0 in /usr/lib/python3/dist-packages (from yarl<2.0,>=1.17.0->aiohttp->datasets<=2.19.2) (3.3)
Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /home/nvidia/.local/lib/python3.10/site-packages (from huggingface-hub>=0.21.2->datasets<=2.19.2) (1.1.8)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/nvidia/.local/lib/python3.10/site-packages (from requests>=2.32.1->datasets<=2.19.2) (3.4.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.32.1->datasets<=2.19.2) (2020.6.20)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/nvidia/.local/lib/python3.10/site-packages (from pandas->datasets<=2.19.2) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3/dist-packages (from pandas->datasets<=2.19.2) (2022.1)
Requirement already satisfied: tzdata>=2022.7 in /home/nvidia/.local/lib/python3.10/site-packages (from pandas->datasets<=2.19.2) (2025.2)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas->datasets<=2.19.2) (1.16.0)
Downloading datasets-2.19.2-py3-none-any.whl (542 kB)
???????????????????????????????????????? 542.1/542.1 kB 30.8 MB/s 0:00:00
Downloading fsspec-2024.3.1-py3-none-any.whl (171 kB)
Downloading pyarrow_hotfix-0.7-py3-none-any.whl (7.9 kB)
Installing collected packages: pyarrow-hotfix, fsspec, datasets
Attempting uninstall: fsspec
Found existing installation: fsspec 2025.3.0
Uninstalling fsspec-2025.3.0:
Successfully uninstalled fsspec-2025.3.0
Attempting uninstall: datasets
Found existing installation: datasets 4.0.0
Uninstalling datasets-4.0.0:
Successfully uninstalled datasets-4.0.0
Successfully installed datasets-2.19.2 fsspec-2024.3.1 pyarrow-hotfix-0.7
[2025-09-13 01:02:00,862 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-generic-python-lib_715fb39e
[2025-09-13 01:02:00,862 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh from tmp-run.sh
[2025-09-13 01:02:00,953 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-13 01:02:00,965 customize.py:152 INFO] - Detected version: 2.19.2
[2025-09-13 01:02:00,995 module.py:2160 INFO] - - cache UID: 715fb39e8a224072
[2025-09-13 01:02:01,088 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-13 01:02:01,104 module.py:574 INFO] - * mlcr get,python3
[2025-09-13 01:02:01,105 module.py:1282 INFO] - ! load /home/nvidia/MLC/repos/local/cache/get-python3_c845d4bd/mlc-cached-state.json
[2025-09-13 01:02:01,106 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-preprocessed-dataset-openorca_a50ada89
[2025-09-13 01:02:01,106 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh

/home/nvidia/MLC/repos/local/cache/get-rclone_ad27db02:/home/nvidia/MLC/repos/local/cache/get-rclone_ad27db02:/home/nvidia/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
[2025-09-13 01:02:13,716 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-rclone/customize.py
[2025-09-13 01:02:13,728 customize.py:95 INFO] - Detected version: 1.65.2
[2025-09-13 01:02:13,759 module.py:2160 INFO] - - cache UID: ad27db024b874939
[2025-09-13 01:02:13,781 module.py:574 INFO] - * mlcr get,rclone-config,_mlc-inference
[2025-09-13 01:02:13,798 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/download-file_openorca-prepro_b8df758c
[2025-09-13 01:02:13,798 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-rclone-config/run.sh from tmp-run.sh
Running:
rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com

[mlc-inference]
type = s3
secret_access_key = fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b
endpoint = https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com
provider = Cloudflare
access_key_id = f65ba5eef400db161ea49967de89f47b
[2025-09-13 01:02:13,919 module.py:5388 INFO] - ! call "postprocess" from /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-rclone-config/customize.py

[2025-09-13 01:02:27,027 module.py:2160 INFO] - - cache UID: a50ada890fdb4930
[2025-09-13 01:02:27,050 module.py:574 INFO] - * mlcr get,python3
[2025-09-13 01:02:27,053 module.py:1282 INFO] - ! load /home/nvidia/MLC/repos/local/cache/get-python3_c845d4bd/mlc-cached-state.json
[2025-09-13 01:02:27,068 module.py:5242 INFO] - ! cd /home/nvidia/MLC/repos/local/cache/get-ml-model-llama2_e9c83175
[2025-09-13 01:02:27,068 module.py:5243 INFO] - ! call /home/nvidia/MLC/repos/mlcommons@mlperf-automations/script/get-ml-model-llama2/run-nvidia.sh from tmp-run.sh
Set tp size is 2
Set pp size is 1
cd /home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322/repo
make: Entering directory '/home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322/repo/docker'
Building docker image: tensorrt_llm/devel:latest
DOCKER_BUILDKIT=1 docker build --pull
--progress auto
--build-arg BASE_IMAGE=nvcr.io/nvidia/pytorch
--build-arg BASE_TAG=25.01-py3
--build-arg BUILD_WHEEL_ARGS="--clean --trt_root /usr/local/tensorrt --python_bindings --benchmarks"
--build-arg TORCH_INSTALL_TYPE="skip"





--build-arg TRT_LLM_VER="0.18.0.dev2025021800"

--build-arg GIT_COMMIT="2ea17cdad28bed0f30e80eea5b1380726a7c6493"


--target devel
--file Dockerfile.multi
--tag tensorrt_llm/devel:latest
..
[+] Building 604.3s (25/25) FINISHED docker:default
=> [internal] load build definition from Dockerfile.multi 0.0s
=> => transferring dockerfile: 4.29kB 0.0s
=> WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 6) 0.0s
=> WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) 0.0s

=> WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 62) 0.0s
=> WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 80) 0.0s
=> [internal] load metadata for nvcr.io/nvidia/pytorch:25.01-py3 0.9s
=> [auth] nvidia/pytorch:pull token for nvcr.io 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 257B 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 15.90kB 0.0s
=> [base 1/2] FROM nvcr.io/nvidia/pytorch:25.01-py3@sha256:96990c82825613c3bdeebb66675c7c91b0123f64a5895623316dc5b824e0d7a9 0.1s
=> => resolve nvcr.io/nvidia/pytorch:25.01-py3@sha256:96990c82825613c3bdeebb66675c7c91b0123f64a5895623316dc5b824e0d7a9 0.0s
=> => sha256:0981807f1a51a156563e28b59dc2e7a9b5c1c7d85d1169d4965c5fd91fa38bcb 11.94kB / 11.94kB 0.0s
=> => sha256:96990c82825613c3bdeebb66675c7c91b0123f64a5895623316dc5b824e0d7a9 745B / 745B 0.0s
=> => sha256:e28aab52093414d43498c6b86d0d327752447b431e78ac9b53acba2f7712661c 53.38kB / 53.38kB 0.0s
=> [base 2/2] RUN echo "Using GitHub mirror: " 0.3s
=> [devel 1/17] RUN echo "Using Python version: 3.12.3" 0.8s
=> [devel 2/17] COPY docker/common/install_base.sh install_base.sh 0.1s
=> [devel 3/17] RUN bash ./install_base.sh 3.12.3 && rm install_base.sh 18.4s
=> [devel 4/17] COPY docker/common/install_cmake.sh install_cmake.sh 0.0s
=> [devel 5/17] RUN bash ./install_cmake.sh && rm install_cmake.sh 3.4s
=> [devel 6/17] COPY docker/common/install_ccache.sh install_ccache.sh 0.0s
=> [devel 7/17] RUN bash ./install_ccache.sh && rm install_ccache.sh 0.6s
=> [devel 8/17] COPY docker/common/install_cuda_toolkit.sh install_cuda_toolkit.sh 0.1s
=> [devel 9/17] RUN bash ./install_cuda_toolkit.sh && rm install_cuda_toolkit.sh 0.9s
=> [devel 10/17] COPY docker/common/install_tensorrt.sh install_tensorrt.sh 0.1s
=> [devel 11/17] RUN bash ./install_tensorrt.sh --TRT_VER=${TRT_VER} --CUDA_VER=${CUDA_VER} --CUDNN_VER=${CUDNN_VER} 413.2s
=> [devel 12/17] COPY docker/common/install_polygraphy.sh install_polygraphy.sh 0.0s
=> [devel 13/17] RUN bash ./install_polygraphy.sh && rm install_polygraphy.sh 2.7s
=> [devel 14/17] COPY docker/common/install_mpi4py.sh install_mpi4py.sh 0.0s
=> [devel 15/17] RUN bash ./install_mpi4py.sh && rm install_mpi4py.sh 74.3s
=> [devel 16/17] COPY docker/common/install_pytorch.sh install_pytorch.sh 0.0s
=> [devel 17/17] RUN bash ./install_pytorch.sh skip && rm install_pytorch.sh 0.7s
=> exporting to image 87.6s
=> => exporting layers 87.5s
=> => writing image sha256:bf89babbe2740d4c798f0a80227b4a02618330fed4cb73c4c74c403aa0a83d82 0.0s
=> => naming to docker.io/tensorrt_llm/devel:latest 0.0s

4 warnings found (use docker --debug to expand):

• FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 62)
• FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 80)
• FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 6)
• FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17)
  make: Leaving directory '/home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322/repo/docker'
  bash -c 'git lfs install && git lfs pull && python3 scripts/build_wheel.py -a=90 --clean --install --use_ccache --benchmarks --trt_root /usr/local/tensorrt/ && python examples/quantization/quantize.py --dtype=float16 --output_dir=/mnt/models/Llama2/fp8-quantized-ammo/llama2-70b-chat-hf-tp2pp1-fp8 --model_dir=/mnt/models/Llama2/Llama-2-70b-chat-hf --qformat=fp8 --kv_cache_dtype=fp8 --tp_size 2 --pp_size 1 --calib_dataset=/calib_dataset'

make: Leaving directory '/home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322/repo/docker'
bash -c 'git lfs install && git lfs pull && python3 scripts/build_wheel.py -a=90 --clean --install --use_ccache --benchmarks --trt_root /usr/local/tensorrt/ && python examples/quantization/quantize.py --dtype=float16 --output_dir=/mnt/models/Llama2/fp8-quantized-ammo/llama2-70b-chat-hf-tp2pp1-fp8 --model_dir=/mnt/models/Llama2/Llama-2-70b-chat-hf --qformat=fp8 --kv_cache_dtype=fp8 --tp_size 2 --pp_size 1 --calib_dataset=/calib_dataset'
make: Entering directory '/home/nvidia/MLC/repos/local/cache/get-git-repo_tensorrt-llm_6edca322/repo/docker'
docker build --progress auto --build-arg BASE_IMAGE_WITH_TAG=tensorrt_llm/devel:latest --build-arg USER_ID=1000 --build-arg USER_NAME=nvidia --build-arg GROUP_ID=1000 --build-arg GROUP_NAME=nvidia --file Dockerfile.user --tag tensorrt_llm/devel:latest-nvidia ..
[+] Building 10.1s (6/6) FINISHED docker:default
=> [internal] load build definition from Dockerfile.user 0.0s
=> => transferring dockerfile: 638B 0.0s
=> WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 3) 0.0s
=> WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE_WITH_TAG} results in empty or invalid base image name (line 3) 0.0s
=> [internal] load metadata for docker.io/tensorrt_llm/devel:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 257B 0.0s
=> [1/3] FROM docker.io/tensorrt_llm/devel:latest 0.0s
=> [2/3] RUN (getent group 1000 || groupadd --gid 1000 nvidia) && (getent passwd 1000 || useradd --gid 1000 --uid 1000 --create- 0.8s
=> ERROR [3/3] RUN apt-get update && apt-get install -y sudo && adduser nvidia sudo && echo '%sudo ALL=(ALL) NOPASSWD:AL 9.2s

[3/3] RUN apt-get update && apt-get install -y sudo && adduser nvidia sudo && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && apt-get clean && rm -rf /var/lib/apt/lists/*:
1.162 Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease [1581 B]
1.214 Get:2 http://archive.ubuntu.com/ubuntu noble InRelease [256 kB]
1.312 Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 Packages [2006 kB]
1.342 Get:4 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
1.599 Get:5 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
1.764 Get:6 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
2.066 Get:7 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [1135 kB]
2.074 Get:8 http://archive.ubuntu.com/ubuntu noble/restricted amd64 Packages [117 kB]
2.192 Get:9 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [19.3 MB]
2.688 Get:10 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1408 kB]
2.837 Get:11 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Packages [23.0 kB]
2.839 Get:12 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2159 kB]
2.956 Get:13 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages [1808 kB]
3.065 Get:14 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Packages [331 kB]
3.142 Get:15 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1920 kB]
3.253 Get:16 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1761 kB]
3.361 Get:17 http://archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Packages [58.1 kB]
3.433 Get:18 http://archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2331 kB]
3.553 Get:19 http://archive.ubuntu.com/ubuntu noble-backports/main amd64 Packages [48.8 kB]
3.625 Get:20 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [35.6 kB]
4.554 Fetched 35.1 MB in 3s (10.1 MB/s)

4.554 Reading package lists...
5.839 Reading package lists...
7.039 Building dependency tree...
7.284 Reading state information...
7.564 The following additional packages will be installed:
7.565 libapparmor1
7.607 The following NEW packages will be installed:
7.608 libapparmor1 sudo
7.782 0 upgraded, 2 newly installed, 0 to remove and 160 not upgraded.
7.782 Need to get 998 kB of archives.
7.782 After this operation, 3669 kB of additional disk space will be used.
7.782 Get:1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libapparmor1 amd64 4.0.1really4.0.1-0ubuntu0.24.04.4 [50.6 kB]
8.002 Get:2 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 sudo amd64 1.9.15p5-3ubuntu5.24.04.1 [948 kB]
8.621 debconf: unable to initialize frontend: Dialog
8.621 debconf: (TERM is not set, so the dialog frontend is not usable.)
8.621 debconf: falling back to frontend: Readline
8.673 debconf: unable to initialize frontend: Readline
8.673 debconf: (This frontend requires a controlling tty.)
8.673 debconf: falling back to frontend: Teletype
8.725 dpkg-preconfigure: unable to re-open stdin:
8.763 Fetched 998 kB in 1s (1465 kB/s)
8.787 Selecting previously unselected package libapparmor1:amd64.
(Reading database ... 26643 files and directories currently installed.)
8.810 Preparing to unpack .../libapparmor1_4.0.1really4.0.1-0ubuntu0.24.04.4_amd64.deb ...
8.811 Unpacking libapparmor1:amd64 (4.0.1really4.0.1-0ubuntu0.24.04.4) ...
8.836 Selecting previously unselected package sudo.
8.839 Preparing to unpack .../sudo_1.9.15p5-3ubuntu5.24.04.1_amd64.deb ...
8.862 Unpacking sudo (1.9.15p5-3ubuntu5.24.04.1) ...
8.915 Setting up libapparmor1:amd64 (4.0.1really4.0.1-0ubuntu0.24.04.4) ...
8.916 Setting up sudo (1.9.15p5-3ubuntu5.24.04.1) ...
8.954 Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
9.114 fatal: The user `nvidia' does not exist.

4.554 Reading package lists...
5.839 Reading package lists...
7.039 Building dependency tree...
7.284 Reading state information...
7.564 The following additional packages will be installed:
7.565 libapparmor1
7.607 The following NEW packages will be installed:
7.608 libapparmor1 sudo
7.782 0 upgraded, 2 newly installed, 0 to remove and 160 not upgraded.
7.782 Need to get 998 kB of archives.
7.782 After this operation, 3669 kB of additional disk space will be used.
7.782 Get:1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libapparmor1 amd64 4.0.1really4.0.1-0ubuntu0.24.04.4 [50.6 kB]
8.002 Get:2 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 sudo amd64 1.9.15p5-3ubuntu5.24.04.1 [948 kB]
8.621 debconf: unable to initialize frontend: Dialog
8.621 debconf: (TERM is not set, so the dialog frontend is not usable.)
8.621 debconf: falling back to frontend: Readline
8.673 debconf: unable to initialize frontend: Readline
8.673 debconf: (This frontend requires a controlling tty.)
8.673 debconf: falling back to frontend: Teletype
8.725 dpkg-preconfigure: unable to re-open stdin:
8.763 Fetched 998 kB in 1s (1465 kB/s)
8.787 Selecting previously unselected package libapparmor1:amd64.
(Reading database ... 26643 files and directories currently installed.)
8.810 Preparing to unpack .../libapparmor1_4.0.1really4.0.1-0ubuntu0.24.04.4_amd64.deb ...
8.811 Unpacking libapparmor1:amd64 (4.0.1really4.0.1-0ubuntu0.24.04.4) ...
8.836 Selecting previously unselected package sudo.
8.839 Preparing to unpack .../sudo_1.9.15p5-3ubuntu5.24.04.1_amd64.deb ...
8.862 Unpacking sudo (1.9.15p5-3ubuntu5.24.04.1) ...
8.915 Setting up libapparmor1:amd64 (4.0.1really4.0.1-0ubuntu0.24.04.4) ...
8.916 Setting up sudo (1.9.15p5-3ubuntu5.24.04.1) ...
8.954 Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
9.114 fatal: The user `nvidia' does not exist.

[skm123]

$ mlc pull repo
[2025-09-13 03:31:47,081 repo_action.py:297 INFO] - Repository mlperf-automations already exists at /home/nvidia/MLC/repos/mlcommons@mlperf-automations. Checking for local changes...
[2025-09-13 03:31:47,129 repo_action.py:308 INFO] - No local changes detected. Pulling latest changes...
remote: Enumerating objects: 23, done.
remote: Counting objects: 100% (23/23), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 23 (delta 13), reused 15 (delta 9), pack-reused 0 (from 0)
Unpacking objects: 100% (23/23), 18.24 KiB | 1.66 MiB/s, done.
From https://github.com/mlcommons/mlperf-automations
4eca4b8..07065a7 dev -> origin/dev
Updating 4eca4b8..07065a7
Fast-forward
script/app-mlperf-inference-nvidia/customize.py | 62 ++++++++++-----------------------------
script/build-mlperf-inference-server-nvidia/run.sh | 4 +++
script/get-aocc/README.md | 73 +++++++++++++++++++++++++++++++++++++++++++++
script/get-aocc/customize.py | 12 +++++++-
script/get-aocc/meta.yaml | 1 +
script/get-ml-model-llama2/README.md | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
script/get-ml-model-llama2/customize.py | 25 ++++++++++++++--
script/get-ml-model-llama2/meta.yaml | 39 ++++++++++++++++++++-----
script/get-ml-model-llama2/run-nvidia.sh | 10 ++-----
script/get-preprocessed-dataset-openorca/customize.py | 22 ++++++++++----
script/get-preprocessed-dataset-openorca/meta.yaml | 3 ++
script/install-llvm-src/run.sh | 1 -
12 files changed, 305 insertions(+), 72 deletions(-)
create mode 100644 script/get-aocc/README.md
create mode 100644 script/get-ml-model-llama2/README.md
[2025-09-13 03:31:47,593 repo_action.py:310 INFO] - Repository successfully pulled.
[2025-09-13 03:31:47,593 repo_action.py:324 INFO] - Registering the repo in repos.json

[skm123]

Very Close to run the script but it still fails. New Custom Hardware defined.
@anandhu-eng any suggestion??

ubuntu@c2e8c0440998:/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/configs/llama2-70b/Offline$ cd ~
ubuntu@c2e8c0440998:$ mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only
--model=llama2-70b-99
--implementation=nvidia
--framework=tensorrt
--category=datacenter
--scenario=Offline
--execution_mode=valid
--device=cuda
--quiet
--tp_size=2
--env.MLC_DATASET_OPENORCA_PREPROCESSED_PATH="nvidia_llama2_dataset_file_path=/home/ubuntu/MLC/repos/local/cache/download-file_openorca-prepro_bc0c5c3b/open_orca/preprocessed_data"
[2025-09-14 16:28:42,141 module.py:574 INFO] - * mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only
[2025-09-14 16:28:43,178 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 16:28:43,179 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_a591a3a0/mlc-cached-state.json
[2025-09-14 16:28:44,187 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 16:28:44,188 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_84205c3f/mlc-cached-state.json
[2025-09-14 16:28:45,197 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:28:45,198 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:28:46,235 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 16:28:46,238 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 16:28:47,271 module.py:574 INFO] - * mlcr get,sut,description
[2025-09-14 16:28:48,306 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 16:28:48,308 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_a591a3a0/mlc-cached-state.json
[2025-09-14 16:28:49,322 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 16:28:49,325 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_84205c3f/mlc-cached-state.json
[2025-09-14 16:28:50,350 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:28:50,352 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:28:51,403 module.py:574 INFO] - * mlcr get,compiler
[2025-09-14 16:28:51,406 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-llvm_31a5c1a3/mlc-cached-state.json
[2025-09-14 16:28:52,445 module.py:574 INFO] - * mlcr get,cuda-devices,_with-pycuda
[2025-09-14 16:28:53,492 module.py:574 INFO] - * mlcr get,cuda,_toolkit
[2025-09-14 16:28:53,496 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_04a97d93/mlc-cached-state.json
[2025-09-14 16:28:53,497 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 16:28:53,498 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-14 16:28:53,498 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 16:28:53,498 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 16:28:53,498 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 16:28:54,553 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:28:54,555 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:28:55,701 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 16:28:56,721 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:28:56,722 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:28:56,723 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:28:56,723 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:28:59,810 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:28:59,823 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 16:29:01,751 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:01,754 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:01,755 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 16:29:02,997 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 16:29:04,057 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:04,058 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:04,059 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:29:04,059 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:29:07,267 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:29:07,278 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:29:08,295 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:08,297 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:08,299 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:29:08,300 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:29:11,463 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:29:11,474 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:29:12,494 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:12,497 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:12,499 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:29:12,499 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:29:15,640 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:29:15,651 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:29:17,546 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:17,548 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:17,550 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 16:29:17,562 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:29:17,563 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
GPU 1:
GPU 2:
GPU 3:
GPU 4:
GPU 5:
GPU 6:
GPU 7:
[2025-09-14 16:29:28,859 module.py:5392 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/customize.py
[2025-09-14 16:29:29,866 module.py:574 INFO] - * mlcr get,cache,dir,_name.mlperf-inference-sut-descriptions
[2025-09-14 16:29:29,869 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cache-dir_mlperf-inferenc_021cfc3d/mlc-cached-state.json
[2025-09-14 16:29:29,879 customize.py:56 INFO] - Generating SUT description file for c2e8c0440998-tensorrt
[2025-09-14 16:29:29,882 module.py:5392 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-description/customize.py
[2025-09-14 16:29:31,826 module.py:574 INFO] - * mlcr get,mlperf,inference,results,dir,_version.r5.1-dev
[2025-09-14 16:29:31,829 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_090e5472/mlc-cached-state.json
[2025-09-14 16:29:32,911 module.py:574 INFO] - * mlcr install,pip-package,for-mlc-python,_package.tabulate
[2025-09-14 16:29:32,915 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/install-pip-package-for-mlc-python_34452db7/mlc-cached-state.json
[2025-09-14 16:29:34,002 module.py:574 INFO] - * mlcr get,mlperf,inference,utils
[2025-09-14 16:29:35,112 module.py:574 INFO] - * mlcr get,mlperf,inference,src
[2025-09-14 16:29:35,114 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 16:29:35,118 module.py:5392 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference
[2025-09-14 16:29:35,129 customize.py:273 INFO] -
Running loadgen scenario: Offline and mode: performance
[2025-09-14 16:29:36,321 module.py:574 INFO] - * mlcr app,mlperf,inference,generic,_nvidia,_llama2-70b-99,_tensorrt,_cuda,_valid,_r5.1-dev_default,_offline
[2025-09-14 16:29:37,356 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 16:29:37,357 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_a591a3a0/mlc-cached-state.json
[2025-09-14 16:29:38,387 module.py:574 INFO] - * mlcr get,sys-utils-mlc
[2025-09-14 16:29:38,388 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-sys-utils-mlc_7f1c41c1/mlc-cached-state.json
[2025-09-14 16:29:39,403 module.py:574 INFO] - * mlcr get,python
[2025-09-14 16:29:39,405 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:40,505 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 16:29:40,508 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 16:29:41,544 module.py:574 INFO] - * mlcr get,mlperf,inference,utils
[2025-09-14 16:29:42,633 module.py:574 INFO] - * mlcr get,mlperf,inference,src
[2025-09-14 16:29:42,635 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 16:29:42,643 module.py:5392 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
[2025-09-14 16:29:43,724 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 16:29:44,786 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:44,787 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:44,788 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:29:44,788 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:29:47,998 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:29:48,027 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:29:49,090 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:49,092 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:49,094 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:29:49,094 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:29:52,227 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:29:52,256 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:29:53,328 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:53,330 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:53,332 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:29:53,332 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:29:56,498 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:29:56,526 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:29:57,590 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:29:57,592 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:29:57,594 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 16:29:58,633 module.py:574 INFO] - * mlcr get,cuda-devices,_with-pycuda
[2025-09-14 16:29:59,667 module.py:574 INFO] - * mlcr get,cuda,_toolkit
[2025-09-14 16:29:59,671 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_04a97d93/mlc-cached-state.json
[2025-09-14 16:29:59,672 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 16:29:59,672 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-14 16:29:59,673 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 16:29:59,673 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 16:29:59,673 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 16:30:01,632 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:01,634 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:02,883 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 16:30:03,965 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:03,967 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:03,967 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:30:03,967 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:30:07,202 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:30:07,216 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 16:30:08,234 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:08,236 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:08,238 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 16:30:09,445 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 16:30:10,498 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:10,499 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:10,500 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:30:10,500 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:30:13,703 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:30:13,718 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:30:14,767 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:14,769 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:14,771 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:30:14,771 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:30:17,919 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:30:17,934 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:30:19,834 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:19,837 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:19,839 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:30:19,839 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:30:23,036 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:30:23,050 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:30:24,074 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:24,076 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:24,078 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 16:30:24,090 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:30:24,091 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
GPU 1:
GPU 2:
GPU 3:
GPU 4:
GPU 5:
GPU 6:
GPU 7:
[2025-09-14 16:30:36,572 module.py:5392 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/customize.py
[2025-09-14 16:30:36,598 customize.py:29 INFO] - Extracted Nvidia GPU name: nvl
[2025-09-14 16:30:37,737 module.py:574 INFO] - * mlcr reproduce,mlperf,nvidia,inference,_run_harness,_offline,_cuda,_tensorrt,_llama2-70b-99,_v5.0,_gpu_memory.144,_num-gpus.8
[2025-09-14 16:30:38,781 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 16:30:38,782 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_a591a3a0/mlc-cached-state.json
[2025-09-14 16:30:39,809 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 16:30:39,810 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_84205c3f/mlc-cached-state.json
[2025-09-14 16:30:40,911 module.py:574 INFO] - * mlcr get,sys-utils-mlc
[2025-09-14 16:30:40,913 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-sys-utils-mlc_7f1c41c1/mlc-cached-state.json
[2025-09-14 16:30:41,978 module.py:574 INFO] - * mlcr get,mlperf,inference,nvidia,scratch,space,_version.5.1-dev
[2025-09-14 16:30:41,981 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_754db738/mlc-cached-state.json
[2025-09-14 16:30:43,197 module.py:574 INFO] - * mlcr get,generic-python-lib,_mlperf_logging
[2025-09-14 16:30:44,246 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:44,248 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:44,249 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:30:44,249 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:30:47,430 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:30:47,445 customize.py:152 INFO] - Detected version: 4.1.29
[2025-09-14 16:30:48,469 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:30:48,471 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:48,473 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_ee23bcb4/mlc-cached-state.json
[2025-09-14 16:30:49,573 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 16:30:49,576 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 16:30:50,641 module.py:574 INFO] - * mlcr get,nvidia,mlperf,inference,common-code,_mlcommons,_v5.1-dev
[2025-09-14 16:30:50,644 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-common-code_3e7b2606/mlc-cached-state.json
[2025-09-14 16:30:51,703 module.py:574 INFO] - * mlcr generate,user-conf,mlperf,inference
[2025-09-14 16:30:52,707 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 16:30:52,709 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_a591a3a0/mlc-cached-state.json
[2025-09-14 16:30:53,733 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 16:30:53,736 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_84205c3f/mlc-cached-state.json
[2025-09-14 16:30:54,736 module.py:574 INFO] - * mlcr get,python
[2025-09-14 16:30:54,738 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:30:55,738 module.py:574 INFO] - * mlcr get,sut,configs
[2025-09-14 16:30:56,757 module.py:574 INFO] - * mlcr get,cache,dir,_name.mlperf-inference-sut-configs
[2025-09-14 16:30:56,760 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cache-dir_mlperf-inferenc_78040a08/mlc-cached-state.json
[2025-09-14 16:30:56,764 module.py:5392 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-configs/customize.py
[2025-09-14 16:30:57,820 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 16:30:57,824 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
Using MLCommons Inference source from '/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference'
Original configuration value 0.1 target_qps
Adjusted configuration value 0.101 target_qps
[2025-09-14 16:30:57,838 customize.py:429 INFO] - Output Dir: '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_090e5472/valid_results/c2e8c0440998-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1'
[2025-09-14 16:30:57,838 customize.py:430 INFO] - llama2-70b.Offline.target_qps = 0.101

[2025-09-14 16:30:58,995 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 16:31:00,041 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:00,042 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:00,043 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:00,043 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:31:04,259 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:31:04,274 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 16:31:05,365 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:05,368 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:05,370 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 16:31:06,398 module.py:574 INFO] - * mlcr get,nvidia,mitten
[2025-09-14 16:31:06,402 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-nvidia-mitten_9dd9c352/mlc-cached-state.json
[2025-09-14 16:31:07,466 module.py:574 INFO] - * mlcr get,cuda,_cudnn
[2025-09-14 16:31:07,471 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_d9dfc834/mlc-cached-state.json
[2025-09-14 16:31:07,472 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 16:31:07,473 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: yes
[2025-09-14 16:31:07,473 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 16:31:07,473 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 16:31:07,473 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 16:31:08,527 module.py:574 INFO] - * mlcr get,tensorrt
[2025-09-14 16:31:08,531 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-tensorrt_3f386f9c/mlc-cached-state.json
[2025-09-14 16:31:09,654 module.py:574 INFO] - * mlcr build,nvidia,inference,server,_mlcommons
[2025-09-14 16:31:09,656 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/build-mlperf-inference-server-nvidia_e3f09246/mlc-cached-state.json
[2025-09-14 16:31:10,998 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_build_engine,_offline,_cuda,_tensorrt,_llama2-70b-99,_v5.0,llama2-70b,_v5.0+
[2025-09-14 16:31:11,010 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_f535cd1e/mlc-cached-state.json
[2025-09-14 16:31:12,294 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_preprocess_data,_cuda,_tensorrt,_llama2-70b-99,_v5.0,llama2-70b,_v5.0+
[2025-09-14 16:31:12,304 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_c5e21dc3/mlc-cached-state.json
[2025-09-14 16:31:13,609 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_download_model,_cuda,_tensorrt,_llama2-70b-99,_v5.0,llama2-70b,_v5.0+
[2025-09-14 16:31:13,620 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_0b3f083d/mlc-cached-state.json
[2025-09-14 16:31:14,724 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.transformers
[2025-09-14 16:31:15,757 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:15,758 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:15,759 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:15,760 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:31:18,845 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:31:18,855 customize.py:152 INFO] - Detected version: 4.47.1
[2025-09-14 16:31:19,893 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:19,895 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:19,897 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:19,898 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:31:23,957 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:31:23,973 customize.py:152 INFO] - Detected version: 4.47.1
[2025-09-14 16:31:25,018 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:25,021 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:25,023 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_f3083fc3/mlc-cached-state.json
[2025-09-14 16:31:26,185 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.datasets
[2025-09-14 16:31:27,201 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:27,202 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:27,203 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:27,203 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:31:31,224 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:31:31,231 customize.py:152 INFO] - Detected version: 2.19.2
[2025-09-14 16:31:32,322 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:32,324 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:32,327 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:32,327 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:31:35,566 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:31:35,582 customize.py:152 INFO] - Detected version: 2.19.2
[2025-09-14 16:31:36,622 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:36,625 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:36,627 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_1f0bb503/mlc-cached-state.json
[2025-09-14 16:31:37,785 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.sentencepiece
[2025-09-14 16:31:38,837 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:38,838 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:38,839 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:38,839 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:31:42,018 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:31:42,035 customize.py:152 INFO] - Detected version: 0.2.1
[2025-09-14 16:31:43,065 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:43,068 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:43,070 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_1acc6303/mlc-cached-state.json
[2025-09-14 16:31:44,257 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.protobuf
[2025-09-14 16:31:45,287 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:45,289 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:45,290 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:45,290 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:31:48,461 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:31:48,477 customize.py:152 INFO] - Detected version: 4.24.4
[2025-09-14 16:31:49,518 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:49,520 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:49,523 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_4af006f9/mlc-cached-state.json
[2025-09-14 16:31:50,689 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.accelerate
[2025-09-14 16:31:51,701 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:51,702 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:51,703 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:51,703 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:31:54,861 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:31:54,878 customize.py:152 INFO] - Detected version: 1.10.1
[2025-09-14 16:31:55,920 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:55,923 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:55,925 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_5fa243d2/mlc-cached-state.json
[2025-09-14 16:31:57,136 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.absl-py
[2025-09-14 16:31:58,183 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:31:58,184 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:31:58,185 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:31:58,185 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:32:02,295 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:32:02,310 customize.py:152 INFO] - Detected version: 2.1.0
[2025-09-14 16:32:03,419 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:03,422 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:03,424 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_c3612a1d/mlc-cached-state.json
[2025-09-14 16:32:04,671 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.evaluate
[2025-09-14 16:32:05,748 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:05,749 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:05,750 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:32:05,750 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:32:08,961 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:32:08,978 customize.py:152 INFO] - Detected version: 0.4.5
[2025-09-14 16:32:10,008 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:10,011 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:10,013 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_3eec5236/mlc-cached-state.json
[2025-09-14 16:32:11,233 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.nltk
[2025-09-14 16:32:12,277 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:12,278 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:12,279 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:32:12,279 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:32:15,446 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:32:15,461 customize.py:152 INFO] - Detected version: 3.9.1
[2025-09-14 16:32:16,481 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:16,483 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:16,486 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_1befabbf/mlc-cached-state.json
[2025-09-14 16:32:17,670 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 16:32:18,716 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:18,717 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:18,718 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:32:18,718 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:32:21,858 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:32:21,872 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:32:22,909 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:22,912 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:22,915 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:32:22,915 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:32:27,027 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:32:27,044 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:32:28,069 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:28,072 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:28,074 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:32:28,074 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:32:32,154 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:32:32,171 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 16:32:33,274 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:33,277 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:33,280 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 16:32:34,541 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.rouge-score
[2025-09-14 16:32:35,606 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:35,608 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:35,609 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:32:35,609 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 16:32:38,741 module.py:5392 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 16:32:38,758 customize.py:152 INFO] - Detected version: 0.1.2
[2025-09-14 16:32:39,823 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 16:32:39,826 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 16:32:39,829 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_9dac29fe/mlc-cached-state.json
[2025-09-14 16:32:39,849 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:32:39,850 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/run.sh from tmp-run.sh
[2025-09-14 16:32:42,993 module.py:5392 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/customize.py
[2025-09-14 16:32:44,038 module.py:574 INFO] - * mlcr benchmark-mlperf
[2025-09-14 16:32:44,055 module.py:5392 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program-mlperf/customize.py
[2025-09-14 16:32:45,089 module.py:574 INFO] - * mlcr benchmark-program,program
[2025-09-14 16:32:46,118 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 16:32:46,122 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_84205c3f/mlc-cached-state.json
[2025-09-14 16:32:46,136 customize.py:107 INFO] - ***************************************************************************
[2025-09-14 16:32:46,136 customize.py:109 INFO] - MLC script::benchmark-program/run.sh
[2025-09-14 16:32:46,137 customize.py:110 INFO] -
[2025-09-14 16:32:46,137 customize.py:111 INFO] - Run Directory: /home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA
[2025-09-14 16:32:46,137 customize.py:113 INFO] -
[2025-09-14 16:32:46,137 customize.py:114 INFO] - CMD: make run_harness RUN_ARGS=' --benchmarks=llama2-70b --scenarios=offline --test_mode=PerformanceOnly --user_conf_path=/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/753882f7702943aaaea83ed3ebfeb551.conf --mlperf_conf_path=/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference/mlperf.conf --precision=fp8 --use_deque_limit --use_fp8 --checkpoint_dir=/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_473fcbb9/models/Llama2/fp8-quantized-ammo/llama-2-70b-chat-hf-tp2pp1-fp8 --trtllm_build_flags=tensor_parallelism:2,pipeline_parallelism:1 --no_audit_verify ' 2>&1 | tee '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_090e5472/valid_results/c2e8c0440998-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
[2025-09-14 16:32:46,137 customize.py:116 INFO] -
[2025-09-14 16:32:46,140 module.py:5246 INFO] - ! cd /home/ubuntu
[2025-09-14 16:32:46,141 module.py:5247 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program/run-ubuntu.sh from tmp-run.sh

make run_harness RUN_ARGS=' --benchmarks=llama2-70b --scenarios=offline --test_mode=PerformanceOnly --user_conf_path=/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/753882f7702943aaaea83ed3ebfeb551.conf --mlperf_conf_path=/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference/mlperf.conf --precision=fp8 --use_deque_limit --use_fp8 --checkpoint_dir=/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_473fcbb9/models/Llama2/fp8-quantized-ammo/llama-2-70b-chat-hf-tp2pp1-fp8 --trtllm_build_flags=tensor_parallelism:2,pipeline_parallelism:1 --no_audit_verify ' 2>&1 | tee '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_090e5472/valid_results/c2e8c0440998-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
[2025-09-14 16:33:54,089 main.py:237 INFO] Detected system ID: H200_NVL_141GB
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/main.py", line 239, in
main(main_args, DETECTED_SYSTEM)
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/main.py", line 103, in main
load_config_fn(benchmarks, scenarios)
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/main.py", line 53, in populate_config_registry
ConfigRegistry.load_configs(benchmark, scenario)
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/configs/configuration.py", line 119, in load_configs
importlib.import_module(f"{base_module}.custom")
File "/usr/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 995, in exec_module
File "", line 488, in _call_with_frames_removed
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/configs/llama2-70b/Offline/custom.py", line 47, in
@ConfigRegistry.register(HarnessType.Custom, AccuracyTarget.k_99, PowerSetting.MaxP)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/configs/configuration.py", line 206, in _do_register
raise KeyError("Config for {} is already registered.".format("/".join(map(str, keyspace))))
KeyError: 'Config for Benchmark.LLAMA2/Scenario.Offline/c2e8c0440998/WorkloadSetting(HarnessType.Custom, AccuracyTarget.k_99, PowerSetting.MaxP) is already registered.'
make: *** [Makefile:53: run_harness] Error 1
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/mlcr", line 8, in
sys.exit(mlcr())
^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/main.py", line 87, in mlcr
main()
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/main.py", line 274, in main
res = method(run_args)
^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1779, in _run
r = customize_code.preprocess(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 285, in preprocess
r = mlc.access(ii)
^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1853, in _run
r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3364, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3556, in _run_deps
r = self.action_object.access(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1897, in _run
r = self._run_deps(post_deps, clean_env_keys_post_deps, env, state, const, const_state, add_deps_recursive, recursion_spaces,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3556, in _run_deps
r = self.action_object.access(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1897, in _run
r = self._run_deps(post_deps, clean_env_keys_post_deps, env, state, const, const_state, add_deps_recursive, recursion_spaces,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3556, in _run_deps
r = self.action_object.access(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 248, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = benchmark-program, return code = 512)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

ubuntu@c2e8c0440998:~$

[anandhu-eng]

Hi @skm123 , could you try to exit from the container and rerun the docker run command with an additional flag to skip adding the custom system

steps:

1. Exit from the container
2. do mlc pull repo
3. Rerun the docker run command:

mlcr run-mlperf,inference,_find-performance,_full,_r5.1-dev --model=llama2-70b-99 --implementation=nvidia --framework=tensorrt --category=datacenter --scenario=Offline --execution_mode=test --device=cuda  --docker --quiet --test_query_count=50 --tp_size=2 --rerun --env.MLC_CUSTOM_SYSTEM_NVIDIA=no

[skm123]

@anandhu-eng

Ran the steps you provided...new container launched but script shows error

make generate_engines RUN_ARGS=' --benchmarks=llama2-70b --scenarios=offline --test_mode=PerformanceOnly --offline_expected_qps=0 --precision=fp8 --use_deque_limit --use_fp8 --checkpoint_dir=/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_473fcbb9/models/Llama2/fp8-quantized-ammo/llama-2-70b-chat-hf-tp2pp1-fp8 --trtllm_build_flags=tensor_parallelism:2,pipeline_parallelism:1 --no_audit_verify '
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/main.py", line 37, in
from code.actionhandler import *
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/actionhandler/init.py", line 17, in
from code.actionhandler.generate_engines import GenerateEngineHandler
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 27, in
from code.common.fields import get_applicable_fields
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/common/fields.py", line 25, in
from typeguard import check_type, TypeCheckError
ModuleNotFoundError: No module named 'typeguard'
make: *** [Makefile:37: generate_engines] Error 1
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/mlcr", line 8, in
sys.exit(mlcr())
^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/main.py", line 87, in mlcr
main()
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/main.py", line 274, in main
res = method(run_args)
^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1853, in _run
r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3364, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3556, in _run_deps
r = self.action_object.access(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1642, in _run
r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3364, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3556, in _run_deps
r = self.action_object.access(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
result = method(options)

File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 248, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = app-mlperf-inference-nvidia, return code = 256)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

ubuntu@1442ab301356:~$

[anandhu-eng]

Hi @skm123 , could you try to do mlc pull repo and run the same command without docker. Im attaching the reference command here:

mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only \
   --model=llama2-70b-99 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=datacenter \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --quiet  \
   --tp_size=2 \

[anandhu-eng]

Hi @skm123 , Just wanted to make the previous reply a bit clear. The commands could be executed inside the docker itself, there is no need to build a new one.

[skm123]

ok. Ran commands inside docker

ubuntu@1442ab301356:~$ mlc pull repo
[2025-09-14 19:05:43,592 repo_action.py:297 INFO] - Repository mlperf-automations already exists at /home/ubuntu/MLC/repos/mlcommons@mlperf-automations. Checking for local changes...
[2025-09-14 19:05:43,608 repo_action.py:308 INFO] - No local changes detected. Pulling latest changes...
Already up to date.
[2025-09-14 19:05:44,025 repo_action.py:310 INFO] - Repository successfully pulled.
[2025-09-14 19:05:44,026 repo_action.py:324 INFO] - Registering the repo in repos.json

ubuntu@1442ab301356:~$ mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only --model=llama2-70b-99 --implementation=nvidia --framework=tensorrt --category=datacenter --scenario=Offline --execution_mode=valid --device=cuda --quiet --tp_size=2
[2025-09-14 19:06:12,494 module.py:574 INFO] - * mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only
[2025-09-14 19:06:13,544 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:06:13,545 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:06:14,582 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:06:14,583 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:06:15,650 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:15,652 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:16,691 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 19:06:16,694 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:06:17,723 module.py:574 INFO] - * mlcr get,sut,description
[2025-09-14 19:06:18,747 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:06:18,750 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:06:19,760 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:06:19,763 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:06:20,783 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:20,785 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:21,850 module.py:574 INFO] - * mlcr get,compiler
[2025-09-14 19:06:21,853 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-llvm_ce28d3c3/mlc-cached-state.json
[2025-09-14 19:06:22,871 module.py:574 INFO] - * mlcr get,cuda-devices,_with-pycuda
[2025-09-14 19:06:23,921 module.py:574 INFO] - * mlcr get,cuda,_toolkit
[2025-09-14 19:06:23,924 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_385956b8/mlc-cached-state.json
[2025-09-14 19:06:23,926 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 19:06:23,926 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-14 19:06:23,926 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 19:06:23,927 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 19:06:23,927 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 19:06:25,010 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:25,012 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:26,244 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 19:06:27,284 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:27,286 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:27,286 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:06:27,286 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:06:31,379 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:06:31,387 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 19:06:32,482 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:32,485 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:32,486 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 19:06:33,735 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:06:34,821 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:34,822 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:34,823 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:06:34,823 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:06:37,994 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:06:38,005 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:06:39,111 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:39,114 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:39,116 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:06:39,116 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:06:42,361 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:06:42,373 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:06:43,406 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:43,408 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:43,410 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:06:43,411 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:06:46,542 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:06:46,564 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:06:47,622 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:06:47,624 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:06:47,626 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:06:47,637 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:06:47,638 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
GPU 1:
GPU 2:
GPU 3:
GPU 4:
GPU 5:
GPU 6:
GPU 7:
[2025-09-14 19:06:59,873 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/customize.py
[2025-09-14 19:07:01,774 module.py:574 INFO] - * mlcr get,cache,dir,_name.mlperf-inference-sut-descriptions
[2025-09-14 19:07:01,777 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cache-dir_mlperf-inferenc_6beef993/mlc-cached-state.json
[2025-09-14 19:07:01,787 customize.py:56 INFO] - Generating SUT description file for 1442ab301356-tensorrt
[2025-09-14 19:07:01,790 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-description/customize.py
[2025-09-14 19:07:02,873 module.py:574 INFO] - * mlcr get,mlperf,inference,results,dir,_version.r5.1-dev
[2025-09-14 19:07:02,877 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_47be03e6/mlc-cached-state.json
[2025-09-14 19:07:03,961 module.py:574 INFO] - * mlcr install,pip-package,for-mlc-python,_package.tabulate
[2025-09-14 19:07:03,963 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/install-pip-package-for-mlc-python_32632320/mlc-cached-state.json
[2025-09-14 19:07:05,071 module.py:574 INFO] - * mlcr get,mlperf,inference,utils
[2025-09-14 19:07:06,158 module.py:574 INFO] - * mlcr get,mlperf,inference,src
[2025-09-14 19:07:06,161 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:07:06,171 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference
[2025-09-14 19:07:06,194 customize.py:273 INFO] -
Running loadgen scenario: Offline and mode: performance
[2025-09-14 19:07:07,430 module.py:574 INFO] - * mlcr app,mlperf,inference,generic,_nvidia,_llama2-70b-99,_tensorrt,_cuda,_valid,_r5.1-dev_default,_offline
[2025-09-14 19:07:08,467 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:07:08,468 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:07:09,493 module.py:574 INFO] - * mlcr get,sys-utils-mlc
[2025-09-14 19:07:09,495 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-sys-utils-mlc_7f1c41c1/mlc-cached-state.json
[2025-09-14 19:07:10,532 module.py:574 INFO] - * mlcr get,python
[2025-09-14 19:07:10,534 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:11,636 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 19:07:11,639 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:07:12,662 module.py:574 INFO] - * mlcr get,mlperf,inference,utils
[2025-09-14 19:07:13,745 module.py:574 INFO] - * mlcr get,mlperf,inference,src
[2025-09-14 19:07:13,748 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:07:13,756 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
[2025-09-14 19:07:14,846 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:07:15,906 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:15,907 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:15,908 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:07:15,908 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:07:19,126 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:07:19,155 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:07:20,213 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:20,215 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:20,217 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:07:20,217 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:07:23,435 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:07:23,463 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:07:24,508 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:24,511 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:24,512 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:07:24,513 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:07:27,601 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:07:27,630 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:07:28,648 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:28,650 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:28,651 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:07:29,690 module.py:574 INFO] - * mlcr get,cuda-devices,_with-pycuda
[2025-09-14 19:07:31,649 module.py:574 INFO] - * mlcr get,cuda,_toolkit
[2025-09-14 19:07:31,653 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_385956b8/mlc-cached-state.json
[2025-09-14 19:07:31,654 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 19:07:31,654 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-14 19:07:31,654 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 19:07:31,654 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 19:07:31,654 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 19:07:32,768 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:32,770 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:34,016 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 19:07:35,101 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:35,102 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:35,103 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:07:35,103 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:07:38,253 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:07:38,266 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 19:07:39,343 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:39,346 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:39,348 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 19:07:40,526 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:07:41,562 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:41,563 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:41,564 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:07:41,564 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:07:44,716 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:07:44,726 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:07:45,831 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:45,834 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:45,836 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:07:45,836 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:07:48,987 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:07:49,001 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:07:50,050 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:50,053 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:50,055 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:07:50,055 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:07:53,226 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:07:53,240 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:07:54,278 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:07:54,281 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:07:54,283 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:07:54,295 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:07:54,295 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
GPU 1:
GPU 2:
GPU 3:
GPU 4:
GPU 5:
GPU 6:
GPU 7:
[2025-09-14 19:08:07,788 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/customize.py
[2025-09-14 19:08:07,800 customize.py:29 INFO] - Extracted Nvidia GPU name: nvl
[2025-09-14 19:08:08,953 module.py:574 INFO] - * mlcr reproduce,mlperf,nvidia,inference,_run_harness,_tensorrt,_llama2-70b-99,_offline,_cuda,_v5.0,_gpu_memory.144,_num-gpus.8
[2025-09-14 19:08:10,005 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:08:10,006 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:08:11,096 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:08:11,097 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:08:12,161 module.py:574 INFO] - * mlcr get,sys-utils-mlc
[2025-09-14 19:08:12,163 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-sys-utils-mlc_7f1c41c1/mlc-cached-state.json
[2025-09-14 19:08:13,181 module.py:574 INFO] - * mlcr get,mlperf,inference,nvidia,scratch,space,_version.5.1-dev
[2025-09-14 19:08:13,184 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_7c03d596/mlc-cached-state.json
[2025-09-14 19:08:14,405 module.py:574 INFO] - * mlcr get,generic-python-lib,_mlperf_logging
[2025-09-14 19:08:15,427 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:08:15,428 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:15,429 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:08:15,429 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:08:18,575 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:08:18,589 customize.py:152 INFO] - Detected version: 4.1.29
[2025-09-14 19:08:19,630 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:08:19,633 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:19,635 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_ee23bcb4/mlc-cached-state.json
[2025-09-14 19:08:20,747 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 19:08:20,750 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:08:21,779 module.py:574 INFO] - * mlcr get,nvidia,mlperf,inference,common-code,_mlcommons,_v5.1-dev
[2025-09-14 19:08:21,782 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-common-code_3e7b2606/mlc-cached-state.json
[2025-09-14 19:08:22,842 module.py:574 INFO] - * mlcr generate,user-conf,mlperf,inference
[2025-09-14 19:08:23,896 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:08:23,898 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:08:24,926 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:08:24,929 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:08:25,978 module.py:574 INFO] - * mlcr get,python
[2025-09-14 19:08:25,981 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:27,032 module.py:574 INFO] - * mlcr get,sut,configs
[2025-09-14 19:08:28,062 module.py:574 INFO] - * mlcr get,cache,dir,_name.mlperf-inference-sut-configs
[2025-09-14 19:08:28,065 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cache-dir_mlperf-inferenc_b525f9a9/mlc-cached-state.json
[2025-09-14 19:08:28,070 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-configs/customize.py
[2025-09-14 19:08:29,157 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 19:08:29,162 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
Using MLCommons Inference source from '/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference'
Original configuration value 0.1 target_qps
Adjusted configuration value 0.101 target_qps
[2025-09-14 19:08:29,174 customize.py:429 INFO] - Output Dir: '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_47be03e6/valid_results/1442ab301356-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1'
[2025-09-14 19:08:29,174 customize.py:430 INFO] - llama2-70b.Offline.target_qps = 0.101

[2025-09-14 19:08:31,269 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 19:08:32,358 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:08:32,359 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:32,360 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:08:32,360 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:08:35,697 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:08:35,713 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 19:08:36,747 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:08:36,750 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:36,752 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 19:08:37,790 module.py:574 INFO] - * mlcr get,nvidia,mitten
[2025-09-14 19:08:37,794 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-nvidia-mitten_9dd9c352/mlc-cached-state.json
[2025-09-14 19:08:38,846 module.py:574 INFO] - * mlcr get,cuda,_cudnn
[2025-09-14 19:08:38,851 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_d9dfc834/mlc-cached-state.json
[2025-09-14 19:08:38,853 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 19:08:38,853 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: yes
[2025-09-14 19:08:38,853 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 19:08:38,853 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 19:08:38,853 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 19:08:39,940 module.py:574 INFO] - * mlcr get,tensorrt
[2025-09-14 19:08:39,943 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-tensorrt_3f386f9c/mlc-cached-state.json
[2025-09-14 19:08:41,116 module.py:574 INFO] - * mlcr build,nvidia,inference,server,_mlcommons
[2025-09-14 19:08:41,119 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/build-mlperf-inference-server-nvidia_f138b972/mlc-cached-state.json
[2025-09-14 19:08:42,438 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_build_engine,_tensorrt,_llama2-70b-99,_offline,_cuda,_v5.0,llama2-70b,_v5.0+
[2025-09-14 19:08:42,450 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_826c4e13/mlc-cached-state.json
[2025-09-14 19:08:43,744 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_preprocess_data,_tensorrt,_llama2-70b-99,_cuda,_v5.0,llama2-70b,_v5.0+
[2025-09-14 19:08:43,755 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_30f2709b/mlc-cached-state.json
[2025-09-14 19:08:45,041 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_download_model,_tensorrt,_llama2-70b-99,_cuda,_v5.0,llama2-70b,_v5.0+
[2025-09-14 19:08:45,052 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_25eb9d71/mlc-cached-state.json
[2025-09-14 19:08:46,234 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.transformers
[2025-09-14 19:08:47,257 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:08:47,258 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:47,259 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:08:47,259 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:08:50,442 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:08:50,456 customize.py:152 INFO] - Detected version: 4.47.1
[2025-09-14 19:08:51,480 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:08:51,483 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:51,485 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:08:51,485 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:08:54,622 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:08:54,638 customize.py:152 INFO] - Detected version: 4.47.1
[2025-09-14 19:08:55,709 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:08:55,712 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:55,714 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_c8c40326/mlc-cached-state.json
[2025-09-14 19:08:56,889 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.datasets
[2025-09-14 19:08:57,928 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:08:57,929 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:08:57,930 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:08:57,930 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:02,032 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:02,047 customize.py:152 INFO] - Detected version: 2.19.2
[2025-09-14 19:09:03,139 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:03,141 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:03,144 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:03,144 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:07,295 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:07,312 customize.py:152 INFO] - Detected version: 2.19.2
[2025-09-14 19:09:08,387 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:08,389 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:08,391 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_7da9c89b/mlc-cached-state.json
[2025-09-14 19:09:09,625 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.sentencepiece
[2025-09-14 19:09:10,657 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:10,658 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:10,659 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:10,659 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:13,782 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:13,793 customize.py:152 INFO] - Detected version: 0.2.1
[2025-09-14 19:09:14,815 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:14,818 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:14,820 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_1acc6303/mlc-cached-state.json
[2025-09-14 19:09:16,027 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.protobuf
[2025-09-14 19:09:17,050 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:17,052 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:17,053 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:17,053 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:20,246 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:20,262 customize.py:152 INFO] - Detected version: 4.24.4
[2025-09-14 19:09:21,318 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:21,321 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:21,323 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_4af006f9/mlc-cached-state.json
[2025-09-14 19:09:22,480 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.accelerate
[2025-09-14 19:09:23,484 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:23,486 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:23,487 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:23,487 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:26,630 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:26,644 customize.py:152 INFO] - Detected version: 1.10.1
[2025-09-14 19:09:27,683 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:27,686 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:27,688 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_5fa243d2/mlc-cached-state.json
[2025-09-14 19:09:28,869 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.absl-py
[2025-09-14 19:09:29,933 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:29,934 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:29,935 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:29,935 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:34,129 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:34,143 customize.py:152 INFO] - Detected version: 2.1.0
[2025-09-14 19:09:35,206 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:35,209 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:35,211 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_c3612a1d/mlc-cached-state.json
[2025-09-14 19:09:36,429 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.evaluate
[2025-09-14 19:09:37,481 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:37,482 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:37,483 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:37,483 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:40,716 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:40,732 customize.py:152 INFO] - Detected version: 0.4.5
[2025-09-14 19:09:41,762 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:41,765 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:41,767 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_3eec5236/mlc-cached-state.json
[2025-09-14 19:09:42,969 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.nltk
[2025-09-14 19:09:43,995 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:43,997 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:43,998 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:43,998 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:47,196 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:47,212 customize.py:152 INFO] - Detected version: 3.9.1
[2025-09-14 19:09:48,233 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:48,236 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:48,238 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_1befabbf/mlc-cached-state.json
[2025-09-14 19:09:49,450 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:09:50,501 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:50,502 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:50,503 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:50,504 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:53,592 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:53,606 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:09:54,626 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:54,629 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:54,631 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:54,631 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:09:57,804 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:09:57,821 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:09:58,895 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:09:58,898 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:09:58,900 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:09:58,900 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:10:03,051 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:10:03,067 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:10:04,169 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:10:04,172 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:10:04,174 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:10:05,413 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.rouge-score
[2025-09-14 19:10:07,308 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:10:07,309 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:10:07,310 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:10:07,310 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:10:10,534 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:10:10,550 customize.py:152 INFO] - Detected version: 0.1.2
[2025-09-14 19:10:11,592 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:10:11,595 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:10:11,598 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_9dac29fe/mlc-cached-state.json
[2025-09-14 19:10:12,784 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.typeguard
[2025-09-14 19:10:13,807 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:10:13,809 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:10:13,810 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:10:13,810 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:10:17,033 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:10:17,044 customize.py:152 INFO] - Detected version: 4.4.4
[2025-09-14 19:10:18,099 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:10:18,101 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:10:18,103 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_c1c782bb/mlc-cached-state.json
[2025-09-14 19:10:18,123 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:10:18,123 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/run.sh from tmp-run.sh
[2025-09-14 19:10:21,177 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/customize.py
[2025-09-14 19:10:22,226 module.py:574 INFO] - * mlcr benchmark-mlperf
[2025-09-14 19:10:22,242 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program-mlperf/customize.py
[2025-09-14 19:10:23,306 module.py:574 INFO] - * mlcr benchmark-program,program
[2025-09-14 19:10:24,310 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:10:24,313 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:10:24,326 customize.py:107 INFO] - ***************************************************************************
[2025-09-14 19:10:24,326 customize.py:109 INFO] - MLC script::benchmark-program/run.sh
[2025-09-14 19:10:24,326 customize.py:110 INFO] -
[2025-09-14 19:10:24,326 customize.py:111 INFO] - Run Directory: /home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA
[2025-09-14 19:10:24,326 customize.py:113 INFO] -
[2025-09-14 19:10:24,327 customize.py:114 INFO] - CMD: make run_harness RUN_ARGS=' --benchmarks=llama2-70b --scenarios=offline --test_mode=PerformanceOnly --user_conf_path=/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/eaca58861fb849b090ef6f5db6954abd.conf --mlperf_conf_path=/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference/mlperf.conf --precision=fp8 --use_deque_limit --use_fp8 --checkpoint_dir=/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_473fcbb9/models/Llama2/fp8-quantized-ammo/llama-2-70b-chat-hf-tp2pp1-fp8 --trtllm_build_flags=tensor_parallelism:2,pipeline_parallelism:1 --no_audit_verify ' 2>&1 | tee '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_47be03e6/valid_results/1442ab301356-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
[2025-09-14 19:10:24,327 customize.py:116 INFO] -
[2025-09-14 19:10:24,330 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:10:24,330 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program/run-ubuntu.sh from tmp-run.sh

make run_harness RUN_ARGS=' --benchmarks=llama2-70b --scenarios=offline --test_mode=PerformanceOnly --user_conf_path=/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/eaca58861fb849b090ef6f5db6954abd.conf --mlperf_conf_path=/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference/mlperf.conf --precision=fp8 --use_deque_limit --use_fp8 --checkpoint_dir=/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_473fcbb9/models/Llama2/fp8-quantized-ammo/llama-2-70b-chat-hf-tp2pp1-fp8 --trtllm_build_flags=tensor_parallelism:2,pipeline_parallelism:1 --no_audit_verify ' 2>&1 | tee '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_47be03e6/valid_results/1442ab301356-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
[2025-09-14 19:11:32,520 main.py:235 INFO] Detected system did not match any known systems. Exiting. System(cpu=CPU(name='INTEL(R) XEON(R) PLATINUM 8580', architecture=<CPUArchitecture.x86_64: AliasedName(name='x86_64', aliases=(), patterns=())>, vendor='GenuineIntel', cores_per_group=60, threads_per_core=2, n_groups=2, group_type=<GroupType.Socket: 'socket'>, numa_nodes=[[Interval(start=0, end=29), Interval(start=120, end=149)], [Interval(start=30, end=59), Interval(start=150, end=179)], [Interval(start=60, end=89), Interval(start=180, end=209)], [Interval(start=90, end=119), Interval(start=210, end=239)]], flags={'monitor', 'rdpid', 'art', 'ss', 'vme', 'vmx', 'smx', 'tm', 'rdrand', 'ssbd', 'sha_ni', 'gfni', 'adx', 'stibp', 'smap', 'msr', 'avx512f', 'avx512_vnni', 'rep_good', 'sse4_2', 'bus_lock_detect', 'cmov', 'serialize', 'cqm_llc', 'fxsr', 'movbe', 'cat_l2', 'dtherm', 'dtes64', 'cqm_mbm_total', 'cldemote', 'xsaveopt', 'lm', 'ssse3', 'mba', 'avx512cd', 'pse36', 'avx512dq', 'ospke', 'arch_perfmon', 'cat_l3', 'pku', 'sep', 'ht', 'avx512_vbmi2', 'cqm_occup_llc', 'sse2', 'avx512_vpopcntdq', 'avx512vbmi', 'avx512bw', 'mce', 'bts', 'avx512_bitalg', 'ds_cpl', 'cx16', 'xsaves', 'ida', 'pts', 'ept', 'pge', 'xgetbv1', 'md_clear', 'pse', 'cdp_l3', 'tsc', 'pbe', 'invpcid_single', 'de', 'pae', 'sdbg', 'intel_ppin', 'cx8', 'movdiri', 'clflushopt', 'tsc_adjust', 'pconfig', 'arat', 'tm2', 'waitpkg', 'arch_capabilities', 'clwb', 'movdir64b', 'cqm_mbm_local', 'avx512_bf16', 'amx_bf16', '3dnowprefetch', 'ibrs_enhanced', 'enqcmd', 'popcnt', 'tpr_shadow', 'dca', 'split_lock_detect', 'constant_tsc', 'mca', 'avx512_fp16', 'nonstop_tsc', 'avx512vl', 'arch_lbr', 'pni', 'pcid', 'vpid', 'x2apic', 'pclmulqdq', 'fsgsbase', 'flush_l1d', 'avx_vnni', 'clflush', 'rdseed', 'nx', 'cpuid_fault', 'fsrm', 'bmi2', 'amx_tile', 'lahf_lm', 'amx_int8', 'vpclmulqdq', 'pdpe1gb', 'erms', 'vaes', 'xtopology', 'fma', 'nopl', 'intel_pt', 'pdcm', 'flexpriority', 'smep', 'ibpb', 'mmx', 'avx512ifma', 'acpi', 'pat', 'pebs', 'umip', 'avx2', 'fpu', 'syscall', 'dts', 'tsxldtrk', 'vnmi', 'aperfmperf', 'invpcid', 'sse', 'cpuid', 'est', 'cdp_l2', 'f16c', 'cqm', 'mtrr', 'sse4_1', 'tme', 'tsc_deadline_timer', 'bmi1', 'pln', 'apic', 'la57', 'ibrs', 'rdt_a', 'epb', 'avx', 'xtpr', 'tsc_known_freq', 'wbnoinvd', 'aes', 'xsave', 'rdtscp', 'xsavec', 'ept_ad', 'abm'}, vulnerabilities={'spec store bypass': 'Mitigation; Speculative Store Bypass disabled via prctl and seccomp', 'spectre v1': 'Mitigation; usercopy/swapgs barriers and __user pointer sanitization', 'spectre v2': 'Mitigation; Enhanced / Automatic IBRS; IBPB disabled; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S'}), host_memory=HostMemory(capacity=Memory(quantity=2.113405092, byte_suffix=<ByteSuffix.TB: (1000, 4)>, _num_bytes=2113405092000)), accelerators={<class 'nvmitten.nvidia.accelerator.GPU'>: [GPU(name='NVIDIA H200 NVL', pci_id='0x233B10DE', compute_sm=ComputeSM(major=9, minor=0), vram=Memory(quantity=139.81109619140625, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=150121021440), max_power_limit=600.0, is_integrated=False, gpu_index=0), GPU(name='NVIDIA H200 NVL', pci_id='0x233B10DE', compute_sm=ComputeSM(major=9, minor=0), vram=Memory(quantity=139.81109619140625, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=150121021440), max_power_limit=600.0, is_integrated=False, gpu_index=1), GPU(name='NVIDIA H200 NVL', pci_id='0x233B10DE', compute_sm=ComputeSM(major=9, minor=0), vram=Memory(quantity=139.81109619140625, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=150121021440), max_power_limit=600.0, is_integrated=False, gpu_index=2), GPU(name='NVIDIA H200 NVL', pci_id='0x233B10DE', compute_sm=ComputeSM(major=9, minor=0), vram=Memory(quantity=139.81109619140625, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=150121021440), max_power_limit=600.0, is_integrated=False, gpu_index=3), GPU(name='NVIDIA H200 NVL', pci_id='0x233B10DE', compute_sm=ComputeSM(major=9, minor=0), vram=Memory(quantity=139.81109619140625, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=150121021440), max_power_limit=600.0, is_integrated=False, gpu_index=4), GPU(name='NVIDIA H200 NVL', pci_id='0x233B10DE', compute_sm=ComputeSM(major=9, minor=0), vram=Memory(quantity=139.81109619140625, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=150121021440), max_power_limit=600.0, is_integrated=False, gpu_index=5), GPU(name='NVIDIA H200 NVL', pci_id='0x233B10DE', compute_sm=ComputeSM(major=9, minor=0), vram=Memory(quantity=139.81109619140625, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=150121021440), max_power_limit=600.0, is_integrated=False, gpu_index=6), GPU(name='NVIDIA H200 NVL', pci_id='0x233B10DE', compute_sm=ComputeSM(major=9, minor=0), vram=Memory(quantity=139.81109619140625, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=150121021440), max_power_limit=600.0, is_integrated=False, gpu_index=7)], <class 'nvmitten.nvidia.accelerator.DLA'>: []}, extras={})

======================== Result summaries: ========================

[2025-09-14 19:11:34,807 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program/customize.py
[2025-09-14 19:11:35,861 module.py:574 INFO] - * mlcr save,mlperf,inference,state
[2025-09-14 19:11:35,878 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/save-mlperf-inference-implementation-state/customize.py
[2025-09-14 19:11:35,904 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:11:35,904 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/run.sh from tmp-run.sh
[2025-09-14 19:11:39,033 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/customize.py
[2025-09-14 19:11:40,068 module.py:574 INFO] - * mlcr get,mlperf,sut,description
[2025-09-14 19:11:41,153 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:11:41,156 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:11:42,204 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:11:42,207 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:11:43,278 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:11:43,281 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:11:44,374 module.py:574 INFO] - * mlcr get,compiler
[2025-09-14 19:11:44,376 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-llvm_ce28d3c3/mlc-cached-state.json
[2025-09-14 19:11:45,410 module.py:574 INFO] - * mlcr get,cuda-devices,_with-pycuda
[2025-09-14 19:11:46,470 module.py:574 INFO] - * mlcr get,cuda,_toolkit
[2025-09-14 19:11:46,473 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_385956b8/mlc-cached-state.json
[2025-09-14 19:11:46,475 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 19:11:46,475 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-14 19:11:46,475 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 19:11:46,475 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 19:11:46,475 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 19:11:47,491 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:11:47,493 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:11:48,586 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 19:11:49,606 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:11:49,607 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:11:49,609 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:11:49,609 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:11:52,752 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:11:52,766 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 19:11:53,811 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:11:53,814 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:11:53,816 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 19:11:54,929 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:11:55,944 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:11:55,945 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:11:55,947 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:11:55,947 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:11:59,092 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:11:59,106 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:12:00,996 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:12:00,999 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:12:01,002 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:12:01,002 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:12:04,323 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:12:04,336 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:12:05,403 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:12:05,406 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:12:05,409 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:12:05,409 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:12:08,578 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:12:08,590 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:12:09,641 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:12:09,644 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:12:09,647 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:12:09,667 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:12:09,667 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
GPU 1:
GPU 2:
GPU 3:
GPU 4:
GPU 5:
GPU 6:
GPU 7:
[2025-09-14 19:12:21,947 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/customize.py
[2025-09-14 19:12:22,960 module.py:574 INFO] - * mlcr get,cache,dir,_name.mlperf-inference-sut-descriptions
[2025-09-14 19:12:22,964 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cache-dir_mlperf-inferenc_6beef993/mlc-cached-state.json
[2025-09-14 19:12:22,977 customize.py:56 INFO] - Generating SUT description file for 1442ab301356-tensorrt
[2025-09-14 19:12:22,982 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-description/customize.py
[2025-09-14 19:12:23,017 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py
ubuntu@1442ab301356:~$

[anandhu-eng]

Could you try running the following command:

mlcr add,custom,system,nvidia

When prompting for system name, try to give a meaningfull name to system like test_system_llama2 and rerun the mlcr command along with --rerun

mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only \
   --model=llama2-70b-99 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=datacenter \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --quiet  \
   --tp_size=2 --rerun

[skm123]

[2025-09-14 19:32:22,973 module.py:5255 INFO] - ! cd /home/ubuntu/MLC/repos/local/cache/add-custom-nvidia-system_2140fa2a
[2025-09-14 19:32:22,973 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/add-custom-nvidia-system/run.sh from tmp-run.sh
This script creates a custom system definition within the MLPerf Inference codebase that matches the
hardware specifications of the system that it is run on. The script then does the following:

- Backs up NVIDIA's workload configuration files
- Creates new workload configuration files (configs/<Benchmark name>/<Scenario>/__init__.py) with dummy values
    - The user should fill out these dummy values with the correct values

============= DETECTED SYSTEM ==============

System
INTEL(R) XEON(R) PLATINUM 8580 (240 Threads, x86_64)
2.11 TB Host Memory
nvmitten.nvidia.accelerator.GPU
NVIDIA H200 NVL (PCI_ID: 0x233B10DE)
Memory Capacity: 139.81 GiB
Max Power Limit: 600.0 W
Compute Capability: 90
Is Integrated Graphics: False
NVIDIA H200 NVL (PCI_ID: 0x233B10DE)
Memory Capacity: 139.81 GiB
Max Power Limit: 600.0 W
Compute Capability: 90
Is Integrated Graphics: False
NVIDIA H200 NVL (PCI_ID: 0x233B10DE)
Memory Capacity: 139.81 GiB
Max Power Limit: 600.0 W
Compute Capability: 90
Is Integrated Graphics: False
NVIDIA H200 NVL (PCI_ID: 0x233B10DE)
Memory Capacity: 139.81 GiB
Max Power Limit: 600.0 W
Compute Capability: 90
Is Integrated Graphics: False
NVIDIA H200 NVL (PCI_ID: 0x233B10DE)
Memory Capacity: 139.81 GiB
Max Power Limit: 600.0 W
Compute Capability: 90
Is Integrated Graphics: False
NVIDIA H200 NVL (PCI_ID: 0x233B10DE)
Memory Capacity: 139.81 GiB
Max Power Limit: 600.0 W
Compute Capability: 90
Is Integrated Graphics: False
NVIDIA H200 NVL (PCI_ID: 0x233B10DE)
Memory Capacity: 139.81 GiB
Max Power Limit: 600.0 W
Compute Capability: 90
Is Integrated Graphics: False
NVIDIA H200 NVL (PCI_ID: 0x233B10DE)
Memory Capacity: 139.81 GiB
Max Power Limit: 600.0 W
Compute Capability: 90
Is Integrated Graphics: False

============================================
=> A system ID is a string containing only letters, numbers, and underscores
=> that is used as the human-readable name of the system. It is also used as
=> the system name when creating the measurements/ and results/ entries.
=> This string should also start with a letter to be a valid Python enum member name.
=> Specify the system ID to use for the current system [Default: 1442ab301356]: H200_NVL_141GB
=> Reloaded system list. Matched System ID: H200_NVL_141GB
=> This script will generate Benchmark Configuration stubs for the detected system.
Continue? [Y/n]: Y
=> Generating configs for H200_NVL_141GB...
=> Writing config stubs to configs/dlrm-v2/Offline/custom.py
=> Writing config stubs to configs/dlrm-v2/Server/custom.py
=> Writing config stubs to configs/gptj/Offline/custom.py
=> Writing config stubs to configs/gptj/Server/custom.py
=> Writing config stubs to configs/gptj/SingleStream/custom.py
=> Writing config stubs to configs/llama2-70b/Offline/custom.py
=> Writing config stubs to configs/llama2-70b/Server/custom.py
=> Writing config stubs to configs/llama2-70b-interactive/Offline/custom.py
=> Writing config stubs to configs/llama2-70b-interactive/Server/custom.py
=> Writing config stubs to configs/llama3_1-405b/Offline/custom.py
=> Writing config stubs to configs/llama3_1-405b/Server/custom.py
=> Writing config stubs to configs/mixtral-8x7b/Offline/custom.py
=> Writing config stubs to configs/mixtral-8x7b/Server/custom.py
=> Writing config stubs to configs/resnet50/Offline/custom.py
=> Writing config stubs to configs/resnet50/Server/custom.py
=> Writing config stubs to configs/resnet50/SingleStream/custom.py
=> Writing config stubs to configs/resnet50/MultiStream/custom.py
=> Writing config stubs to configs/retinanet/Offline/custom.py
=> Writing config stubs to configs/retinanet/Server/custom.py
=> Writing config stubs to configs/retinanet/SingleStream/custom.py
=> Writing config stubs to configs/retinanet/MultiStream/custom.py
=> Writing config stubs to configs/stable-diffusion-xl/Offline/custom.py
=> Writing config stubs to configs/stable-diffusion-xl/Server/custom.py
=> Writing config stubs to configs/stable-diffusion-xl/SingleStream/custom.py
=> Writing config stubs to configs/3d-unet/Offline/custom.py
=> Writing config stubs to configs/3d-unet/SingleStream/custom.py
=> Writing config stubs to configs/rgat/Offline/custom.py
[2025-09-14 19:33:05,311 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/add-custom-nvidia-system/customize.py
[2025-09-14 19:33:07,367 module.py:2165 INFO] - - cache UID: 2140fa2ac14348e3
ubuntu@1442ab301356:~$

########

ubuntu@1442ab301356:~$ mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only
--model=llama2-70b-99
--implementation=nvidia
--framework=tensorrt
--category=datacenter
--scenario=Offline
--execution_mode=valid
--device=cuda
--quiet
--tp_size=2 --rerun
[2025-09-14 19:36:11,522 module.py:574 INFO] - * mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only
[2025-09-14 19:36:12,548 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:36:12,549 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:36:13,592 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:36:13,594 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:36:14,622 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:14,624 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:15,701 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 19:36:15,704 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:36:16,761 module.py:574 INFO] - * mlcr get,sut,description
[2025-09-14 19:36:17,811 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:36:17,813 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:36:18,829 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:36:18,831 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:36:19,877 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:19,878 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:20,940 module.py:574 INFO] - * mlcr get,compiler
[2025-09-14 19:36:20,943 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-llvm_ce28d3c3/mlc-cached-state.json
[2025-09-14 19:36:21,972 module.py:574 INFO] - * mlcr get,cuda-devices,_with-pycuda
[2025-09-14 19:36:23,029 module.py:574 INFO] - * mlcr get,cuda,_toolkit
[2025-09-14 19:36:23,032 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_385956b8/mlc-cached-state.json
[2025-09-14 19:36:23,034 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 19:36:23,034 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-14 19:36:23,034 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 19:36:23,034 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 19:36:23,034 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 19:36:24,107 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:24,110 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:25,291 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 19:36:26,304 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:26,305 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:26,306 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:36:26,306 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:36:29,486 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:36:29,498 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 19:36:31,409 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:31,412 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:31,414 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 19:36:32,654 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:36:33,725 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:33,726 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:33,727 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:36:33,727 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:36:36,965 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:36:36,977 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:36:38,061 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:38,063 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:38,065 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:36:38,065 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:36:41,233 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:36:41,244 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:36:42,287 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:42,289 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:42,291 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:36:42,291 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:36:45,359 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:36:45,379 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:36:46,414 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:36:46,416 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:36:46,418 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:36:46,430 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:36:46,430 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
GPU 1:
GPU 2:
GPU 3:
GPU 4:
GPU 5:
GPU 6:
GPU 7:
[2025-09-14 19:36:58,523 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/customize.py
[2025-09-14 19:36:59,546 module.py:574 INFO] - * mlcr get,cache,dir,_name.mlperf-inference-sut-descriptions
[2025-09-14 19:36:59,549 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cache-dir_mlperf-inferenc_6beef993/mlc-cached-state.json
[2025-09-14 19:36:59,560 customize.py:56 INFO] - Generating SUT description file for 1442ab301356-tensorrt
[2025-09-14 19:36:59,563 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-description/customize.py
[2025-09-14 19:37:01,468 module.py:574 INFO] - * mlcr get,mlperf,inference,results,dir,_version.r5.1-dev
[2025-09-14 19:37:01,472 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_47be03e6/mlc-cached-state.json
[2025-09-14 19:37:02,538 module.py:574 INFO] - * mlcr install,pip-package,for-mlc-python,_package.tabulate
[2025-09-14 19:37:02,541 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/install-pip-package-for-mlc-python_32632320/mlc-cached-state.json
[2025-09-14 19:37:03,598 module.py:574 INFO] - * mlcr get,mlperf,inference,utils
[2025-09-14 19:37:04,702 module.py:574 INFO] - * mlcr get,mlperf,inference,src
[2025-09-14 19:37:04,705 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:37:04,716 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference
[2025-09-14 19:37:04,738 customize.py:273 INFO] -
Running loadgen scenario: Offline and mode: performance
[2025-09-14 19:37:05,936 module.py:574 INFO] - * mlcr app,mlperf,inference,generic,_nvidia,_llama2-70b-99,_tensorrt,_cuda,_valid,_r5.1-dev_default,_offline
[2025-09-14 19:37:06,983 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:37:06,984 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:37:08,029 module.py:574 INFO] - * mlcr get,sys-utils-mlc
[2025-09-14 19:37:08,031 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-sys-utils-mlc_7f1c41c1/mlc-cached-state.json
[2025-09-14 19:37:09,058 module.py:574 INFO] - * mlcr get,python
[2025-09-14 19:37:09,060 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:10,106 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 19:37:10,110 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:37:11,126 module.py:574 INFO] - * mlcr get,mlperf,inference,utils
[2025-09-14 19:37:12,193 module.py:574 INFO] - * mlcr get,mlperf,inference,src
[2025-09-14 19:37:12,196 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:37:12,204 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
[2025-09-14 19:37:13,285 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:37:14,322 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:14,324 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:14,325 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:37:14,325 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:37:17,595 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:37:17,624 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:37:18,654 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:18,657 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:18,659 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:37:18,659 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:37:21,810 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:37:21,839 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:37:22,867 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:22,870 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:22,871 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:37:22,872 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:37:26,004 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:37:26,031 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:37:27,088 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:27,090 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:27,092 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:37:28,150 module.py:574 INFO] - * mlcr get,cuda-devices,_with-pycuda
[2025-09-14 19:37:29,227 module.py:574 INFO] - * mlcr get,cuda,_toolkit
[2025-09-14 19:37:29,230 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_385956b8/mlc-cached-state.json
[2025-09-14 19:37:29,232 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 19:37:29,232 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2025-09-14 19:37:29,232 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 19:37:29,232 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 19:37:29,232 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 19:37:31,186 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:31,188 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:32,401 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 19:37:33,501 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:33,502 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:33,503 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:37:33,503 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:37:36,651 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:37:36,662 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 19:37:37,692 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:37,695 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:37,697 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 19:37:38,871 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:37:39,890 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:39,891 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:39,892 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:37:39,892 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:37:43,074 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:37:43,087 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:37:44,128 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:44,130 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:44,132 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:37:44,132 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:37:47,337 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:37:47,352 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:37:48,366 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:48,367 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:48,369 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:37:48,369 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:37:51,535 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:37:51,549 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:37:52,587 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:37:52,590 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:37:52,592 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:37:52,604 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:37:52,605 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
GPU 1:
GPU 2:
GPU 3:
GPU 4:
GPU 5:
GPU 6:
GPU 7:
[2025-09-14 19:38:06,019 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/customize.py
[2025-09-14 19:38:06,045 customize.py:29 INFO] - Extracted Nvidia GPU name: nvl
[2025-09-14 19:38:07,232 module.py:574 INFO] - * mlcr reproduce,mlperf,nvidia,inference,_run_harness,_cuda,_tensorrt,_offline,_llama2-70b-99,_v5.0,_gpu_memory.144,_num-gpus.8
[2025-09-14 19:38:08,302 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:38:08,303 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:38:09,354 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:38:09,356 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:38:10,386 module.py:574 INFO] - * mlcr get,sys-utils-mlc
[2025-09-14 19:38:10,388 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-sys-utils-mlc_7f1c41c1/mlc-cached-state.json
[2025-09-14 19:38:11,433 module.py:574 INFO] - * mlcr get,mlperf,inference,nvidia,scratch,space,_version.5.1-dev
[2025-09-14 19:38:11,436 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_7c03d596/mlc-cached-state.json
[2025-09-14 19:38:12,609 module.py:574 INFO] - * mlcr get,generic-python-lib,_mlperf_logging
[2025-09-14 19:38:13,623 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:38:13,625 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:13,626 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:38:13,626 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:38:16,841 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:38:16,856 customize.py:152 INFO] - Detected version: 4.1.29
[2025-09-14 19:38:17,870 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:38:17,873 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:17,875 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_ee23bcb4/mlc-cached-state.json
[2025-09-14 19:38:18,991 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 19:38:18,994 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
[2025-09-14 19:38:20,023 module.py:574 INFO] - * mlcr get,nvidia,mlperf,inference,common-code,_v5.1-dev,_mlcommons
[2025-09-14 19:38:20,027 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-common-code_3e7b2606/mlc-cached-state.json
[2025-09-14 19:38:21,058 module.py:574 INFO] - * mlcr generate,user-conf,mlperf,inference
[2025-09-14 19:38:22,085 module.py:574 INFO] - * mlcr detect,os
[2025-09-14 19:38:22,088 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-os_9a8feeb5/mlc-cached-state.json
[2025-09-14 19:38:23,100 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:38:23,103 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:38:24,125 module.py:574 INFO] - * mlcr get,python
[2025-09-14 19:38:24,127 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:25,190 module.py:574 INFO] - * mlcr get,sut,configs
[2025-09-14 19:38:26,221 module.py:574 INFO] - * mlcr get,cache,dir,_name.mlperf-inference-sut-configs
[2025-09-14 19:38:26,224 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cache-dir_mlperf-inferenc_b525f9a9/mlc-cached-state.json
[2025-09-14 19:38:26,229 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-configs/customize.py
[2025-09-14 19:38:27,333 module.py:574 INFO] - * mlcr get,mlcommons,inference,src
[2025-09-14 19:38:27,338 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_39d284fc/mlc-cached-state.json
Using MLCommons Inference source from '/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference'
Original configuration value 0.1 target_qps
Adjusted configuration value 0.101 target_qps
[2025-09-14 19:38:27,350 customize.py:429 INFO] - Output Dir: '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_47be03e6/valid_results/1442ab301356-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1'
[2025-09-14 19:38:27,350 customize.py:430 INFO] - llama2-70b.Offline.target_qps = 0.101

[2025-09-14 19:38:28,525 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.pycuda
[2025-09-14 19:38:29,579 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:38:29,581 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:29,581 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:38:29,582 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:38:33,685 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:38:33,701 customize.py:152 INFO] - Detected version: 2024.1
[2025-09-14 19:38:34,738 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:38:34,741 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:34,743 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_d7059bf4/mlc-cached-state.json
[2025-09-14 19:38:35,809 module.py:574 INFO] - * mlcr get,nvidia,mitten
[2025-09-14 19:38:35,813 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-nvidia-mitten_9dd9c352/mlc-cached-state.json
[2025-09-14 19:38:36,890 module.py:574 INFO] - * mlcr get,cuda,_cudnn
[2025-09-14 19:38:36,895 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-cuda_d9dfc834/mlc-cached-state.json
[2025-09-14 19:38:36,896 module.py:2240 INFO] - ENV[CUDA_HOME]: /usr/local/cuda
[2025-09-14 19:38:36,897 module.py:2240 INFO] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: yes
[2025-09-14 19:38:36,897 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION]: 12.8
[2025-09-14 19:38:36,897 module.py:2240 INFO] - ENV[MLC_CUDA_VERSION_STRING]: cu128
[2025-09-14 19:38:36,897 module.py:2240 INFO] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2025-09-14 19:38:37,935 module.py:574 INFO] - * mlcr get,tensorrt
[2025-09-14 19:38:37,938 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-tensorrt_3f386f9c/mlc-cached-state.json
[2025-09-14 19:38:39,101 module.py:574 INFO] - * mlcr build,nvidia,inference,server,_mlcommons
[2025-09-14 19:38:39,104 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/build-mlperf-inference-server-nvidia_f138b972/mlc-cached-state.json
[2025-09-14 19:38:40,439 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_build_engine,_cuda,_tensorrt,_offline,_llama2-70b-99,_v5.0,llama2-70b,_v5.0+
[2025-09-14 19:38:40,450 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_826c4e13/mlc-cached-state.json
[2025-09-14 19:38:41,798 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_preprocess_data,_cuda,_tensorrt,_llama2-70b-99,_v5.0,llama2-70b,_v5.0+
[2025-09-14 19:38:41,809 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_30f2709b/mlc-cached-state.json
[2025-09-14 19:38:43,099 module.py:574 INFO] - * mlcr reproduce,mlperf,inference,nvidia,harness,_download_model,_cuda,_tensorrt,_llama2-70b-99,_v5.0,llama2-70b,_v5.0+
[2025-09-14 19:38:43,109 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/app-mlperf-inference-nvidia_25eb9d71/mlc-cached-state.json
[2025-09-14 19:38:44,277 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.transformers
[2025-09-14 19:38:45,297 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:38:45,298 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:45,299 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:38:45,299 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:38:48,521 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:38:48,531 customize.py:152 INFO] - Detected version: 4.47.1
[2025-09-14 19:38:49,544 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:38:49,547 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:49,549 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:38:49,550 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:38:52,721 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:38:52,737 customize.py:152 INFO] - Detected version: 4.47.1
[2025-09-14 19:38:53,778 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:38:53,781 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:53,783 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_c8c40326/mlc-cached-state.json
[2025-09-14 19:38:54,963 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.datasets
[2025-09-14 19:38:55,975 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:38:55,977 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:38:55,978 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:38:55,978 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:38:59,977 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:38:59,994 customize.py:152 INFO] - Detected version: 2.19.2
[2025-09-14 19:39:01,907 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:01,910 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:01,912 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:01,912 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:05,228 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:05,244 customize.py:152 INFO] - Detected version: 2.19.2
[2025-09-14 19:39:06,292 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:06,295 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:06,297 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_7da9c89b/mlc-cached-state.json
[2025-09-14 19:39:07,472 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.sentencepiece
[2025-09-14 19:39:08,492 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:08,493 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:08,494 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:08,494 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:11,710 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:11,726 customize.py:152 INFO] - Detected version: 0.2.1
[2025-09-14 19:39:12,752 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:12,755 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:12,757 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_1acc6303/mlc-cached-state.json
[2025-09-14 19:39:13,968 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.protobuf
[2025-09-14 19:39:14,965 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:14,966 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:14,967 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:14,967 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:18,177 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:18,194 customize.py:152 INFO] - Detected version: 4.24.4
[2025-09-14 19:39:19,227 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:19,230 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:19,233 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_4af006f9/mlc-cached-state.json
[2025-09-14 19:39:20,421 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.accelerate
[2025-09-14 19:39:21,434 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:21,435 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:21,437 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:21,437 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:24,555 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:24,569 customize.py:152 INFO] - Detected version: 1.10.1
[2025-09-14 19:39:25,603 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:25,606 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:25,609 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_5fa243d2/mlc-cached-state.json
[2025-09-14 19:39:26,788 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.absl-py
[2025-09-14 19:39:27,803 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:27,804 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:27,805 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:27,805 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:31,887 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:31,904 customize.py:152 INFO] - Detected version: 2.1.0
[2025-09-14 19:39:33,008 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:33,011 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:33,014 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_c3612a1d/mlc-cached-state.json
[2025-09-14 19:39:34,230 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.evaluate
[2025-09-14 19:39:35,295 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:35,296 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:35,297 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:35,297 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:38,447 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:38,458 customize.py:152 INFO] - Detected version: 0.4.5
[2025-09-14 19:39:39,535 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:39,538 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:39,540 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_3eec5236/mlc-cached-state.json
[2025-09-14 19:39:40,764 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.nltk
[2025-09-14 19:39:41,798 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:41,799 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:41,801 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:41,801 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:44,970 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:44,985 customize.py:152 INFO] - Detected version: 3.9.1
[2025-09-14 19:39:46,004 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:46,007 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:46,009 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_1befabbf/mlc-cached-state.json
[2025-09-14 19:39:47,260 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.numpy
[2025-09-14 19:39:48,284 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:48,285 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:48,286 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:48,286 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:51,473 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:51,487 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:39:52,512 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:52,514 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:52,517 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:52,517 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:55,637 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:55,655 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:39:56,675 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:39:56,678 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:39:56,681 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:39:56,681 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:39:59,837 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:39:59,854 customize.py:152 INFO] - Detected version: 1.26.4
[2025-09-14 19:40:01,818 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:40:01,821 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:40:01,823 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_6d1eda38/mlc-cached-state.json
[2025-09-14 19:40:04,023 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.rouge-score
[2025-09-14 19:40:05,131 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:40:05,132 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:40:05,133 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:40:05,133 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:40:08,347 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:40:08,363 customize.py:152 INFO] - Detected version: 0.1.2
[2025-09-14 19:40:09,446 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:40:09,449 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:40:09,451 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_9dac29fe/mlc-cached-state.json
[2025-09-14 19:40:10,657 module.py:574 INFO] - * mlcr get,generic-python-lib,_package.typeguard
[2025-09-14 19:40:11,701 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:40:11,702 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:40:11,704 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:40:11,704 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-09-14 19:40:14,872 module.py:5401 INFO] - ! call "detect_version" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2025-09-14 19:40:14,889 customize.py:152 INFO] - Detected version: 4.4.4
[2025-09-14 19:40:15,924 module.py:574 INFO] - * mlcr get,python3
[2025-09-14 19:40:15,927 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-python3_55ffc3f4/mlc-cached-state.json
[2025-09-14 19:40:15,929 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_c1c782bb/mlc-cached-state.json
[2025-09-14 19:40:15,950 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:40:15,950 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/run.sh from tmp-run.sh
[2025-09-14 19:40:19,100 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/customize.py
[2025-09-14 19:40:20,179 module.py:574 INFO] - * mlcr benchmark-mlperf
[2025-09-14 19:40:20,195 module.py:5401 INFO] - ! call "postprocess" from /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program-mlperf/customize.py
[2025-09-14 19:40:21,259 module.py:574 INFO] - * mlcr benchmark-program,program
[2025-09-14 19:40:22,324 module.py:574 INFO] - * mlcr detect,cpu
[2025-09-14 19:40:22,327 module.py:1283 INFO] - ! load /home/ubuntu/MLC/repos/local/cache/detect-cpu_9f04c3cf/mlc-cached-state.json
[2025-09-14 19:40:22,340 customize.py:107 INFO] - ***************************************************************************
[2025-09-14 19:40:22,340 customize.py:109 INFO] - MLC script::benchmark-program/run.sh
[2025-09-14 19:40:22,341 customize.py:110 INFO] -
[2025-09-14 19:40:22,341 customize.py:111 INFO] - Run Directory: /home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA
[2025-09-14 19:40:22,341 customize.py:113 INFO] -
[2025-09-14 19:40:22,341 customize.py:114 INFO] - CMD: make run_harness RUN_ARGS=' --benchmarks=llama2-70b --scenarios=offline --test_mode=PerformanceOnly --user_conf_path=/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/2c06598b0db84f8db747694aee8dae5e.conf --mlperf_conf_path=/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference/mlperf.conf --precision=fp8 --use_deque_limit --use_fp8 --checkpoint_dir=/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_473fcbb9/models/Llama2/fp8-quantized-ammo/llama-2-70b-chat-hf-tp2pp1-fp8 --trtllm_build_flags=tensor_parallelism:2,pipeline_parallelism:1 --no_audit_verify ' 2>&1 | tee '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_47be03e6/valid_results/1442ab301356-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
[2025-09-14 19:40:22,341 customize.py:116 INFO] -
[2025-09-14 19:40:22,345 module.py:5255 INFO] - ! cd /home/ubuntu
[2025-09-14 19:40:22,345 module.py:5256 INFO] - ! call /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program/run-ubuntu.sh from tmp-run.sh

make run_harness RUN_ARGS=' --benchmarks=llama2-70b --scenarios=offline --test_mode=PerformanceOnly --user_conf_path=/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/2c06598b0db84f8db747694aee8dae5e.conf --mlperf_conf_path=/home/ubuntu/MLC/repos/local/cache/get-git-repo_inference-src_32e7a8f3/inference/mlperf.conf --precision=fp8 --use_deque_limit --use_fp8 --checkpoint_dir=/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_473fcbb9/models/Llama2/fp8-quantized-ammo/llama-2-70b-chat-hf-tp2pp1-fp8 --trtllm_build_flags=tensor_parallelism:2,pipeline_parallelism:1 --no_audit_verify ' 2>&1 | tee '/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_47be03e6/valid_results/1442ab301356-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
[2025-09-14 19:41:29,866 main.py:237 INFO] Detected system ID: H200_NVL_141GB
[2025-09-14 19:41:32,207 font_manager.py:1639 INFO] generated new fontManager
/usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
[2025-09-14 19:41:35,173 config.py:58 INFO] PyTorch version 2.6.0a0+ecf3bae40a.nv25.1 available.
[2025-09-14 19:41:35,175 config.py:70 INFO] Polars version 1.8.2 available.
[1757878900.361163] [1442ab301356:16296:0] ib_iface.c:1230 UCX ERROR mlx5_2: ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes)
[1757878900.361199] [1442ab301356:16296:0] ucp_worker.c:1413 UCX ERROR uct_iface_open(rc_verbs/mlx5_2:1) failed: Input/output error
[1442ab301356:16296] pml_ucx.c:309 Error: Failed to create UCP worker
[TensorRT-LLM] TensorRT-LLM version: 0.18.0.dev2025021800
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/main.py", line 239, in
main(main_args, DETECTED_SYSTEM)
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/main.py", line 144, in main
dispatch_action(main_args, config_dict, workload_setting)
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/main.py", line 210, in dispatch_action
handler.run()
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/actionhandler/base.py", line 74, in run
self.setup()
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/actionhandler/run_harness.py", line 85, in setup
duper.run()
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/common/protected_super.py", line 137, in _f
return r(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/actionhandler/base.py", line 74, in run
self.setup()
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/actionhandler/generate_conf_files.py", line 70, in setup
self.harness, self.benchmark_conf = get_harness(self.benchmark_conf, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/init.py", line 237, in get_harness
harness = get_cls(G_HARNESS_CLASS_MAP[k])(conf, benchmark)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/harness/harness_llm_py/mlperf_frontend.py", line 121, in init
super().init(args, benchmark)
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/common/harness.py", line 104, in init
self.enumerate_engines()
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/common/harness.py", line 191, in enumerate_engines
gpu_e2e_batch_size = get_e2e_batch_size(self.args["gpu_batch_size"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_ca15e111/repo/closed/NVIDIA/code/common/utils.py", line 68, in get_e2e_batch_size
raise ValueError(f"{batch_size_dict} input batch_size_dict is empty.")
ValueError: {} input batch_size_dict is empty.
make: *** [Makefile:53: run_harness] Error 1
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/mlcr", line 8, in
sys.exit(mlcr())
^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/main.py", line 87, in mlcr
main()
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/main.py", line 274, in main
res = method(run_args)
^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1779, in _run
r = customize_code.preprocess(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 285, in preprocess
r = mlc.access(ii)
^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1853, in _run
r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3364, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3556, in _run_deps
r = self.action_object.access(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1897, in _run
r = self._run_deps(post_deps, clean_env_keys_post_deps, env, state, const, const_state, add_deps_recursive, recursion_spaces,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3556, in _run_deps
r = self.action_object.access(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1897, in _run
r = self._run_deps(post_deps, clean_env_keys_post_deps, env, state, const, const_state, add_deps_recursive, recursion_spaces,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3556, in _run_deps
r = self.action_object.access(ii)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/action.py", line 56, in access
result = method(options)
^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 316, in run
return self.call_script_module_function("run", run_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/mlc/script_action.py", line 248, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = benchmark-program, return code = 512)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

ubuntu@1442ab301356:~$

[anandhu-eng]

Hi @skm123 , could you add --batch_size=llama2-70b:4096 to the run command. Since we have not tested in 8xh200_NVL, the run is not picking any of the preset we have fixed.

EDIT: 4096 might not be the optimal one for 8xH200_NVL. Please feel free to adjust the value.