GPT-J(CUDA with Docker)failing with "Failed building wheel for pycuda"

[barsha-barua-sndk]

Command run
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
--model=gptj-99
--implementation=nvidia
--framework=tensorrt
--category=edge
--scenario=Offline
--execution_mode=test
--device=cuda
--docker --quiet
--test_query_count=50 --rerun

Please find the terminal log:

GPTJ-99_pycudafail.txt

Highlighting the main failure:
L/usr/lib/stubs -L/usr/lib64/stubs -L/usr/lib/x86_64-linux-gnu -lcuda -lcurand -o build/lib.linux-x86_64-cpython-310/pycuda/_driver.cpython-310-x86_64-linux-gnu.so
/usr/bin/ld: cannot find /root/MLC/repos/local/cache/install-cuda-prebuilt_b949f23d/install/lib64: file format not recognized
collect2: error: ld returned 1 exit status
error: command '/usr/bin/x86_64-linux-gnu-g++' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycuda
Failed to build pycuda
ERROR: Could not build wheels for pycuda, which is required to install pyproject.toml-based projects

[arjunsuresh]

Are you running on aarch64?

[barsha-barua-sndk]

@arjunsuresh , No I am using 64bit Intel station(X86_64) with NVIDIA RTX 3070 GPU for my testing

[barsha-barua-sndk]

Failing to build wheel was solved by setting the Environment variable in the python env to the correct CUDA version.(I also changed the python version to 3.8 to 3.10)
@arjunsuresh Is there is any Ubuntu OS dependencies to run each model ? I am currently running 20.04

But Post that , It is failing at preparing metadata , could you please help look into the issue

Attaching the error snippet
Collecting lark (from -r requirements.txt (line 8))
Downloading lark-1.2.2-py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: mpi4py in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 9)) (3.1.5)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (1.24.4)
Requirement already satisfied: onnx>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 11)) (1.15.0rc2)
Requirement already satisfied: polygraphy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (0.48.1)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 13)) (5.9.4)
Collecting pynvml>=11.5.0 (from -r requirements.txt (line 14))
Downloading pynvml-12.0.0-py3-none-any.whl.metadata (5.4 kB)
Collecting sentencepiece>=0.1.99 (from -r requirements.txt (line 15))
Downloading sentencepiece-0.2.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (10 kB)
Requirement already satisfied: tensorrt==9.2.0.post12.dev5 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 16)) (9.2.0.post12.dev5)
Requirement already satisfied: torch<=2.2.0a in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 17)) (2.2.0a0+81ea7a4)
Collecting nvidia-ammo~=0.7.0 (from -r requirements.txt (line 18))
Downloading nvidia-ammo-0.7.4.tar.gz (6.9 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-yjq96flx/nvidia-ammo_224465c572d74f8aa5e61043a05e3ffa/setup.py", line 90, in
raise RuntimeError("Bad params")
RuntimeError: Bad params
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.3.1 -> 25.2
[notice] To update, run: python3 -m pip install --upgrade pip
Traceback (most recent call last):
File "/code/tensorrt_llm/scripts/build_wheel.py", line 319, in
main(**vars(args))
File "/code/tensorrt_llm/scripts/build_wheel.py", line 67, in main
build_run(
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '"/usr/bin/python3" -m pip install -r requirements-dev.txt --extra-index-url https://pypi.ngc.nvidia.com' returned non-zero exit status 1.
make: *** [Makefile:102: devel_run] Error 1
make: Leaving directory '/root/MLC/repos/local/cache/get-git-repo_tensorrt-llm_f4f157f5/repo/docker'
Traceback (most recent call last):
File "/home/qa/mlc/bin/mlcr", line 8, in
sys.exit(mlcr())
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/main.py", line 86, in mlcr
main()
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/main.py", line 273, in main
res = method(run_args)
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 311, in run
return self.call_script_module_function("run", run_args)
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
r = self._run(i)
File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1780, in _run
r = customize_code.preprocess(ii)
File "/root/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 285, in preprocess
r = mlc.access(ii)
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 288, in docker
return self.call_script_module_function("docker", run_args)
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 232, in call_script_module_function
result = automation_instance.docker(run_args) # Pass args to the run method
File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 4573, in docker
return docker_run(self, i)
File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py", line 338, in docker_run
r = self_module._run_deps(
File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3552, in _run_deps
r = self.action_object.access(ii)
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 311, in run
return self.call_script_module_function("run", run_args)
File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 246, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-ml-model-gptj, return code = 256)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

[arjunsuresh]

@barsha-barua-sndk No. In Nvidia implementation everything happens via docker containers and there shouldn't be any host OS dependencies. But gptj or any LLM models haven't been tested for Nvidia implementation for a while and may not work as expected.