feat: Added --build_variant flag for cpu only build.

[Sunidhi-Gaonkar1]

What does the PR do?

Checklist

• I have read the Contribution guidelines and signed the Contributor License
  Agreement
• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• I ran pre-commit locally (pre-commit install, pre-commit run --all)
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

Where should the reviewer start?

Test plan:

Used below commands to create triton server docker instance and starting up the triton server:
1.sudo docker run --privileged -it -p8000:8000 --volume /model_repository:/var/models --name triton_server_test tritonserver
2../bin/tritonserver --model-repository=/var/models

Tested the simple identity model using below commands in a different session:
1.curl -X POST localhost:8000/v2/repository/index | jq

curl -X POST localhost:8000/v2/repository/index | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    58  100    58    0     0  29000      0 --:--:-- --:--:-- --:--:-- 29000
[
  {
    "name": "simple_identity",
    "version": "1",
    "state": "READY"
  }
]

2.curl localhost:8000/v2/models/simple_identity | jq

curl localhost:8000/v2/models/simple_identity | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   193  100   193    0     0   188k      0 --:--:-- --:--:-- --:--:--  188k
{
  "name": "simple_identity",
  "versions": [
    "1"
  ],
  "platform": "python",
  "inputs": [
    {
      "name": "INPUT0",
      "datatype": "BYTES",
      "shape": [
        -1,
        -1
      ]
    }
  ],
  "outputs": [
    {
      "name": "OUTPUT0",
      "datatype": "BYTES",
      "shape": [
        -1,
        -1
      ]
    }
  ]
}

Caveats:

Background

Hi team,
This PR includes changes made to add Power support to triton-inference-server for python backend. Can you please review/merge this?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

[mkumatag]

I feel that the cpu variant could be applicable for other use cases as well, instead of limiting it to the ppc64le architecture. Can we just parameterize this variant and set all as a default value (this way we can safely remove this uname check here)?

[Sunidhi-Gaonkar1]

Sure, we are working on these changes.

[Sunidhi-Gaonkar1]

I have made the changes to remove uname check , added a --build_variant flag which is set to all by default.

[mkumatag]

why are we installing explicitly on ppc64le architecture and not on other architectures?

[Sunidhi-Gaonkar1]

We were facing linker issue for numpy as it was getting built from source. Hence, need to install clang-15 for ppc64le architecture. For other architectures numpy wheels are available.

[Sunidhi-Gaonkar1]

I have removed these changes as we are no longer facing the linker issue.

[mkumatag]

LGTM,

squash the commits and change the PR heading accordingly

[alhad-deshpande]

@mc-nv @dmitry-tokarev-nv @whoisj
Can you please review/approve this PR?

[whoisj]

These changes make sense to me. @mc-nv to review and determine if the changes are risky or compatible w/ out internal build process or not (I think they should be).

[mc-nv]

To me looks like the change is about to select the required wheel package dependencies during installation aka all vs cpu.

1. I'm struggling to find in any of our pyproject.toml or setup.py CPU requirements that are using optional dependency flag cpu. It's also not a part of this change.
2. On other side we have flag --enable-gpu which is probably can be reused for same purpose in current context, there why introduction of new flag is questionable.

[whoisj]

@Sunidhi-Gaonkar1 please address @mc-nv's concerns. Thank you.

[Sunidhi-Gaonkar1]

Hi @whoisj @mc-nv Sorry for the delayed response.

We tried using enable_gpu flag to build cpu only wheel but it ran into issues due to following reasons:

1. When we set enable_gpu flag in the build command then all gpu related dependencies like cuda-keyring gets installed using apt command.

2. When we don't set the enable_gpu flag in the build command then wheels are built for "all" variant but cupy_cuda library gets installed via pip command.

Hence regardless of enable_gpu flag is set or unset it tries loading up some or the other cuda related library.

If any architecture does not have gpu support then this new build_variant flag can be used to enable cpu only build. When build_variant is passed as "cpu" it will not load any cuda related library regardless of whether enable_gpu is set or unset. This creates separate responsibilities for both the flags enable_gpu and build_variant. The default value of the build_variant has been kept as "all".

Please let us know your thoughts.

[Sunidhi-Gaonkar1]

We are also currently facing build failure due to this cuda related library , libnvshmem3-cuda-13, we are working to resolve this issue.

[Sunidhi-Gaonkar1]

Hi @whoisj @mc-nv, we have fixed the libnvshmem3-cuda-13 library issue by adding build_variant check. Please review the PR and let us know if any changes are required.