[Misc] Batch API envoy integration fix, E2E verification, and document update #1671

zhangjyr · 2025-10-15T17:25:41Z

Pull Request Description

This PR completes the last piece of the Batch API integration. Specifically:

Envoy integration
Added e2e test script to verify batch job execution
Update the document to customize job execution

Note that the k8s job is disabled by default. Config object storage (S3 or TOS) to enable it.
The default behavior is to upload files to the metadata-local and running job in the dummy engine, which is only for testing purposes.

Related Issues

Resolves: #1277

Important: Before submitting, please complete the description above and review the checklist below.

Contribution Guidelines (Expand for Details)

We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:

Pull Request Title Format

Your PR title should start with one of these prefixes to indicate the nature of the change:

[Bug]: Corrections to existing functionality
[CI]: Changes to build process or CI pipeline
[Docs]: Updates or additions to documentation
[API]: Modifications to aibrix's API or interface
[CLI]: Changes or additions to the Command Line Interface
[Misc]: For changes not covered above (use sparingly)

Note: For changes spanning multiple categories, use multiple prefixes in order of importance.

Submission Checklist

PR title includes appropriate prefix(es)
Changes are clearly explained in the PR description
New and existing tests pass successfully
Code adheres to project style and best practices
Documentation updated to reflect changes (if applicable)
Thorough testing completed, no regressions introduced

By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.

…ntegration Signed-off-by: Jingyuan Zhang <[email protected]>

Signed-off-by: Jingyuan Zhang <[email protected]>

…openai_batch_envoy_integration Signed-off-by: Jingyuan Zhang <[email protected]> # Conflicts: # python/aibrix/aibrix/batch/README.md # python/aibrix/aibrix/batch/scheduler.py # python/aibrix/aibrix/metadata/app.py # python/aibrix/tests/batch/conftest.py # python/aibrix/tests/batch/test_e2e_openai_batch_api.py # python/aibrix/tests/batch/test_job_cache.py # python/aibrix/tests/metadata/test_app_integration.py

Signed-off-by: Jingyuan Zhang <[email protected]>

…openai_batch_envoy_integration Signed-off-by: Jingyuan Zhang <[email protected]>

Signed-off-by: Jingyuan Zhang <[email protected]>

gemini-code-assist · 2025-10-15T17:26:14Z

Summary of Changes

Hello @zhangjyr, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request finalizes the integration of the Batch API by establishing the necessary Envoy routing for batch-related endpoints, introducing robust end-to-end tests to ensure the functionality of the batch processing workflow, and providing detailed documentation and configuration options for customizing batch job execution within Kubernetes. These changes collectively enable a fully functional and verifiable Batch API experience.

Highlights

Envoy Integration for Batch API: Configured Envoy Gateway to route '/v1/files' and '/v1/batches' endpoints to the metadata service, completing the network integration for the Batch API.
End-to-End Batch API Verification: Introduced a new comprehensive end-to-end test suite ('test_batch_api.py') to validate the entire batch processing workflow against a real service instance, covering file upload, job creation, status polling, and result retrieval.
Customizable Batch Job Execution: Added a Kubernetes Job template ('job_template_patch.yaml') and updated documentation to allow users to customize batch job execution environments, including container images, parallelism, and resource requirements.
Metadata Service Enhancements: Updated the metadata service Dockerfile, Kubernetes deployment, and RBAC rules to support Kubernetes Job creation and management, including new permissions and the ability to load custom job templates.
Object Storage Integration: Enabled the metadata service to configure S3 and TOS object storage credentials via Kubernetes secrets, facilitating file storage for batch operations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request effectively completes the Batch API integration by adding Envoy configuration, E2E tests, and comprehensive documentation. The changes are well-structured and consistent across the repository. My review focuses on improving the robustness of documentation and test setup, enhancing security by adhering to the principle of least privilege in RBAC configurations, and modernizing asynchronous code patterns in the new E2E tests.

config/metadata/metadata.yaml

docs/source/features/batch-api.rst

python/aibrix/tests/e2e/README.md

python/aibrix/tests/e2e/test_batch_api.py

config/gateway/gateway-plugin/gateway-plugin.yaml

Jeffwan · 2025-10-15T17:37:50Z

config/metadata/kustomization.yaml

 - metadata.yaml
 - redis.yaml

+configMapGenerator:


do we need configmap here? I see python folder have many skeleton templates

The job template patch has to be mapped to the container folder to take effect. This configmap can achieve this without rebuilding the image.

then should we remove file based templates?

I am glad to hear the options. Can you elaborate on how users might customize the job template?
The bottom line is users have to change the k8s_job_template.yaml and rebuild the image.

template should be managed by operation folks. We can have one template now, leave the flexibility support to later . Once we get more feedback, we can start to work on it

Jeffwan · 2025-10-15T17:38:34Z

config/metadata/metadata.yaml

+    resources: ["jobs"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  # For batch job ServiceAccount management
+  - apiGroups: [""]


can we use a fix service account, along with aibrix installation? instead of reply on permission here?

I was originally fixed. I later found that the service account must be under the default namespace, while the kustomize overrides the namespace and puts everything in the aibrix-system.

em. we can put the file under a separate config/manifest folder without aibrix-system override.

in helm, I think we can support it as well, something like

namespace: {{ .Values.job.namespace }}

I am not sure if it works. How is kustomize going to load config/manifest without overriding the namespace? I can have a try.

We can ask user to follow a guidance to create a RBAC in short term, we do not want the controller to have such high permission.

Jeffwan · 2025-10-15T17:39:15Z

dist/chart/templates/metadata-service/deployment.yaml

        - name: metadata-service
          image: {{ .Values.metadata.service.container.image.repository }}:{{ .Values.metadata.service.container.image.tag }}
          imagePullPolicy: {{ .Values.metadata.service.container.image.imagePullPolicy | default "IfNotPresent" }}
-          command: ["python", "-m", "aibrix.metadata.app"]


let's keep the explicit commands here?

Oh, I can add an explicit command, but it would be aibrix_metadata

that's ok. user sometimes need to adjust parameters, if there's no config here, it automatically use container entrypoints, and user do not know it unless they know the commands in Dockerfile

I found aibrix_metadatadoes not work in the helm. I have to change to
["python", "-m", "aibrix.metadata.app"].
One more thing, in the config, I disabled k8s-job by default, so users can run other metadata services (v1/models) if they do not configure external object storage (in this case, the k8s job will not work anyway). Do I need to apply the same logic in chart, too? This means users have to adjust values to enable object storage and k8s-job.

python/aibrix/.python-version

Cleanup files Signed-off-by: Jingyuan Zhang <[email protected]>

Signed-off-by: Jingyuan Zhang <[email protected]>

…t store to work. Signed-off-by: Jingyuan Zhang <[email protected]>

Signed-off-by: Jingyuan Zhang <[email protected]>

zhangjyr · 2025-10-16T06:33:03Z

@Jeffwan Do you have any idea why the helm chart test failed? It looks the metadata healthz and readyz endpoints not accessible. I ran the same chart-testing locally and it all fine. Does it has anything to do with the image problem you saw in the metadata service migration PR?

Jeffwan · 2025-10-16T06:42:06Z

@zhangjyr the problem is the helm doesn't use the images build from this branch, it just use nightly image. this is a known issue and we should fix it, but we can ignore the problem at this moment

config/metadata/s3-env-patch.yaml

dist/chart/templates/metadata-service/rbac.yaml

Jeffwan

overall lgtm. I still have some questions on the permission

…ig/job is create for user to deploy job rbac. Signed-off-by: Jingyuan Zhang <[email protected]>

Jingyuan Zhang added 6 commits October 2, 2025 14:26

Simplified openai batch llm worker implementation without component i…

b1276c7

…ntegration Signed-off-by: Jingyuan Zhang <[email protected]>

envoy integration

375a42b

Signed-off-by: Jingyuan Zhang <[email protected]>

Fix envoy routes. Pass e2e test, and update document

cf173d8

Signed-off-by: Jingyuan Zhang <[email protected]>

Merge commit '66022321c8fca5fde82ea6e486cf87dbb8c3a0c2' into feature/…

cd1c8e7

…openai_batch_envoy_integration Signed-off-by: Jingyuan Zhang <[email protected]>

Lint fix

91fa1f6

Signed-off-by: Jingyuan Zhang <[email protected]>

zhangjyr requested review from Jeffwan and varungup90 October 15, 2025 17:25

Merge branch 'main' into feature/openai_batch_envoy_integration

1d3d8e5

gemini-code-assist bot reviewed Oct 15, 2025

View reviewed changes

Jeffwan reviewed Oct 15, 2025

View reviewed changes

config/gateway/gateway-plugin/gateway-plugin.yaml Show resolved Hide resolved

Jeffwan reviewed Oct 15, 2025

View reviewed changes

python/aibrix/.python-version Outdated Show resolved Hide resolved

Jingyuan Zhang added 6 commits October 15, 2025 14:21

Update document for storage configuration.

d328454

Cleanup files Signed-off-by: Jingyuan Zhang <[email protected]>

Explicit expose commands

c35aa2b

Signed-off-by: Jingyuan Zhang <[email protected]>

Fix comand line in helm chart.

36b01d2

Signed-off-by: Jingyuan Zhang <[email protected]>

use full command for aibrix_metadata

c59b1cb

Signed-off-by: Jingyuan Zhang <[email protected]>

Disable k8s-job in metadata service by default. For it requires objec…

d5d247c

…t store to work. Signed-off-by: Jingyuan Zhang <[email protected]>

Lint fix

9711d67

Signed-off-by: Jingyuan Zhang <[email protected]>

zhangjyr requested a review from Jeffwan October 16, 2025 06:33

Jeffwan reviewed Oct 16, 2025

View reviewed changes

config/metadata/s3-env-patch.yaml Show resolved Hide resolved

Jeffwan reviewed Oct 16, 2025

View reviewed changes

dist/chart/templates/metadata-service/rbac.yaml Show resolved Hide resolved

Jeffwan reviewed Oct 16, 2025

View reviewed changes

Metadata will not create serviceaccount for job anymore, and new conf…

23bfb48

…ig/job is create for user to deploy job rbac. Signed-off-by: Jingyuan Zhang <[email protected]>

Jeffwan merged commit 402521d into vllm-project:main Oct 17, 2025
13 of 14 checks passed

[Misc] Batch API envoy integration fix, E2E verification, and document update #1671

[Misc] Batch API envoy integration fix, E2E verification, and document update #1671

Uh oh!

Conversation

zhangjyr commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Related Issues

Pull Request Title Format

Submission Checklist

Uh oh!

gemini-code-assist bot commented Oct 15, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangjyr Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhangjyr commented Oct 16, 2025

Uh oh!

Jeffwan commented Oct 16, 2025

Uh oh!

Uh oh!

Uh oh!

Jeffwan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhangjyr commented Oct 15, 2025 •

edited

Loading

zhangjyr Oct 16, 2025 •

edited

Loading