[FEATURE]: Add --mm-prompt-template support to SGLang for multimodal prompt customization

[nancya-nv]

Feature request

Description

Add --mm-prompt-template argument to SGLang to achieve feature parity with vLLM

Describe the problem you're encountering

Problem

Currently, vLLM (args.py:165) supports --mm-prompt-template for customizing multimodal prompts, but SGLang does not. This means:

• SGLang users cannot customize how images and text are combined in prompts
• Users migrating from vLLM to SGLang lose this functionality
• Different vision models may require different prompt formats

Proposed Solution

Add --mm-prompt-template argument to SGLang that:

• Accepts a template string with <prompt> and <image> placeholders
• Replaces <prompt> with user text
• Replaces <image> with the model's image token
• Has a sensible default: "USER: <image>\n<prompt> ASSISTANT:"

Example Usage

python -m dynamo.sglang --model Qwen/Qwen2.5-VL-7B-Instruct
--multimodal-processor
--mm-prompt-template "Analyze \n\nUser: \n\nAssistant:"## Implementation

This requires changes to:

1. components/src/dynamo/sglang/args.py - Add argument definition
2. components/src/dynamo/sglang/main.py - Pass to handler
3. components/src/dynamo/sglang/request_handlers/multimodal/processor_handler.py - Accept template
4. components/src/dynamo/sglang/multimodal_utils/multimodal_chat_processor.py - Use template
5. components/src/dynamo/sglang/tests/test_sglang_unit.py - Add tests

References

• vLLM implementation: components/src/dynamo/vllm/args.py:165-176
• TensorRT-LLM implementation: components/src/dynamo/trtllm/utils/trtllm_utils.py:278

I Can Work On This

I'm ready to implement this feature and submit a PR if approved.

Describe alternatives you've tried

No response

[indrajit96]

Hi @nancya-nv thanks for reaching out.
I would recommend first experimenting with the sglang backend itself.

1. How does MM support behave with different models currently. AFAIK the chat_template in the sglang should already take care of this when chat_template is applied here
2. Can you document how sglang_request gets set for different models? If it already gets formatted depending on the model and we are able to inference correct we should be good.
3. SGLang relies on --chat-template (uses SGLang's built-in conversation templates). The template automatically inserts conv.image_token for images. (This is my understanding, I make be wrong)
4. Here are models you should try with that are already supported in vllm
   LLAVA_1_5_7B = "llava-hf/llava-1.5-7b-hf" QWEN_2_5_VL_7B = "Qwen/Qwen2.5-VL-7B-Instruct" LLAVA_NEXT_VIDEO_7B = "llava-hf/LLaVA-NeXT-Video-7B-hf"

IF inference is broken for above models then your suggestion makes sense and we can have a new --mm-prompt-template arg for sglang