trl vllm server generating stuck

[AdaChambers]

Reproduction

I'm training Qwen3-1.7B with GRPO:

CUDA_VISIBLE_DEVICES=0 trl vllm-serve --host 127.0.0.1 --port 8014 --max_model_len 512 --model "Qwen/Qwen3-1.7B"
python train_grpo.py

os.environ["CUDA_VISIBLE_DEVICES"] = '2'
training_args = GRPOConfig(output_dir="Qwen3-1.7B-GRPO",
                            logging_steps=20,    
                            eval_strategy='steps',
                            save_strategy="steps",
                            save_steps=500,
                            num_train_epochs=2000,
                            
                            max_completion_length=512,
                        #    use_vllm=True,
                        #    bf16=True,
                           report_to=("wandb" if use_wandb else None),  
                            run_name=("sucai_1:1_all" if use_wandb else None),
                            per_device_train_batch_size=4,
                            per_device_eval_batch_size=4,
                            num_generations=4,
                            bf16=True,
                            gradient_accumulation_steps=4,
                            beta=0.004,

                            use_vllm=True,
                            vllm_mode="server",
                            # vllm_mode="colocate"
                            vllm_server_host='127.0.0.1',
                            vllm_server_port=8014,
                            # vllm_gpu_memory_utilization=0.3,
                            # vllm_guided_decoding_regex=r'<think>(.*?)</think><answer>(.*?)</answer>'
                           )

training_args.model_name=model_name
training_args.train_data_num=len(ds_train)
training_args.test_num=len(ds_test)
trainer = GRPOTrainer(
    model=model,
    reward_funcs=reward_len,
    args=training_args,
    train_dataset=ds_train,
    eval_dataset=ds_test,
)
trainer.train()

After init vllm client:

self.vllm_client = VLLMClient(
                    args.vllm_server_host, args.vllm_server_port, connection_timeout=args.vllm_server_timeout
                )
self.vllm_client.init_communicator()

It got stuck when it started generating post and didn't report an error both on the client or on the server:

url = f"http://{self.host}:{self.server_port}/generate/"
prompts=['xx','xx','xx','xx']
n=4
temperature=0.9
repetition_penalty=1.0
top_p=1.0
top_k=50
min_p=0
max_tokens=512
guided_decoding_regex=None
response = self.session.post(
    url,
    json={
        "prompts": prompts,
        "n": n,
        "repetition_penalty": repetition_penalty,
        "temperature": temperature,
        "top_p": top_p,
        "top_k": top_k,
        "min_p": min_p,
        "max_tokens": max_tokens,
        "guided_decoding_regex": guided_decoding_regex,
    },
)

no error is reported:
server:

INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:35524 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO 05-19 16:17:14 [block_pool.py:264] Successfully reset prefix cache
INFO:     127.0.0.1:35524 - "POST /reset_prefix_cache/ HTTP/1.1" 200 OK
<<stuck here>>

client:

 0%|                | 0/6152000 [00:00<?, ?it/s]

I've tried some other port. However nothing changed.

System Info

• Platform: Linux-5.4.0-169-generic-x86_64-with-glibc2.31
• Python version: 3.10.16
• TRL version: 0.18.0.dev0+4da4dc9
• PyTorch version: 2.6.0
• CUDA device(s): NVIDIA H20, NVIDIA H20, NVIDIA H20, NVIDIA H20, NVIDIA H20, NVIDIA H20, NVIDIA H20, NVIDIA H20
• Transformers version: 4.51.3
• Accelerate version: 1.6.0
• Accelerate config: not found
• Datasets version: 3.2.0
• HF Hub version: 0.31.1
• bitsandbytes version: 0.45.5
• DeepSpeed version: 0.16.7
• Diffusers version: not installed
• Liger-Kernel version: not installed
• LLM-Blender version: not installed
• OpenAI version: 1.78.1
• PEFT version: 0.15.2
• vLLM version: 0.8.5.post1

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete

[AdaChambers]

'Colocate' mode worked well.

[shirinyamani]

Hi @AdaChambers, thanks for reporting. Its surprising that it hangs for generation, bc looking at your code, I see you define two gpus, 2 for training and 0 for vllm.
One question for clarification sake, why are you initiating the Client separately?
Bc in the grpo trainer if you set the use_vllm=True in case vllm is available (which it is in your case) it will automatically init the client. The corresponding code is here

[shirinyamani]

Also I'm wondering for the colocate mode, did you use the same setup?
bc to the best of my knowledge, the major difference between these two modes is that in server mode, the vllm will sit on a totally different device than the device where training is happening, therefore, they do not share memory/device, while in colocate mode it sit on the same device as the client trainer.

[AdaChambers]

Hi @AdaChambers, thanks for reporting. Its surprising that it hangs for generation, bc looking at your code, I see you define two gpus, 2 for training and 0 for vllm. One question for clarification sake, why are you initiating the Client separately? Bc in the grpo trainer if you set the use_vllm=True in case vllm is available (which it is in your case) it will automatically init the client. The corresponding code is here

Hi @shirinyamani , thanks for the reply! Haha, the client initiating code is in trl pack. And the session post code is in vllm pack. I debugged into the packs and copy the code here.

[AdaChambers]

Also I'm wondering for the colocate mode, did you use the same setup? bc to the best of my knowledge, the major difference between these two modes is that in server mode, the vllm will sit on a totally different device than the device where training is happening, therefore, they do not share memory/device, while in colocate mode it sit on the same device as the client trainer.

I set vllm_gpu_memory_utilization=0.3 in the colocate mode but not in the server mode. Does that matter?

[shirinyamani]

Reproduction

I'm training Qwen3-1.7B with GRPO:

CUDA_VISIBLE_DEVICES=0 trl vllm-serve --host 127.0.0.1 --port 8014 --max_model_len 512 --model "Qwen/Qwen3-1.7B"
python train_grpo.py

@AdaChambers One thing I just realized by re-reading your command, this command is actually not correct!
trl vllm-serve is only for initializing the server, you cannot input your python train_grpo.py here to the best of knowledge.

I tested the following from my end on server-mode on two gpus and it successfully worked!
Here is what I did;

to run the server I used this command, which successfully initialized the server:

CUDA_VISIBLE_DEVICES=0 trl vllm-serve  --max_model_len 512 --model "Qwen/Qwen3-1.7B"

for training script I used the following:(please note bc I was unsure abt your train and eval dataset I commented them out but else is identical to your config except for the host n port):

from datasets import load_dataset
from trl import GRPOTrainer, GRPOConfig

dataset = load_dataset("trl-lib/tldr", split="train[:1%]")

# Dummy reward function: count the number of unique characters in the completions
def reward_num_unique_chars(completions, **kwargs):
    return [len(set(c)) for c in completions]

training_args = GRPOConfig(
    output_dir="issue-3467",
    logging_steps=20,    
    #eval_strategy='steps',
    # save_strategy="steps",
    # save_steps=500,
    num_train_epochs=2000,
    max_completion_length=512,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_generations=4,
    bf16=True,
    gradient_accumulation_steps=4,
    beta=0.004,
    use_vllm=True,
    vllm_mode="server",
)

trainer = GRPOTrainer(
    model="Qwen/Qwen3-1.7B",
    args=training_args,
    reward_funcs=reward_num_unique_chars,
    train_dataset=dataset,
)

trainer.train()

For running the train script I used:

 CUDA_VISIBLE_DEVICES=1 accelerate launch issue_3467.py

And this is a part of the server output as successful processing;

INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:52740 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO 05-22 21:22:01 [block_pool.py:264] Successfully reset prefix cache
INFO:     127.0.0.1:52740 - "POST /reset_prefix_cache/ HTTP/1.1" 200 OK
Processed prompts: 100%|█████████| 16/16 [00:00<00:00, 18.88it/s, est. speed input: 6849.10 toks/s, output: 2817.96 toks/s]
INFO:     127.0.0.1:52740 - "POST /generate/ HTTP/1.1" 200 OK

[shirinyamani]

I set vllm_gpu_memory_utilization=0.3 in the colocate mode but not in the server mode. Does that matter?

In colocate model bc client and the vllm are fed from the same device (i.e. sitting on same device, sharing memory) so it makes more sense to define the memory usage for vllm, cuz same memory is being shared with the client trainer.

[sravan500]

@shirinyamani Nice information above.
I am running GRPO using multinode. 8 H100 * 2Nodes.
I need to start the train.py in each node. How Do i integrate vllm to that.
Example my script is train.py.
Do I need to start vllm server on each node separately.
Please help.