Fix issue with async_scheduling when dealing with chunked input #359
Conversation
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Tianmu Li <[email protected]>
tianmu-li
force-pushed
the
async_scheduling_chunk_fix
branch
from
1f87a51 to
09c183d
Compare
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
tianmu-li
requested review from
mgawarkiewicz-intel,
piotrbocian and
wpyszka
as code owners
tianmu-li
changed the title
Signed-off-by: Tianmu Li <[email protected]>
Signed-off-by: Tianmu Li <[email protected]>
|
@afierka-intel Can you check if this fixes the issue with long context? It works from my tests. |
|
/run-gaudi-tests |
1 similar comment
|
/run-gaudi-tests |
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
|
/run-gaudi-tests |
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
3 participants
When dealing with chunked prompt (input sequence length > max_num_batched_tokens), sometimes there is no output token due to the chunked prompt, but the scheduler expects one. This PR addresses this issue and aligns the behavior with gpu_model_runner.