Fix issue with async_scheduling when dealing with chunked input #360
Conversation
Signed-off-by: Tianmu Li <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
tianmu-li
changed the title
tianmu-li
requested review from
adobrzyn,
afierka-intel,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski,
mswiniarsk,
vivekgoe and
xuechendi
as code owners
Signed-off-by: Tianmu Li <[email protected]>
tianmu-li
force-pushed
the
async_scheduling_chunk_fix_main
branch
from
e2cc7ce to
f075944
Compare
| if structured_output or self.use_async_scheduling: | ||
| logits_append = torch.tensor([torch.sum(prompt_len) - 1], | ||
| device=token_ids.device, | ||
| dtype=torch.int32) |
There was a problem hiding this comment.
I didn't get this part, why torch.sum(prompt_len) - 1 instead something like len(req_id) - logits_indices.shape[0]
There was a problem hiding this comment.
Why logits_indices is shorter, because we skipped num_decodes right? Why padding is append after not before?
There was a problem hiding this comment.
This is for chunked prompt, where it shouldn't generate a new token yet. In gpu_model_runner, a new token still gets generated for the incomplete prompt and then gets discarded. This is to align to that behavior.
There was a problem hiding this comment.
This depends on the fact that there can only be one incomplete prompt, and that prompt is always the last one if it exists.
| if self.use_async_scheduling: | ||
| # Discard partial prefill logit for async scheduling | ||
| # Depends on 1 decode token/batch | ||
| invalid_req_indices.append(num_decodes + idx) |
There was a problem hiding this comment.
maybe do something as, so it will be easier to understand?
prefill_start_idx = num_decodes
invalid_req_indices.append(prefill_start_idx + idx)
There was a problem hiding this comment.
Added clarification.
Signed-off-by: Tianmu Li <[email protected]>
|
/run-gaudi-tests |
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
3 participants
Cherry-pick of #359