Skip to content

Support prompt hidden states #24202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Support prompt hidden states #24202

wants to merge 2 commits into from
+188 −2

Conversation

@charlotte12l
Copy link
Contributor

@charlotte12l charlotte12l commented Sep 3, 2025 •
edited by github-actions bot
Loading

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@mergify mergify bot added v1 tpu Related to Google TPUs labels Sep 3, 2025
@ywang96 ywang96 self-assigned this Sep 3, 2025
luccafong
luccafong reviewed Sep 4, 2025
View reviewed changes
vllm/v1/worker/gpu_model_runner.py

return prompt_logprobs_dict

def _get_prompt_hidden_states_dict(
Copy link
Collaborator

@luccafong luccafong Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try extact to separate utils.py?

luccafong
luccafong reviewed Sep 4, 2025
View reviewed changes
vllm/v1/worker/gpu_model_runner.py
hidden_states[:num_scheduled_tokens],
scheduler_output.num_scheduled_tokens,
)
prompt_hidden_states_dict = self._get_prompt_hidden_states_dict(
Copy link
Collaborator

@luccafong luccafong Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make this an option by default disabled from vllm serve arguments

@charlotte12l
clean comments
d0944b2 
Signed-off-by: Xingyu Liu <[email protected]>
@charlotte12l charlotte12l mentioned this pull request Sep 5, 2025
Open
1 task
@mergify
Copy link

mergify bot commented Sep 8, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @charlotte12l.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@luccafong luccafong luccafong left review comments

@NickLucche NickLucche Awaiting requested review from NickLucche NickLucche will be requested when the pull request is marked ready for review NickLucche is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon will be requested when the pull request is marked ready for review WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat will be requested when the pull request is marked ready for review robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill will be requested when the pull request is marked ready for review njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 will be requested when the pull request is marked ready for review ywang96 is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat will be requested when the pull request is marked ready for review alexm-redhat is a code owner

@heheda12345 heheda12345 Awaiting requested review from heheda12345 heheda12345 will be requested when the pull request is marked ready for review heheda12345 is a code owner

@ApostaC ApostaC Awaiting requested review from ApostaC ApostaC will be requested when the pull request is marked ready for review ApostaC is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@ywang96 ywang96

Labels

needs-rebase tpu Related to Google TPUs v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@charlotte12l @luccafong @ywang96