Skip to content

Conversation

sunlei1024
Copy link
Collaborator

@sunlei1024 sunlei1024 commented Sep 26, 2025

描述:
本 PR 为 MTP 的 Speculative Decode MTP 功能增加 draft_logprobs 支持,并在 OpenAI 兼容接口上做了对应扩展。

主要改动:

  1. 新增请求参数 include_draft_logprobs

    • /chat/completions/completions 接口中,用户可以通过该参数控制是否返回 draft_logprobs
  2. 新增响应参数 draft_logprobs

    • include_draft_logprobstrue 时,响应中会包含 draft_logprobs,用于记录推测解码过程中中间的概率值。
  3. 改动说明: 非Speculative Decode MTP 核心解码逻辑保持不变,与现有解码流程兼容。

目的:

  • 增强对 Speculative Decode 的可观测性和调试能力。
  • 完善 OpenAI 兼容接口,方便用户在调试或实验中获取中间预测概率信息。

示例请求 (curl):

  1. /completions 接口
curl https://{ip}:{port}/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model",
    "prompt": "Hello, world!",
    "logprobs": 5,
    "include_draft_logprobs": true
  }'
  1. /chat/completions 接口
curl https://{ip}:{port}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello, world!"}
    ],
    "logprobs": true,
    "top_logprobs": 5,
    "include_draft_logprobs": true
  }'

示例响应片段:

{
  "id": "cmpl-xxx",
  "object": "text_completion",
  "choices": [
    {
      "text": "Hello",
      "logprobs": [ ... ], 
      "draft_logprobs": [ ... ]
    }
  ]
}
{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Hello",
        "logprobs": [ ... ], 
        "draft_logprobs": [ ... ]
      }
    }
  ]
}

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ sunlei1024
❌ sunlei18


sunlei18 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants