Skip to content

[serve] Forward tool_calls/tool_call_id in processor inputs#45418

Open
qgallouedec wants to merge 1 commit intomainfrom
forward-tool-use
Open

[serve] Forward tool_calls/tool_call_id in processor inputs#45418
qgallouedec wants to merge 1 commit intomainfrom
forward-tool-use

Conversation

@qgallouedec
Copy link
Copy Markdown
Member

  • get_processor_inputs_from_messages builds a new dict per message with only role and content, dropping tool_calls and tool_call_id. This means apply_chat_template can't properly render multi-turn tool-use conversations.
  • Forward these fields so the chat template sees the full tool-use context.

Example

Multi-turn tool-use conversation sent to transformers serve:

from transformers.cli.serving.utils import BaseHandler, Modality

messages = [
    {"role": "user", "content": "What's the weather in Paris?"},
    {"role": "assistant", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": '{"city": "Paris"}'}}]},
    {"role": "tool", "content": "22°C, sunny", "tool_call_id": "call_1"},
]
print(BaseHandler.get_processor_inputs_from_messages(messages, Modality.LLM))

Before this PR, tool_calls and tool_call_id were dropped:

[
    {'role': 'user', 'content': "What's the weather in Paris?"},
    {'role': 'assistant', 'content': []},
    {'role': 'tool', 'content': '22°C, sunny'}
]

After this PR, they are forwarded to apply_chat_template:

[
    {'role': 'user', 'content': "What's the weather in Paris?"},
    {'role': 'assistant', 'content': [], 'tool_calls': [{'id': 'call_1', 'type': 'function', 'function': {'name': 'get_weather', 'arguments': '{"city": "Paris"}'}}]},
    {'role': 'tool', 'content': '22°C, sunny', 'tool_call_id': 'call_1'}
]

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants