Multi-turn tool calling support

[qgallouedec]

from datasets import Dataset
from trl import GRPOTrainer, GRPOConfig
import random

factors = [(random.randint(0, 9999), random.randint(0, 9999)) for _ in range(100)]
dataset = Dataset.from_dict(
    {
        "prompt": [[{"role": "user", "content": f"Multiply {a} and {b}."}] for a, b in factors],
        "result": [a * b for a, b in factors],
    }
)


def multiply(a: int, b: int) -> int:
    """Multiply two integers.

    Args:
        a: The first integer.
        b: The second integer.

    Returns:
        The product of the two integers.
    """
    return a * b

def accuracy_reward(completions, result, **kwargs):
    return [int(str(r) in c[0]["content"]) for c, r in zip(completions, result)]


trainer = GRPOTrainer(
    model="Qwen/Qwen3-0.6B",
    args=GRPOConfig(use_vllm=True, vllm_mode="colocate", vllm_importance_sampling_correction=False),
    reward_funcs=accuracy_reward,
    train_dataset=dataset,
    tools=[multiply],
)
trainer.train()

[lewtun]

There is unfortunately no standardisation for the tool call tags across model families, but Matt is working on extending the chat templates so they can auto-parse tools calls (internal Slack thread): https://huggingface.slack.com/archives/C06JKEMK6BZ/p1757691450090859

Note that vllm works around this by providing a dedicated set of parsers that can be set when spinning up the server: https://docs.vllm.ai/en/stable/features/tool_calling.html

I'm not sure we want to go down this route, since it's quite messy in my experience to match the parser to the desired model (e.g. some Qwen models use the hermes parser, others not)

So in the meantime, we might want to give uses the ability to provide their own parsing function and default to yours (which is the most common I've seen)

[lewtun]

Perfect timing :)

https://moon-ci-docs.huggingface.co/docs/transformers/pr_40894/en/chat_response_parsing

[albertvillanova]

That is the approach we usually followed on smolagents: to provide a sensible default, but allow users to fully customize the function/object instance.

[lewtun]

Would be cool to have a unit test for this masking so we're confident it is behaving as expected

[albertvillanova]

Awesome!! Looking forward to have this feature!

Some comments, suggestions and question below.

[albertvillanova]

That is the approach we usually followed on smolagents: to provide a sensible default, but allow users to fully customize the function/object instance.

[albertvillanova]

What about moving this function to a non-specific trainer module, so it can be used by any trainer in the future?

[albertvillanova]

Note that the compile function will be called at each function call. If optimizing performance is necessary in this case, we should set the compilation as a constant at the module level, so it is called only once at import time.

[albertvillanova]

You will need to add the tools param to the trainer docstring. And give a type hint.

[albertvillanova]

You only return the first match?

[albertvillanova]

Not sure of this handles potential multiple tool calls in a single completion...

[albertvillanova]

Before the messages with the tool results ("role": "tool"), shouldn't we prepend the messages with the tool calls themselves ("role": "assistant", "tool_calls":...)? Not sure of this though... A real question! 😅

[August-murr]

I suggest that we remove any references to including tools as a parameter in the GRPOTrainer:

trainer = GRPOTrainer(…,
tools = [tool_1, tool_2]
)

While I am not necessarily against chat templates or tool parsing implementations, I believe the attribute tools should be eliminated.

To maximize scalability, a better approach would be to create an all-in-one tools sandbox. This means that the only tool the training script interacts with would be the sandboxed code executors, with all the necessary tools defined within it, like using a Docker image that contains the dependencies of the tools to initialize the sandbox

Therefore, I propose that we only add the parameter Environment, which would encompass the code executor along with all initialized tools.

The environment needs to be responsible for handling tool usage and execution.

Then you could create your own built-in environments using your preferred parsing and chat templates.

[qgallouedec]

Yes, I have seen some works that suggest that when scaling up, multiplying tools works less well than a smaller, more generic set of tools, such as the one you describe. However, it seems to me that the approach proposed here is in fact compatible:

trainer = GRPOTrainer(…,
    tools = [my_big_containerized_all_in_one_tool],
)

in the end it's up to the user decide how to design it, most important here is to allow for this flexibility

[August-murr]

but the name tools as a param in GRPOTrainer is misleading.
What I suggest is more like

MyEnv = DefaultEnv(code_executer=my_big_containerized_all_in_one_tool)

trainer = GRPOTrainer(…,
Environment=MyEnv
)

The user can then customize tool use in their own environment.

[qgallouedec]

the name tool is actually quite standard now:

https://huggingface.co/blog/unified-tool-use

https://huggingface.co/docs/transformers/main/chat_extras

https://huggingface.co/docs/transformers/main/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.apply_chat_template.tools