Inline tag style prompting for tool-calling

[uogbuji]

OpenAI tool-calling is out-of-line from the actual prompt & response text. The available tool listings, tool-call requests & tool results com in through separate structures of the API JSON.

I was trying out mlx-community/Llama-3-Groq-70B-Tool-Use-4bit today and it looks as if it uses a different approach, where special tokens and XML-like tags are used within the accumulated prompt, so for example (source):

<|start_header_id|>system<|end_header_id|>

You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
<tool_call>
{"name": <function-name>,"arguments": <args-dict>}
</tool_call>

Here are the available tools:
<tools> {
    "name": "get_current_weather",
    "description": "Get the current weather in a given location",
    "parameters": {
        "properties": {
            "location": {
                "description": "The city and state, e.g. San Francisco, CA",
                "type": "string"
            },
            "unit": {
                "enum": [
                    "celsius",
                    "fahrenheit"
                ],
                "type": "string"
            }
        },
        "required": [
            "location"
        ],
        "type": "object"
    }
} </tools><|eot_id|><|start_header_id|>user<|end_header_id|>

What is the weather like in San Francisco?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

<tool_call>
{"id":"call_deok","name":"get_current_weather","arguments":{"location":"San Francisco","unit":"celsius"}}
</tool_call><|eot_id|><|start_header_id|>tool<|end_header_id|>

<tool_response>
{"id":"call_deok","result":{"temperature":"72","unit":"celsius"}}
</tool_response><|eot_id|><|start_header_id|>assistant<|end_header_id|>

Indeed when I try the square root example with it, it wants to use this pattern:

echo 'What is the square root of 256?' > /tmp/llmprompt.txt
echo '{"tools": [{"type": "function","function": {"name": "square_root","description": "Get the square root of the given number","parameters": {"type": "object", "properties": {"square": {"type": "number", "description": "Number from which to find the square root"}},"required": ["square"]},"pyfunc": "math|sqrt"}}], "tool_choice": "auto"}' > /tmp/toolspec.json
toolio_request --apibase="http://localhost:8000" --prompt-file=/tmp/llmprompt.txt --tools-file=/tmp/toolspec.json --trace
⚙️Calling tool square_root with args {'square': 256}
⚙️Tool call result: 16.0
Final response:
<|start_header_id|>tool<|end_header_id|>

<tool_response>
{"result":16.0}
</tool_response>

I think this might require some thought, and perhaps a special, steered sampling mode that understands those tool-calling tags in addition to JSON.

The interesting thing is that Groq credits Nous Research right before that example, but mlx-community/Hermes-2-Theta-Llama-3-8B, straight from Nous, works pretty well with Toolio as-is.