Description:

When using the Ultravox RealtimeModel with MCP (Model Context Protocol) tools or any function calling that requires strict type validation, all tool parameters are being converted to strings regardless of their actual types defined in the tool schema. This causes MCP servers and other type-strict tools to reject the function calls with type validation errors.

Environment:

• LiveKit Agents Version: Latest (as of 2025-01-24)
• LiveKit Plugins Ultravox Version: Latest (as of 2025-01-24)
• Python Version: 3.11+
• OS: Windows 10
• MCP Integration: Yes (HTTP-based MCP server with strict type validation)

Expected Behavior:

Tool parameters should preserve their JSON types as defined in the tool schema:

• Arrays should remain as arrays: ["low_stock"]
• Numbers should remain as numbers: 10
• Booleans should remain as booleans: true
• Null values should remain as null: null

Actual Behavior:

All tool parameters are converted to strings, with nested JSON being double-escaped:

• Arrays become strings: "[\"low_stock\"]" (double-escaped)
• Numbers become strings: "10"
• Booleans become strings: "true"
• Null values become strings: "null"

Steps to Reproduce:

1. Set up an MCP server with a tool that has typed parameters:

from livekit.agents import Agent, AgentSession, mcp
from livekit.plugins import ultravox, silero

# Example tool with typed parameters
# Tool expects: alert_type (array), limit (number), include_valuation (boolean)
agent = Agent(
    instructions="You are a helpful assistant with access to tools.",
    mcp_servers=[
        mcp.MCPServerHTTP(
            url="http://localhost:8000/mcp",  # MCP server with strict type validation
            timeout=30.0,
        )
    ],
)

2. Configure AgentSession with Ultravox RealtimeModel:

session = AgentSession(
    vad=silero.VAD.load(),
    llm=ultravox.realtime.RealtimeModel(
        system_prompt="You are a helpful assistant. Use tools to answer user queries.",
        temperature=0.5,
    ),
)

3. Connect to room and start session:

await session.start(room=ctx.room, agent=agent)

4. Trigger a tool call that requires typed parameters (e.g., "Check inventory status for low stock items")

5. Observe the MCP validation errors in logs

Error Logs:

2025-10-24 13:58:27,693 - ERROR livekit.agents - exception occurred while executing tool {"function": "getInventoryStatus", "speech_id": "speech_6a5f536fad01"}
Traceback (most recent call last):
  File "<venv>/site-packages/livekit/agents/voice/generation.py", line 526, in _traceable_fnc_tool
    val = await function_callable()
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<venv>/site-packages/livekit/agents/llm/mcp.py", line 100, in _tool_called
    tool_result = await self._client.call_tool(name, raw_arguments)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<venv>/site-packages/mcp/client/session.py", line 279, in call_tool
    result = await self.send_request(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<venv>/site-packages/mcp/shared/session.py", line 286, in send_request
    raise McpError(response_or_error.error)
mcp.shared.exceptions.McpError: MCP error -32602: Invalid arguments for tool getInventoryStatus: [
  {
    "code": "invalid_type",
    "expected": "array",
    "received": "string",
    "path": ["alert_type"],
    "message": "Expected array, received string"
  },
  {
    "code": "invalid_type",
    "expected": "number",
    "received": "string",
    "path": ["limit"],
    "message": "Expected number, received string"
  },
  {
    "code": "invalid_type",
    "expected": "boolean",
    "received": "string",
    "path": ["include_valuation"],
    "message": "Expected boolean, received string"
  },
  {
    "code": "invalid_type",
    "expected": "number",
    "received": "string",
    "path": ["category_id"],
    "message": "Expected number, received string"
  }
]

Tool call arguments as logged:

{
  "connection": {
    "password": "admin",
    "db": "test-db",
    "context": {},
    "uid": 11,
    "url": "http://example.com"
  },
  "alert_type": "[\"low_stock\"]",      // ❌ Should be: ["low_stock"]
  "category_id": "null",                 // ❌ Should be: null
  "warehouse_id": "null",                // ❌ Should be: null
  "include_valuation": "true",           // ❌ Should be: true
  "limit": "10"                          // ❌ Should be: 10
}

Additional Context:

According to the LiveKit Ultravox plugin documentation:

"The plugin does not provide explicit type validation for tool parameters. Parameters are passed as raw dictionaries from the Ultravox API and serialized to JSON strings..."

The tool invocation flow in the Ultravox plugin:

1. Ultravox sends ClientToolInvocationEvent with parameters dictionary
2. LiveKit serializes it: arguments=json.dumps(event.parameters)
3. No type coercion/validation occurs

This suggests the issue originates from the Ultravox API sending parameters as strings, which are then JSON-serialized by LiveKit, resulting in double-stringification for complex types like arrays.

Comparison with OpenAI Realtime Model:

When using openai.realtime.RealtimeModel() with the exact same MCP tools and agent configuration, tool parameters are correctly typed and MCP tool calls succeed without type validation errors. This confirms the issue is specific to the Ultravox integration.

Working example with OpenAI:

llm=openai.realtime.RealtimeModel(
    voice="alloy",
    temperature=0.5,
)
# Tool calls succeed with proper types: arrays, numbers, booleans, null

Impact:

This bug makes Ultravox RealtimeModel incompatible with:

• ❌ MCP (Model Context Protocol) servers with strict type validation
• ❌ Any function calling system that validates parameter types against schemas
• ❌ Tools that require arrays, numbers, booleans, or null values
• ❌ Integration with type-strict APIs (REST APIs, database queries, etc.)

This significantly limits the usability of Ultravox RealtimeModel in production environments where type safety is critical.

Root Cause Analysis:

The issue appears to be in the parameter serialization chain:

# In ultravox plugin (inferred from docs):
# When ClientToolInvocationEvent arrives:
llm.FunctionCall(
    arguments=json.dumps(event.parameters)  # ← Problem: event.parameters already stringified
)

If event.parameters from Ultravox API contains:

{
    "alert_type": "[\"low_stock\"]",  # Already a string
    "limit": "10"                      # Already a string
}

Then json.dumps() will preserve these as strings in the final JSON.

Suggested Fix:

Option 1: Type Coercion in LiveKit Plugin (Preferred)

The parse_tools() function already extracts type information from schemas. Use this to deserialize parameters:

def _coerce_parameter(value: Any, param_schema: dict) -> Any:
    """Coerce parameter value to match schema type."""
    param_type = param_schema.get("type")

    if param_type == "array" and isinstance(value, str):
        return json.loads(value)  # Parse stringified arrays
    elif param_type == "number" and isinstance(value, str):
        return float(value) if "." in value else int(value)
    elif param_type == "boolean" and isinstance(value, str):
        return value.lower() == "true"
    elif param_type == "null" and value == "null":
        return None
    return value

Option 2: Fix in Ultravox API

If the issue originates from Ultravox's API stringifying parameters, this should be fixed upstream.

Workaround:

Currently, the only workaround is to use OpenAI's Realtime Model instead:

llm=openai.realtime.RealtimeModel(
    voice="alloy",
    temperature=0.5,
)

Related Issues:

• Ultravox plugin: precomputable tools break audio output #3659 - Ultravox plugin: precomputable tools break audio output
• Tool Call Results Lost During User Interruption Leading to Duplicate Executions #3702 - Tool Call Results Lost During User Interruption

Additional Information:

• Tested with multiple MCP tools (inventory, projects, products) - all exhibit the same type coercion issue
• Issue is 100% reproducible with any MCP tool that has non-string parameters
• No issues when using traditional pipeline (STT + LLM + TTS) with MCP tools
• Only affects Ultravox RealtimeModel specifically