-
Notifications
You must be signed in to change notification settings - Fork 845
Description
OpenAI has recently added support for "custom" tools, where the input for the function isn't split across multiple schematized parameters but instead is just free-form text. These also support using a context-free grammar instead of a schema for guiding the model what that input can be.
We need to figure out the right way to expose this concept in M.E.AI. There are a few things we'll need to handle. Today:
AIFunction/AIFunctionDeclarationbakes in the notion that inputs are guided by a JSON schema.FunctionCallContentexpects a collection a parameters.- Neither
FunctionCallContentnorFunctionResultContentcarry with them an indication of whether they're tied to a normal function tool or a "custom" tool, but OpenAI differentiates, and would expect aFunctionResultContentproduced in response to a custom tool to be on the wire differently from a regular function tool.
We have a variety of options for how we could expose this, e.g.
- We could add a
string? FunctionContractorstring? Grammaror something like that onAIFunctionDeclaration, which could be used by a leaf client to indicate it's a custom thing.AIFunctionFactoryCreateOptionscould be augmented with one as well, and the resultingAIFunctionit produces would likely behave differently in itsInvokeAsync, such as by only allowing a singlestring/TextContent/etc. input parameter and sourcing it from a single parameter in theFunctionCallContent, regardless of the name of that entry.FunctionCallContentandFunctionResultContentcould be augmented with an additional property that indicates with which what kind of tool it's associated.FunctionInvokingChatClientwould be updated to know about the special-ness of certainAIFunctions (e.g. whether that property was set) and would tweak how it createsFunctionResultContent, or maybe it would just propagate the kind property from theFCCto theFRC, regardless of whatAIFunctionis used. - We could introduce a new hierarchy of tools, e.g.
AICustomFunction : AICustomFunctionDeclaration : AITool.FunctionInvokingChatClientwould be updated to be aware of it and know how to invoke it. We would still need updates toFunctionCallContent/FunctionResultContentas in (1). - Something so cool I don't know about it.
There are variations on all of these, too, e.g. we could utilize AdditionalProperties on the various content types rather than strongly-typed properties (but as everyone throughout the pipeline would need to agree on the meaning of things, strongly-typed properties are probably better.
We need to prototype this out end-to-end to determine the right course, and then do it. The .NET OpenAI library doesn't yet expose types for this, but it will in the near future. We should also investigate how other abstractions are handling this, and if other services like Gemini and Anthropic are introducing similar capabilities.