feat: provide unified thinking config among anthropic and gemini models

[hustxiayang]

Description
thinking_config among anthropic and gemini models are similar, for example: thinking and thinking_config; budget_tokens and thinkingBudget. Our users do not need to set up different thinking configs for different models as we can provide a unified interface among different providers.

Related Issues/PRs (if applicable)

Related to #1463

[codecov-commenter]

Codecov Report

❌ Patch coverage is 50.72464% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.03%. Comparing base (9c7af75) to head (44b9efd).

Files with missing lines	Patch %	Lines
internal/apischema/openai/openai.go	0.00%	25 Missing ⚠️
internal/translator/openai_gcpvertexai.go	64.28%	3 Missing and 2 partials ⚠️
internal/translator/openai_awsbedrock.go	87.50%	1 Missing and 1 partial ⚠️
internal/translator/openai_gcpanthropic.go	85.71%	1 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (50.72%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1461      +/-   ##
==========================================
- Coverage   84.22%   84.03%   -0.20%     
==========================================
  Files         141      141              
  Lines       12978    13039      +61     
==========================================
+ Hits        10931    10957      +26     
- Misses       1428     1460      +32     
- Partials      619      622       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hustxiayang]

/retest

[mathetake]

so if i understand correctly, this goes completely beyond the original @sukumargaonkar 's proposal https://github.com/envoyproxy/ai-gateway/blob/main/docs/proposals/004-vendor-specific-fields/proposal.md

This is trying to define "AI Gateway Schema", and the scope and impact is huge. Could you write up a design doc and have more comments from community here?

At the very least, I would request for very detailed documents and future planning on this direction as well as this kind of thing requires user facing documentation in the website, not just code. sg?

[mathetake]

Please note that breaking change regarding this is extremely impactful than the control plane API. Once this is exposed, you cannot assume that clients will migrate properly, so we have to be very careful

[hustxiayang]

@mathetake Sure, thanks a lot for the comments! One key motivation is that we do not want to break the existing codes internally. Would prepare a doc also. Thanks!

[yuzisun]

so if i understand correctly, this goes completely beyond the original @sukumargaonkar 's proposal https://github.com/envoyproxy/ai-gateway/blob/main/docs/proposals/004-vendor-specific-fields/proposal.md

This is trying to define "AI Gateway Schema", and the scope and impact is huge. Could you write up a design doc and have more comments from community here?

At the very least, I would request for very detailed documents and future planning on this direction as well as this kind of thing requires user facing documentation in the website, not just code. sg?

@mathetake It is not against @sukumargaonkar's original proposal for vendor specific fields. However for thinking we found out that different providers have very similar looking definitions, so it is worth unifying at the gateway level. @hustxiayang will provider a detailed writeup.

[yuzisun]

Please note that breaking change regarding this is extremely impactful than the control plane API. Once this is exposed, you cannot assume that clients will migrate properly, so we have to be very careful

agreed, I think we can still keep the GCP thinking vendor field but recommend the new unified thinking API shape.

[mathetake]

ah sorry i should clarify that my comment about breaking change is about this new "unified api portion", not about vendor stuff already in place, and that's why I am saying we should not take this addition lightly without any very careful consideration.

[mathetake]

basically API added here in this PRs will never be reverted without very long migration period (think about openai/anthropic, or any other public API's deprecation window)

[yuzisun]

basically API added here in this PRs will never be reverted without very long migration period (think about openai/anthropic, or any other public API's deprecation window)

Adding an agenda tomorrow to discuss the deprecation policy.

[nacx]

Premature abstractions can come at a significant maintenance cost, so we need proper, informed decisions before pushing for them.

thinking_config among different providers are similar, for example: thinking and thinking_config; budget_tokens and thinkingBudget. Our users do not need to set up different thinking configs for different models as we can provide a unified interface among different providers.

This PR description is quite vague and doesn't give a good understanding on what will be covered in reality and the real user benefits. Please edit the description and add a detailed breakdown of the providers that are supported today and how their APIs look like. Let's include all all the currently supported providers, so we can get a good understanding on what this change would really cover (because not all of them support the same features through the OpenAI comapt APIs. For example, Cohere does not support the reasoning_effort field).

That would help. lot understand what this really covers and what it's optimizing for.

[mathetake]

yeah so i won't be able to make it to the meeting tomorrow (sorry 🙏) but as long as the deprecation policy is agreed upon in a sane way as well as the API design is reviewed by multiple people given the prior art research (not limited to litellm), then all good to me.

of course the user facing documentation is a must but that's after the agreement we can do that

[hustxiayang]

@mathetake @yuzisun @nacx It's been a while after the discussion. Any plans/concerns to merge this PR?

[mathetake]

@yuzisun @hustxiayang this needs the end user documentation change https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/vendor-specific-fields

[yuzisun]

@yuzisun @hustxiayang this needs the end user documentation change https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/vendor-specific-fields

Updated #1590