-
Notifications
You must be signed in to change notification settings - Fork 521
feat: support zai as inference provider #1766
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
we need to allowlist your org on the Hub side. (we can add a note about this to the doc maybe @SBrandeis @Wauplin) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haven't tested yet but implementation looks good. Left a comment regarding text-generation vs chat-completion.
export class ZaiTextGenerationTask extends BaseTextGenerationTask { | ||
constructor() { | ||
super("zai", ZAI_API_BASE_URL); | ||
} | ||
|
||
override prepareHeaders(params: HeaderParams, binary: boolean): Record<string, string> { | ||
const headers = super.prepareHeaders(params, binary); | ||
headers["x-source-channel"] = "hugging_face"; | ||
return headers; | ||
} | ||
|
||
override makeRoute(): string { | ||
return "/chat/completions"; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The text-generation
task is meant for non-conversational LLMs. Do you plan to serve any? Usually Inference Providers are starting with conversational ones only i.e. "chat completion".
Note that a model cannot be served through Inference Providers both with the Chat Completion and the Text Generation APIs. If a model is tagged as "conversational" on the Hub (https://huggingface.co/models?other=conversational&sort=trending&search=zai-org), then it must be served as Chat Completion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you do want to serve some non-conversational text-generation models, then I suppose the API route must be updated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi @Wauplin Thanks for the reminder. We’re providing conversational models, I’ll update the pr.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi, i have removed the text-generation
. PTAL, thanks
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
hi @julien-c , could you help add our team zai-org to the allowlist? we would like to register the provider as |
we try to have provider id === organization namespace on the Hub Would you rather have zai-org as the provider id? Or we can look into moving your org to hf.co/zai Let me and @SBrandeis know |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks @tomsun28! this looks good
Co-authored-by: célina <[email protected]>
hi @julien-c @SBrandeis , got it, Is this mandatory? If zai cannot be used without changing the organization, we can use zai-org as the provider. Thanks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, overseen this
Took https://github.com/tomsun28/huggingface.js/blob/3a4b909e563ef7e86ffc600045e22be25f0579b9/packages/inference/src/providers/novita.ts for reference
Co-authored-by: Simon Brandeis <[email protected]>
Co-authored-by: Simon Brandeis <[email protected]>
hi @SBrandeis Thanks, the base_url has been updated. |
@SBrandeis Hi and could you help add us to the allowlist? so we can proceed with the model mapping registration process. |
Hey @tomsun28
Merging this PR is a pre-requisite before we can enable the model mapping API
It would be nice if we could match the provider name with the name of the org yes! |
Got it thanks, i will update the provider id. |
@tomsun28 you sure you don't want us to rename your org to hf.co/zai instead (all links will be redirected)? |
Hi @julien-c @SBrandeis the provider id has updated to zai-org, PTAL thanks |
Hi 🤗 team,
We’d like to apply to register ZAI as inference provider. Thanks
One more question: when we tried to register the model mapping with the endpoint
POST api/partners/zai/models
, we received a 404 response. https://huggingface.co/docs/inference-providers/register-as-a-provider#3-model-mapping-api Does this action only work after the PR has been merged?