Skip to content

Conversation

@YoungVor
Copy link
Contributor

@YoungVor YoungVor commented Aug 8, 2025

TL;DR

Added documentation for contributing model support to the project.

What changed?

  • Added a new section "Adding model support" to the CONTRIBUTING.md file with detailed instructions for:
    • Adding language or embedding models to the model catalog
    • Adding model parameters to the Profile
    • Adding model-specific support for parameters
  • Updated the note about API keys to only mention OpenAI (removed Anthropic)

Copy link
Contributor Author

YoungVor commented Aug 8, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@YoungVor YoungVor changed the title Contributor documentation for adding models and params chore: Contributor documentation for adding models and params Aug 8, 2025
@YoungVor YoungVor force-pushed the 08-08-contributor_documentation_for_adding_models_and_params branch from c50791d to af14bbf Compare September 4, 2025 22:56

---

## Adding model support

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a bit more context on how inference works in fenic? I think it'd make sense to add a readme in the /_inference directory and link to it here.

Specifically it would be good to mention:

  • inference currently runs in a map_batches Polars udf
  • we have LanguageModel/EmbeddingModel classes that each own a ModelClient that the udfs use
  • ModelClients work though a producer/consumer mechanism - Polars threads produce requests, consumer threads submit them asynchronously in an event loop to maximize thoughput without spawning too many threads
  • ModelClients have RateLimitStrategies to make sure we don't overwhelm inference providers and exceed rate limits

It's also worth mentioning how each provider has its own set of embedding/language model configs set in the sessionconfig where we specify the model name, rate limits, and any special per model/provider functionality.

Lastly, I'd describe the nuances of token accounting and how an integration needs to fit in there. Would be useful to link some related PRs for past model integration to the guide.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely agree, there is a lot of detail around adding provider/model support and it deserves its own readme. That also means we can add more detail.

I'll take a second pass and add this detail as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants