[FEATURE]: Multi-Tenancy

[nithin8702]

Feature request

We would like to Serve Single Model for Multiple Projects using api keys. Each Project should have unique api keys. Also capture the tokens for each Projects or API Keys.

Describe the problem you're encountering

In an Enterprise, we are duplicating instances for each project, Causing lesser GPU Utilisation.

Describe alternatives you've tried

LiteLLM Proxy Server which has Teams, Projects, Org. It also has good token capturing.

[rmccorm4]

Hi @nithin8702, thanks for raising this issue. CC @athreesh @itay @hutm for viz

[nnshah1]

One question: is the goal here for monitoring usage metrics by different projects or for ensuring QOS / SLA?

[nithin8702]

Hi

In an Pharmaceutical Enterprise, we will have different LOB like R&D, Oncology, BioPharma, R&I, Commercial, GBS, etc. Under each LOB we will different Teams/Projects.

Here each API Key is Mapped with Teams and so we will have unique keys for each Teams. So if the endpoint is accessed by API Key then the dynamo should track Token Usage (Input, Output, Total, Timings) for all hits.

Coming to your question: Yes to track token usage based on API Keys which is mapped for specific teams.

API Key Team Input Output Total
sk-xxxx T1 5 8 13
sk-yyyy T2 12 20 32

Happy to have a on-call conversation.

[azzaea]

Excellent! Would love to see this indeed- it's exactly the same need on our end (multi-user, university-wide, model serving)

[nithin8702]

vLLM Production Stack has this similar feature but its not multi-tenancy.
https://github.com/vllm-project/production-stack/blob/main/tutorials/assets/values-11-secure-vllm.yaml#L3

But its available in LiteLLM Proxy Server
https://docs.litellm.ai/docs/providers/litellm_proxy

Its good to have this feature in Dynamo.

[nnshah1]

Thanks for the additional details! If I understand correctly the goal here is to:

1. Secure who has access to frontend via an api key
2. Segment the metrics by api key
3. Share the same set of models and gpus across api keys

Is that correct?

One potential quick path would be to deploy multiple frontends for the same dynamo deployment. Each frontend would be dedicated to a set of clients with api key and metrics would be collected automatically.

[nithin8702]

Thats correct. And do we have an example for multiple frontend with api key?

[hutm]

Would envoy ai gateway be helpful here ? https://aigateway.envoyproxy.io/docs/0.1/capabilities/usage-based-ratelimiting/ Adding it in front of a dynamo pipeline should get us what we want

[nithin8702]

Envoy AI Gateway will solve it but we are planning to have this Q2. For now we have nginx only.

[nithin8702]

And does dynamo support api key now?