Add over saturation constraint #438

AlonKellner-RedHat · 2025-10-29T14:24:29Z

Summary

This PR adds over-saturation stopping to the GuideLLM CLI.
It's based on the OSD (Over-Saturation Detection) algorithm we developed and evaluated at Jounce.
Use --stop-over-saturated or --stop-osd to enable.

Details

This PR adds:

Over-saturation stopping (--stop-over-saturated)
Comprehensive OSD unit tests

Test Plan

Currently, only unit tests
When test(e2e): basic E2Es #440 lands, we'll enable its over-saturation stopping E2E test

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes AI-assisted code completion
Includes code generated by an AI application
Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes ## WRITTEN BY AI ##)

## Summary E2E tests which check basic GuideLLM functionality, using vLLM simulator. ## Details - [x] Max requests test - [x] Max duration test - [ ] Over-saturation stopping test - skipped for now, will be enabled when #438 lands ## Test Plan - [x] Local testing - [x] GitHub action --- - [x] "I certify that all code in this PR is my own, except as noted below." ## Use of AI - [x] Includes AI-assisted code completion - [ ] Includes code generated by an AI application - [ ] Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes `## WRITTEN BY AI ##`)

markurtz

prompt-python_module_docs.md
Overall looks good. I have a few comments/discussion points to run through around names, types, package layout, etc. I haven't gone through the full logic yet to validate, will do that soon, but adding a writeup of the algorithm would help at the top.

Also, can you ensure all public methods have docs? I'll attach the prompt I use to generate docs that is generally fairly self sufficient and accurate in case you'd like to use that so it doesn't take too much time.

src/guidellm/scheduler/constraints/__init__.py

src/guidellm/benchmark/schemas/generative/entrypoints.py

src/guidellm/benchmark/entrypoints.py

src/guidellm/scheduler/constraints/over_saturation.py

src/guidellm/scheduler/constraints/saturation.py

src/guidellm/__main__.py

tests/unit/scheduler/OVER_SATURATION_TEST_COVERAGE.md

jaredoconnell

I started the process of reviewing this PR. It appears to work, though I need to fix some problems with my VLLM instance to be sure about that.
I think this feature needs some more user-facing documentation. It may make sense to add a markdown file in the docs section explaining what the typical end users need to know about this.

AlonKellner-RedHat · 2025-11-30T13:17:53Z

I started the process of reviewing this PR. It appears to work, though I need to fix some problems with my VLLM instance to be sure about that. I think this feature needs some more user-facing documentation. It may make sense to add a markdown file in the docs section explaining what the typical end users need to know about this.

Done

AlonKellner-RedHat · 2025-11-30T13:21:49Z

This PR is now ready again after applying all review changes

sjmonson

Few more nits. Also some of Mark's comments were missed about import style.

src/guidellm/scheduler/constraints/saturation.py

tests/e2e/README.md

tox.ini

src/guidellm/__main__.py

src/guidellm/scheduler/constraints/saturation.py

src/guidellm/benchmark/schemas/generative/entrypoints.py

src/guidellm/scheduler/constraints/saturation.py

AlonKellner-RedHat · 2025-12-04T17:16:05Z

@markurtz @sjmonson I went through everything (made sure I didn't miss anything this time) and the E2Es pass again 💪

src/guidellm/utils/cli.py

src/guidellm/__main__.py

src/guidellm/scheduler/constraints/request.py

jaredoconnell

This is close to ready. The documentation needs to be updated, the other reviewers' latest comments need to be addressed, and one E2E test is still failing.

docs/guides/over_saturation_stopping.md

Add over-saturation detection and stopping capability to GuideLLM CLI. - Implement OverSaturationDetector with statistical slope detection - Add OverSaturationConstraint for scheduler integration - Add CLI flags --stop-over-saturated and --stop-osd - Integrate with benchmark entrypoints and main CLI Signed-off-by: Alon Kellner <[email protected]>

Signed-off-by: Alon Kellner <[email protected]>

Signed-off-by: Samuel Monson <[email protected]>

jaredoconnell

Looks good, with the exception of one last out of date documentation part. I'll approve since once that's fixed I don't need to check again.

README.md

markurtz

LGTM pending @jaredoconnell's docs comment

Signed-off-by: Samuel Monson <[email protected]>

AlonKellner-RedHat mentioned this pull request Oct 30, 2025

test(e2e): basic E2Es #440

Merged

9 tasks

AlonKellner-RedHat requested review from markurtz and sjmonson November 4, 2025 16:46

AlonKellner-RedHat force-pushed the feat/over-saturation-stopping branch 4 times, most recently from 85cf65e to f996254 Compare November 6, 2025 11:11

AlonKellner-RedHat force-pushed the feat/over-saturation-stopping branch 4 times, most recently from ce85b1c to 46bc491 Compare November 18, 2025 08:15

markurtz requested changes Nov 19, 2025

View reviewed changes

AlonKellner-RedHat force-pushed the feat/over-saturation-stopping branch from 59ffa16 to 3ebdcaa Compare November 20, 2025 11:35

markurtz mentioned this pull request Nov 21, 2025

Over-Saturation stopping #242

Open

jaredoconnell reviewed Nov 24, 2025

View reviewed changes

AlonKellner-RedHat force-pushed the feat/over-saturation-stopping branch 2 times, most recently from e8a0bb6 to c1e97fe Compare November 30, 2025 12:40

sjmonson requested changes Dec 2, 2025

View reviewed changes

AlonKellner-RedHat force-pushed the feat/over-saturation-stopping branch from ccab11e to 4ef1f98 Compare December 4, 2025 17:15

sjmonson requested changes Dec 4, 2025

View reviewed changes

src/guidellm/utils/cli.py Outdated Show resolved Hide resolved

src/guidellm/utils/cli.py Outdated Show resolved Hide resolved

src/guidellm/__main__.py Outdated Show resolved Hide resolved

sjmonson requested changes Dec 4, 2025

View reviewed changes

src/guidellm/scheduler/constraints/request.py Outdated Show resolved Hide resolved

sjmonson force-pushed the feat/over-saturation-stopping branch from 4ef1f98 to 44a668b Compare December 4, 2025 21:37

jaredoconnell reviewed Dec 5, 2025

View reviewed changes

docs/guides/over_saturation_stopping.md Show resolved Hide resolved

docs/guides/over_saturation_stopping.md Outdated Show resolved Hide resolved

sjmonson force-pushed the feat/over-saturation-stopping branch 2 times, most recently from 4b1774f to 265a146 Compare December 5, 2025 16:44

AlonKellner-RedHat and others added 16 commits December 5, 2025 11:44

fix: CI errors

16568d5

Signed-off-by: Alon Kellner <[email protected]>

fix: E2E tests

e3a9556

Signed-off-by: Alon Kellner <[email protected]>

fix: E2E tests

cee4417

Signed-off-by: Alon Kellner <[email protected]>

fix: E2E tests

7d16fb1

Signed-off-by: Alon Kellner <[email protected]>

feat: macos llm-d simulator dockerfile

be1af98

Signed-off-by: Alon Kellner <[email protected]>

fix: over-saturation settings handling

9ed67af

Signed-off-by: Alon Kellner <[email protected]>

feat: over-saturation docs

72fe7fb

Signed-off-by: Alon Kellner <[email protected]>

fix: mdformat over-saturation docs

832f7df

Signed-off-by: Alon Kellner <[email protected]>

fix: review suggestions

2579f41

Signed-off-by: Alon Kellner <[email protected]>

Address feedback

95628aa

Signed-off-by: Samuel Monson <[email protected]>

Update over saturation flag name in docs

241aceb

Signed-off-by: Samuel Monson <[email protected]>

Update over sat test to pass in min seconds

800bec6

Signed-off-by: Samuel Monson <[email protected]>

Fix e2e tests writing output to test dir

3bd986b

Signed-off-by: Samuel Monson <[email protected]>

Treat OverSaturationConstraint as a Constraint

ca5363c

Signed-off-by: Samuel Monson <[email protected]>

Drop benchmark report delete in e2e

1d16296

Signed-off-by: Samuel Monson <[email protected]>

Add info property to sat constraint

89b145d

Signed-off-by: Samuel Monson <[email protected]>

sjmonson force-pushed the feat/over-saturation-stopping branch from 265a146 to 89b145d Compare December 5, 2025 16:44

sjmonson requested review from jaredoconnell, markurtz and sjmonson December 5, 2025 16:46

sjmonson changed the title ~~Feat/over saturation stopping~~ Add over saturation constraint Dec 5, 2025

sjmonson requested review from sjmonson and removed request for sjmonson December 5, 2025 16:48

jaredoconnell previously approved these changes Dec 5, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

markurtz previously approved these changes Dec 5, 2025

View reviewed changes

Update main README

913a601

Signed-off-by: Samuel Monson <[email protected]>

sjmonson dismissed stale reviews from markurtz and jaredoconnell via 913a601 December 5, 2025 18:24

sjmonson approved these changes Dec 5, 2025

View reviewed changes

sjmonson merged commit a98a644 into vllm-project:main Dec 5, 2025
29 checks passed

Add over saturation constraint #438

Add over saturation constraint #438

Uh oh!

Conversation

AlonKellner-RedHat commented Oct 29, 2025 • edited by sjmonson Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Test Plan

Use of AI

Uh oh!

markurtz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

AlonKellner-RedHat commented Nov 30, 2025

Uh oh!

AlonKellner-RedHat commented Nov 30, 2025

Uh oh!

sjmonson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlonKellner-RedHat commented Dec 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

markurtz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AlonKellner-RedHat commented Oct 29, 2025 •

edited by sjmonson

Loading

markurtz left a comment •

edited

Loading