Skip to content

Conversation

@Chibionos
Copy link
Contributor

@Chibionos Chibionos commented Jan 15, 2026

Summary

Enables evaluators to execute and produce scores after agent resumes from suspended state.

Changes

  1. Consistent thread_id for suspend/resume

    • Changed from execution_id (new UUID each run) to eval_item.id (consistent identifier)
    • Ensures checkpoints can be found when resuming
  2. Command API integration

    • Pass Command(resume=data) when --resume flag is set
    • Provides resume data to waiting interrupt() call
  3. Enhanced logging

    • Added logging to track resume mode execution
    • Shows which eval_item.id is used for checkpoint loading

Testing

Tested with suspend/resume evaluation samples in uipath-langchain-python:

  • ✅ Suspend phase creates checkpoints with eval_item.id as thread_id
  • ✅ Resume mode detected and logged
  • ✅ Checkpoint successfully loaded using consistent thread_id
  • 🚧 Resume execution encounters Pydantic validation - needs investigation

Built on top of

🤖 Generated with Claude Code

Development Package

  • Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath==2.5.14.dev1011324039",

  # Any version from PR
  "uipath>=2.5.14.dev1011320000,<2.5.14.dev1011330000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath>=2.5.14.dev1011320000,<2.5.14.dev1011330000",
]

@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Jan 15, 2026
@Chibionos Chibionos force-pushed the feat/enable-evaluators-on-resume branch 2 times, most recently from 540ed7a to e905c9c Compare January 16, 2026 01:23
@Chibionos Chibionos force-pushed the feat/enable-evaluators-on-resume branch from 52aaa6b to 33d66ea Compare January 19, 2026 03:59
@Chibionos Chibionos added the build:dev Create a dev build from the pr label Jan 19, 2026
Chibi Vikram added 2 commits January 19, 2026 09:16
- Use eval_item.id as runtime_id to maintain consistent thread_id across suspend/resume
- Pass UiPathExecuteOptions(resume=True) when in resume mode
- Add logging to track resume mode execution

This ensures:
1. Checkpoints can be found when resuming (same thread_id)
2. Framework-agnostic approach (uses runtime options, not LangGraph Command)
3. Resume handling is delegated to runtime layer where framework-specific logic belongs
During resume, the eval runtime should pass None as input instead of
inputs_with_overrides. This allows the UiPathResumableRuntime wrapper to:
1. Fetch triggers from storage
2. Read resume data via trigger_manager.read_trigger()
3. Build resume map: {interrupt_id: resume_data}
4. Pass this map to the delegate runtime

The wrapper's _restore_resume_input() method handles the resume data
restoration automatically when input is None. Passing the original input
would skip this mechanism and cause the resume to fail.

This completes the fix for evaluator execution after resume, working in
conjunction with the langchain runtime's Command(resume=...) wrapper.
@Chibionos Chibionos force-pushed the feat/enable-evaluators-on-resume branch from 33d66ea to 5834c18 Compare January 19, 2026 17:21
Changes:
- Bump version from 2.5.13 to 2.5.14
- Add execution_id logging to track evaluation execution across suspend/resume
- Log execution_id at runtime creation
- Log execution_id before initial execution
- Log execution_id before and after resume execution
- Use eval_item.id as execution_id for consistency with runtime_id

This helps with debugging and tracing evaluation runs, especially when
troubleshooting suspend/resume issues. The execution_id is now visible
in logs at all key points in the execution lifecycle.
@Chibionos Chibionos force-pushed the feat/enable-evaluators-on-resume branch from 5834c18 to ce5f2f6 Compare January 19, 2026 17:28
- Remove verbose emoji logs used for debugging
- Keep minimal logging for suspend/resume confirmation:
  - 'Resuming evaluation {id}' when resuming
  - 'Evaluation {id} suspended' when suspended
- Improves user experience with cleaner log output
@Chibionos Chibionos merged commit c483f2d into main Jan 20, 2026
120 of 124 checks passed
@Chibionos Chibionos deleted the feat/enable-evaluators-on-resume branch January 20, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:dev Create a dev build from the pr test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants