Setting debug_mode in a Worker still doesn't allow the user of breakpoints

[togakangaroo]

What are you really trying to do?

I am trying to use the pdb debugger from inside unit tests literally with the breakpoint() function.

Describe the bug

It is pretty straightforward really, change the workflow to use sandboxing and set a breakpoint(). When you do you get an endless scroll with

temporalio.worker.workflow_sandbox._restrictions.RestrictedWorkflowAccessError: Cannot access __builtins__.breakpoint from inside a workflow. If this is code from a module not used in a workflow or known to only be used deterministically from a workflow, mark the import as pass through.
08:48:32 [ WARNING] Failed activation on workflow GreetingWorkflow with ID 68cd5a5e-2ada-40ff-aa67-63eae9ab3055 and run ID 01994355-86ca-7dd3-9335-69a9e5be86ff (_workflow_instance.py:452)
Traceback (most recent call last):
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/worker/_workflow_instance.py", line 412, in activate
    self._run_once(check_conditions=index == 1 or index == 2)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/worker/_workflow_instance.py", line 2130, in _run_once
    raise self._current_activation_error
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/worker/_workflow_instance.py", line 2148, in _run_top_level_workflow_function
    await coro
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/worker/_workflow_instance.py", line 974, in run_workflow
    result = await self._inbound.execute_workflow(input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/testing/_workflow.py", line 518, in execute_workflow
    return await super().execute_workflow(input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/worker/_interceptor.py", line 385, in execute_workflow
    return await self.next.execute_workflow(input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/worker/_workflow_instance.py", line 2529, in execute_workflow
    return await input.run_fn(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/georgemauer/code/temporal-samples/hello/hello_activity.py", line 32, in run
    breakpoint()
    ~~~~~~~~~~^^
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 497, in __call__
    return self.current(*args, **kwargs)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/georgemauer/code/temporal-samples/.venv/lib/python3.13/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 94, in restrict_built_in
    raise RestrictedWorkflowAccessError(f"__builtins__.{name}")

Minimal Reproduction

I have recreated this in temporal samples here

Environment/Versions

• Apple M4 Pro
• Temporal Version: 1.15.0
• Literally just running the samples

Additional context

I am debugging as part of trying to figure out why passing RetryPolicy(max_attempts=1) on client.execute_workflow in my unit tests doesn't seem to turn off retries of activities.

What I ultimately want is to be able to tell my unit tests to only retry once or twice - essentially to set a retry policy specifically for my tests. The documentation implies that doing this by passing retry_policy=RetryPolicy(max_attempts=2) to client.execute_workflow() should work yet it doesn't for me. The only way to get it working is by passing a retry policy to workflow.execute_activity() but that would mean changing my code specifically for simple automation tests which I want to avoid.

So I started trying to inspect the code. Because there are base classes involved I want to inspect things at run time, however setting a breakpoint in my workflow fails. According to the documentation, debug_mode=True should make it possible to use breakpoints but that seems to only be the case when you don't have sandbox=True on your workflows which means I'd have to modify my code specifically for testing.

[togakangaroo]

This seems to be happening with @workflow.defn(sandboxed=False) and workflow_runner=UnsandboxedWorkflowRunner() as well.

I have to emphasize how utterly painful it is to have to develop things without the ability to place a breakpoint. Even when you can test activties in isolation, what we care about is the workflow as whole and that testability is really key. In my real-world case, the activities are basically just doing db loading and projection, not much to test - its the logic in the workflows and ensuring how it works across retries thats important.

[togakangaroo]

Another example of where I'm struggling without breakpoint - I'm troubleshooting a situation in my tests where a mock is not set the way I intend it to. I want to inspect what is going on and I simply cannot.

For what its worth, I have tried

    with workflow.unsafe.sandbox_unrestricted():
        breakpoint()

which does hit the breakpoint, but the breakpoint is not in my code. I have to navigate up the call stack and then do see my code there and can inspect it, but boy is it a big workaround.

[tconley1428]

The documentation implies that doing this by passing retry_policy=RetryPolicy(max_attempts=2) to client.execute_workflow() should work yet it doesn't for me

Can you point to documentation that says this? The retry policy in execute_workflow is the policy for workflow retries, not activities. If the documentation says otherwise or is unclear, it should be fixed.

As to the larger question of breakpoints, they should work with the unsandboxed runner, so that is something we would have to look into. The recommended way to debug would be to debug a replay of the workflow run, as that will avoid any interaction with server timeouts, but that is more of a problem when you have hit and are staying at the breakpoint, not in hitting it in the first place.

[togakangaroo]

@tconley1428 I guess I don't really understand the distinction between workflow and activity retries. If I have a workflow with activities A then B, if activity B fails, then it will retry again later and when it does it will rerun the entire workflow while skipping activity A because it's already completed successfully in the event history. It does not, sticky cache aside, save the entire workflow memory state in between the two runs, the workflow runs from the beginning.

So if the entire workflow reruns again...then what's the distinction? Is it that failed workflows themselves would be retried?

If so, I think that bit is very confusing since the entire workflow does rerun on activity retries. Additionally, the documentation does mention a default RetryPolicy but clicking that link simply takes you to the encyclopedia page on retries in general.

Given the above and, because both use the same RetryPolicy class, it seems a reasonable inference that setting it at the workflow level is what controls the default. I think the docs need to make a very clear distinction on it as well as explaining how to set the default (which is needed since you would want a different policy to be used in unit tests).

I'll write another reply about breakpoints...

[togakangaroo]

Regarding breakpoints. If I take exactly what is in that branch I have in the demo and run the unit test, it actually is hitting the breakpoint now (which is odd, I could have sworn it threw an error earlier, but maybe that was when I was trying in different configurations).

uv run pytest --pdb '/Users/georgemauer/code/temporal-samples/tests/hello/hello_activity_test.py::test_mock_activity'

However, notice in that screenshot that while it does break, it is not in the pdb repl and I cannot interact with the debugger in any way. I cannot inspect a variable, I cannot step, I cannot continue, I can't even seem to quit out of it without having to kil the underlying process in another terminal. This is despite me having debug-mode=True and the workflow being marked as not sandboxed.

Interestingly, in the samples this is happening even if I do workflow_runner=UnsandboxedWorkflowRunner() and the with workflow.unsafe.sandbox_unrestricted() stuff I figured out above. That worked in my real project, but in this samples branch it just hangs.

[togakangaroo]

I want to comment on this as well @tconley1428

The recommended way to debug would be to debug a replay of the workflow run

That precludes test-driven-development as workflow does it not? Since you'd need something deployed somewhere so that you could get something to replay?

[tconley1428]

That precludes test-driven-development as workflow does it not? Since you'd need something deployed somewhere so that you could get something to replay?

Take a look at the Replayer: https://github.com/temporalio/sdk-python/blob/main/temporalio/worker/_replayer.py#L33. You can run a workflow in tests, get its history, then either store it (we do this to test replays across sdk version changes) or run it through the replayer to replay immediately.

If so, I think that bit is very confusing since the entire workflow does rerun on activity retries

Only kinda. The workflow might replay to retry an activity, but doesn't necessarily. And that's different from rerunning the workflow, which would redo any completed activities, being a fresh start of the workflow. That said, it isn't very common and not recommended to rely on workflow retries.

I take your point about the documentation of a default retry policy though. The "default" retry policy is not a mutable one. If you'd like, you can make an additional issue requesting a means of overriding the default activity retry policy at a higher level, and we'll consider it.

On breakpoints, we might need to find some time to reproduce your behavior. With sandboxing off, it shouldn't really be different from a standard python program, but I haven't debugged in that way before.