ML Training step fails to complete

[bachase]

System and Environment Information

Running on mac os 15.6.1, python version 3.13.0 and mqt-predictor version 2.3.0

Bug Description

I'm following the framework setup section in the documentation. I was able to run Step 2 (RL training) in about 4 hours on my M4 Pro, using 50k timesteps for IBM Falcon 27.

For Step 3, I've been running the code for several hours but am not able to tell the progress in the ML training phase. Is there a way to assess the progress? Is there an estimate of how long it should take?

You can look at this log for output from my run before I killed the process. It looks like a call to bqskit is timing out, and then later on, it looks like some subprocesses may not be handling signals properly and lingering.

Steps to Reproduce

1. Setup environment with uv init and uv add mqt.predictor
2. Follow the steps in https://mqt.readthedocs.io/projects/predictor/en/latest/setup.html
3. For step 3, run:

from mqt.predictor.ml import setup_device_predictor
from mqt.bench.targets import get_device

devices = [get_device("ibm_falcon_27")]
setup_device_predictor(
    devices=devices,
    figure_of_merit="expected_fidelity",
)

[flowerthrower]

Hi @bachase .
You are right, there appears to be an issue with one of the bqskit passes.
Thank you for bringing this to our attention, we will look into it.

Regarding the training duration: the ML training itself should complete within a few minutes to an hour, depending on your device.
The error seems to occur during the data generation phase, i.e., while compiling the circuits using the trained RL model.
This step typically takes significantly longer.
You can track its progress by monitoring the training_circuits_compiled folder.
Once the training data preparation is finished, there should be one compiled version of each training circuit for every device.

There are several possible solutions.
You may skip the failing circuit by removing it from the training set.
Since some compilation passes are nondeterministic, you can also try rerunning the compilation, as the agent may avoid the unfavourable configuration on a subsequent attempt.
Alternatively, @Shaobo-Zhou pointed out, some bqskit passes can have unexpectedly long runtimes, so increasing the timeout beyond the default 600s may also resolve the issue.

[bachase]

Thanks for the advice @flowerthrower. Unfortunately, I tried both (made timeout 1200s and skipped the circuit causing trouble), and am still getting similar behavior. I had to timebox my exploration, so won't be able to dig further now. Feel free to close this issue accordingly if there isn't enough to investigate further.