'generator raised StopIteration' error when running 'randomstats' with multiple processes

[tparket]

Hi,

First of all - thank you for your amazing work. pybedtools has been super useful for my research so far and I am very grateful.

I'm trying to run 'randomstats' with the following args:

results_dict = a.randomstats(b, iterations=1000, new=True, genome_fn=chromsizes_fn, processes=4, shuffle_kwargs={"chrom": True}, intersect_kwargs={"f": 1})

`---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
~/.local/lib/python3.7/site-packages/pybedtools/bedtool.py in parallel_apply(self, iterations, func, func_args, func_kwargs, processes, _orig_pool)
2932 for it in range(iterations):
-> 2933 yield func(*func_args, **func_kwargs)
2934 raise StopIteration

~/.local/lib/python3.7/site-packages/pybedtools/stats.py in random_intersection(x, y, genome_fn, shuffle_kwargs, intersect_kwargs)
16 result = len(zz)
---> 17 helpers.close_or_delete(z, zz)
18 return result

~/.local/lib/python3.7/site-packages/pybedtools/helpers.py in close_or_delete(*args)
547 if hasattr(x.fn, "throw"):
--> 548 x.fn.throw(StopIteration)
549

StopIteration:

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
in

~/.local/lib/python3.7/site-packages/pybedtools/bedtool.py in randomstats(self, other, iterations, new, genome_fn, include_distribution, **kwargs)
2846 )
2847 distribution = self._randomintersection(
-> 2848 other, iterations=iterations, genome_fn=genome_fn, **kwargs
2849 )
2850

~/.local/lib/python3.7/site-packages/pybedtools/bedtool.py in _randomintersection(self, other, iterations, genome_fn, intersect_kwargs, _orig_pool, shuffle_kwargs, processes)
3038 ),
3039 processes=processes,
-> 3040 _orig_pool=_orig_pool,
3041 )
3042 )

RuntimeError: generator raised StopIteration`

The thing is that when I remove the 'processes' argument the 'randomstats' works just fine, but everytime I try to run it with 'processes' (even with a value of 1), I get the aformentioned error.

Other relevant data:

• 'a' and 'b' are both bedtool objects generated from a df. A regular a.intersect(b, f=1) works perfectly.
• 'chromsizes_fn' is the name of a genome file generated from a dict with:
  chromsizes_fn = pybedtools.chromsizes_to_file(chromsizes_dic, fn=temp_genome.name)
  I tried using both fn=False and fn=temp_genome.name
• I tried to run it with both new=True and without it. It crashed on both tries.

I would really appreciate your help. I'm planning to run 'randomstats' on a large number of files, with at least 1000 iterations for each time, and being able to use multiprocessing will make it feasible.

[daler]

Great to hear you find pybedtools useful.

Can you provide an example of the files you're using for a and b so I can test locally?

[tparket]

Thanks for getting back to me so soon.
Please find the files* attached.

Archive.zip

*these are not the original files, but randomly generated intervals. Nevertheless, I’m getting the same errors.

[igoronzy]

I'm getting the same error. Have there been any updates to fix this issue?

[bentyeh]

Bumping this. Might be a Python versioning issue.

Prior to Python 3.7, the StopIteration raised by the generator (parallel_apply()) would have just signaled the end of the iteration. Starting in Python 3.7, a StopIteration raised by a generator is converted into a RuntimeError: see https://docs.python.org/3/library/exceptions.html#StopIteration

A workaround that seems to work for now is to:

1. Comment out these 2 lines in the close_or_delete() function from helpers.py

   if hasattr(x.fn, "throw"):
       x.fn.throw(StopIteration)
   

2. Replace the 2 instances (here and here) of raise StopIteration in BedTool.parallel_apply() with a simple return.

I'm happy to submit a pull request, but this may be part of a larger issue of dealing with Python versions in pybedtools.