Series of teardown failures and final deadlock in pytest observed on the CI

[ogrisel]

Here is an archive of a particularly bad recent run:

https://gist.github.com/ogrisel/371405ac68b5c7a977b907c843cdf798

Observed on #385 but probably unrelated to the change in this PR.