Hi Ken, thanks for a quick reply.

Here are the requested specs.
System:
Python 3.7.6
OS X 10.15.7

Packages:
XGBoost 1.2.0
sklearn 0.22.2
pandas 1.0.5
numpy 1.18.1

I can see that you have reduced the code, which now excludes the RandomizedSearchCV part. This (reduced) code runs without any problems on my side as well, but if running it after the RandomizedSearchCV, the last function fit_model() freezes in a multiprocess.

I will read through the docs, but at first it looks as the actual problem is in the concurrent.futures module, because the easy function just_print() runs without issues. So the freeze is triggered by adding minor complexity into the fit_model() function running in a multiprocess.

> On 2 Nov 2020, at 17:34, Ken Jin <report@bugs.python.org> wrote:
> 
> 
> Ken Jin <kenjin4096@gmail.com> added the comment:
> 
> Hello, it would be great if you can you provide more details. Like your Operating System and version, how many logical CPU cores there are on your machine, and lastly the exact Python version with major and minor versions included (eg. Python 3.8.2). Multiprocessing behaves differently depending on those factors.
> 
> FWIW I reduced your code down to make it easier to read, and removed all the unused variables:
> 
> import concurrent.futures
> from sklearn.datasets import make_regression
> 
> def just_print():
>    print('Just printing')
> 
> def fit_model():
>    data = make_regression(n_samples=500, n_features=100, n_informative=10, n_targets=1, random_state=5)
>    print('Fit complete')
> 
> if __name__ == '__main__':
>    with concurrent.futures.ProcessPoolExecutor() as executor:
>        results_temp = [executor.submit(just_print) for i in range(0,12)]
> 
>    with concurrent.futures.ProcessPoolExecutor() as executor:
>        results_temp = [executor.submit(fit_model) for i in range(0,12)]
> 
> The problem is that I am *unable* to reproduce the bug you are reporting on Windows 10 64-bit, Python 3.7.6. The code runs till completion for both examples. I have a hunch that your problem lies elsewhere in one of the many libraries you imported.
> 
>>>> Note: problem occurs only after performing the RandomizedSearchCV...
> 
> Like you have noted, I went to skim through RandomizedSearchCV's source code and docs. RandomizedSearchCV is purportedly able to use multiprocessing backend for parallel tasks. By setting `n_jobs=-1` in your params, you're telling it to use all logical CPU cores. I'm unsure of how many additional processes and pools RandomizedSearchCV's spawns after calling it, but this sounds suspicious. concurrent.futures specifically warns that this may exhaust available workers and cause tasks to never complete. See https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor (the docs here are for ThreadPoolExecutor, but they still apply).
> 
> A temporary workaround might be to reduce n_jobs OR even better: use scikit-learn's multiprocessing parallel backend that's dedicated for that, and should have the necessary protections in place against such behavior. https://joblib.readthedocs.io/en/latest/parallel.html#joblib.parallel_backend 
> 
> 
> TLDR: I don't think this is a Python bug and I'm in favor of this bug being closed as `not a bug`.
> 
> ----------
> nosy: +kj
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue42245>
> _______________________________________