I have managed to solve the problem by inserting in the beginning of my program:

import multiprocessing
multiprocessing.set_start_method('forkserver')
as this is explained here: https://scikit-learn.org/stable/faq.html#why-do-i-sometime-get-a-crash-freeze-with-n-jobs-1-under-osx-or-linux <https://scikit-learn.org/stable/faq.html#why-do-i-sometime-get-a-crash-freeze-with-n-jobs-1-under-osx-or-linux>
It works, but the shell looses some level of interactivity as the results intermediate results don't get printed as the program is executed.

> On 2 Nov 2020, at 19:03, Ken Jin <report@bugs.python.org> wrote:
> 
> 
> Ken Jin <kenjin4096@gmail.com> added the comment:
> 
> Hmm apologies I'm stumped then. The only things I managed to surmise from xgboost's and scikit-learn's GitHub issues is that this is a recurring issue specifically when using GridSearchCV :
> 
> Threads with discussions on workarounds:
> https://github.com/scikit-learn/scikit-learn/issues/6627
> https://github.com/scikit-learn/scikit-learn/issues/5115
> 
> Issues reported:
> https://github.com/dmlc/xgboost/issues/2163
> https://github.com/scikit-learn/scikit-learn/issues/10533
> https://github.com/scikit-learn/scikit-learn/issues/10538 (this looks quite similar to your issue)
> 
> Some quick workarounds I saw were:
> 1. Remove n_jobs argument from GridSearchCV
> 2. Use parallel_backend from sklearn.externals.joblib rather than concurrent.futures so that the pools from both libraries don't have weird interactions.
> 
> I recommend opening an issue on scikit-learn/XGBoost's GitHub. This seems like a common problem that they face.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue42245>
> _______________________________________