This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: Ivan.K, davin, sbt
Priority: normal Keywords:

Created on 2014-04-05 16:41 by Ivan.K, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (16)
msg215620 - (view) Author: Ivan K (Ivan.K) Date: 2014-04-05 16:40
Here is a code. When I run it in freeze and never end. If I run it without pool with loop or joblib - it works fine.

Also there is a question on stackoverflow:

Python 2.7.6/2.7.3, cpu core i7 if it's important
msg215663 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2014-04-06 18:55
I would guess that the problem is simply that LogisticRegression objects are not picklable.  Does the problem still occur if you do not use freeze?
msg215664 - (view) Author: Ivan K (Ivan.K) Date: 2014-04-06 19:11
Sorry, I'm not sure I describe it correct. Freeze that means = goes fozen, so stop progress. It's do no do anything, but computations still load single core of my cpu for 100% untill I do not kill the python process.

But the same code work's fine if executed outside pool, or on the different platforms (like Mac for example), my friend run it successfully, but for me it not works. So it's look like pretty unpredictable issue with conflicting OS, CPU, and Python I'm afraid.
msg215665 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2014-04-06 19:12
Ah, I misunderstood: you meant that it freezes/hangs, not that you used a freeze tool.
msg215666 - (view) Author: Ivan K (Ivan.K) Date: 2014-04-06 19:13
Yes, I'm not using any tool. Code just not working.
msg215669 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2014-04-06 22:44
Could you try pickling and unpickling the result of func():

import cPickle
data = cPickle.dumps(func([1,2,3]), -1)
print cPickle.loads(data)
msg215670 - (view) Author: Ivan K (Ivan.K) Date: 2014-04-06 22:55
Sorry, could you specify what is 'func' ?
msg215671 - (view) Author: Ivan K (Ivan.K) Date: 2014-04-06 22:56
Sorry, stupdi question. Forget that this is from my gist
msg215672 - (view) Author: Ivan K (Ivan.K) Date: 2014-04-06 22:59
Yes, it work fine and output was 

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, penalty=l2, random_state=None, tol=0.0001)
msg237455 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-07 16:18
I am unable to reproduce the described behavior using the script provided in that gist using Python 2.7.9 on OS X 10.10 with scikit-learn 0.15.2, scipy 0.14.0, numpy 1.9.0.

For me, the sample code runs happily through to completion.

Can the OP still reproduce the behavior?  If not, it looks like scikit-learn and scipy have changed in the interim, addressing this behavior.
msg242789 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-05-09 01:27
It is no longer possible to reproduce the described issue.  Changes to the relevant libraries since this issue was originally opened appear to have addressed the cause.  Going ahead with closing this issue as there have been no further reports from the OP or others suggesting otherwise.
msg242802 - (view) Author: Ivan K (Ivan.K) Date: 2015-05-09 09:27
Sorry, forget to answer.

I think I know the reason.

Issue I think placed inside ATLAS library.
Issue not reproducable on Mac OS X because it's have a bit different thread model.

The only single platform is Linux (not tested on Windows btw). So main guess right now that this code fail if scikit-learn are compiled against a singlethread ATLAS. Then paralleling code brokes.

However there is very few information regarding how to build it with Multithreaded ATLAS so I'm still not 100% confident.
msg242832 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-05-09 19:07
@Ivan.K:  Can you be more specific about which linux platform you are using?  To reproduce, should I use Ubuntu, Fedora, OpenSUSE, ... and which version of that linux distro?
msg242853 - (view) Author: Ivan K (Ivan.K) Date: 2015-05-10 08:53
This behaviour was reproduced on Ubuntu 12.04 and on Centos 7 (centos image running on vagrant) so I don't expect this is ddistro specific
msg251188 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-09-21 02:55
This appears to be highly dependent upon particular versions of operating systems and specific versions of various libraries, some built with certain options.  It does not appear to be an issue in multiprocessing itself.

@Ivan.K:  Unless there is a clear example of how this issue can be reproduced with multiprocessing, I think the appropriate thing to do is close this issue as not a bug with multiprocessing.
msg255026 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-11-20 22:43
I'm going ahead with closing this issue as it originates outside of multiprocessing and Python itself as well as being highly dependent upon specific combinations of build options and versions of 3rd party libraries (such as ATLAS).
Date User Action Args
2022-04-11 14:58:01adminsetgithub: 65361
2015-11-20 22:43:21davinsetstatus: open -> closed

messages: + msg255026
stage: resolved
2015-09-21 02:55:56davinsetmessages: + msg251188
2015-05-10 08:53:03Ivan.Ksetmessages: + msg242853
2015-05-09 19:07:43davinsetmessages: + msg242832
2015-05-09 09:27:51Ivan.Ksetstatus: closed -> open

messages: + msg242802
2015-05-09 01:27:55davinsetstatus: pending -> closed
resolution: works for me
messages: + msg242789
2015-03-07 16:18:57davinsetstatus: open -> pending

nosy: + davin
messages: + msg237455

type: behavior
2014-04-06 22:59:18Ivan.Ksetmessages: + msg215672
2014-04-06 22:56:49Ivan.Ksetmessages: + msg215671
2014-04-06 22:55:34Ivan.Ksetmessages: + msg215670
2014-04-06 22:44:48sbtsetmessages: + msg215669
2014-04-06 19:13:15Ivan.Ksetmessages: + msg215666
2014-04-06 19:12:03sbtsetmessages: + msg215665
2014-04-06 19:11:42Ivan.Ksetmessages: + msg215664
2014-04-06 18:55:27sbtsetmessages: + msg215663
2014-04-05 18:16:55ned.deilysetnosy: + sbt
2014-04-05 16:41:00Ivan.Kcreate