classification
Title: multiprocessing only use one core when C module are imported
Type: behavior Stage: resolved
Components: ctypes, Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: hadim, neologix
Priority: normal Keywords:

Created on 2013-01-26 09:39 by hadim, last changed 2013-01-26 12:33 by hadim. This issue is now closed.

Files
File name Uploaded Description Edit
bug.py hadim, 2013-01-26 09:39 .py file which reproduce the bug
Messages (6)
msg180648 - (view) Author: HadiM (hadim) Date: 2013-01-26 09:39
Hi,

This is the first time I report a bug so if I did something wrong please let me know. I also tried to ask on #python and #python-fr before posting here to know if it was a bug or a possible multiprocessing limitation on Linux. So I'm sorry if it's not a bug... 

The problem appears when a module with .c compiled file is imported (sush as numpy, scipy, pandas, tables, skimage, I did not tested with a custom .c module). Multiprocessing and more specifically imap and map method of Pool class did not distribute processes along all the core but only on one core. This problem does not appears when I don't import a module with .c compiled.

I know distributing processes along cores is specific to the OS implementation and not related to Python but I said that to illustrate the behaviour.

I should notice that I did not see the bug with a Windows 7, 32 bits (Virtualbox).

My laptop run with the latest ubuntu 64bits (tested with both kernel 3.5 and 3.8 last rc). It's a classic Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz with 4 cores.

You can try to reproduce the bug with the attached file. You can also find the same file here : http://pastebin.com/jqq9G5Ph

Thank you for your time !
msg180650 - (view) Author: HadiM (hadim) Date: 2013-01-26 10:03
I test to launch bug.py with pypy (import numpypy instead of import numpy) and the bug did not appear.
msg180659 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2013-01-26 11:28
Hello,

> So I'm sorry if it's not a bug...

Don't be afraid, we don't byte :-)

Concerning your problem, my guess would be that one of the modules you import sets the process CPU affinity (maybe as a workaround to mitigate the GIL impact in multi-threaded code, or whatever) upon import.

And a quick google search returns this:
https://github.com/ipython/ipython/issues/840

To confirm this, you can just do:
strace -e sched_setaffinity python -c "import numpy"


You can also add
"""
    for line in open('/proc/self/status'):
        if 'Cpu' in line:
            print(line)
"""

Right before and after importing the module, and you'll see that the CPU affinity has changed.
msg180661 - (view) Author: HadiM (hadim) Date: 2013-01-26 11:42
Indeed some value change when I print "cpu" line from /proc/self/status but I don't really understand what that mean...

So there is no solution about that ? We cannot use multiprocessing with these modules under Linux ?

Do you think I can manually change the CPU affinity or at least block changes made by other modules (without recompiling them of course) ?

I guess if it's possible to modify CPU affinity, numpy and other scientific libraries won't be efficient as before, no ? Because the won't share the same cache or something like that, if I get the wikipedia page about cpu affinity.
msg180663 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2013-01-26 12:01
> Indeed some value change when I print "cpu" line from /proc/self/status but I don't really understand what that mean...

It means that the CPU affinity is changed, so the scheduler binds your
process to a subset of the available CPUs.

> So there is no solution about that ? We cannot use multiprocessing with these modules under Linux ?
>
> Do you think I can manually change the CPU affinity or at least block changes made by other modules (without recompiling them of course) ?

If you don't want to recompile the offending module (it seems to be
GotoBLAS2), you can change the affinity after importing your modules
with something like:
"""
os.system("taskset -p 0xff %d" % os.getpid())
"""

(Python 3.3 exposes the scheduling API natively, see
http://docs.python.org/dev/library/os.html#interface-to-the-scheduler)

> I guess if it's possible to modify CPU affinity, numpy and other scientific libraries won't be efficient as before, no ? Because the won't share the same cache or something like that, if I get the wikipedia page about cpu affinity.

Soft affinity is naturally done by the operating system scheduler, so
you don't have to worry about cache line bouncing and the like. Hard
affinity is only useful in very specific cases.
msg180664 - (view) Author: HadiM (hadim) Date: 2013-01-26 12:33
Your snippet did the trick ! Thank you for your time. Even if it's not very clean, it's working.

Thank again !
History
Date User Action Args
2013-01-26 12:33:49hadimsetmessages: + msg180664
2013-01-26 12:02:26neologixsetstatus: open -> closed
resolution: not a bug
stage: resolved
2013-01-26 12:01:58neologixsetmessages: + msg180663
2013-01-26 11:42:16hadimsetmessages: + msg180661
2013-01-26 11:28:23neologixsetnosy: + neologix
messages: + msg180659
2013-01-26 10:03:07hadimsetmessages: + msg180650
2013-01-26 09:39:23hadimcreate