classification
Title: Case sensitivity problem in multiprocessing.
Type: behavior Stage: resolved
Components: Interpreter Core, Windows Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: ColinPDavidson, James.Burke, christian.heimes, eric.snow, inlinestyle, ncoghlan, nmz787, pitrou, sbt, zach.ware
Priority: normal Keywords:

Created on 2014-05-30 19:55 by ColinPDavidson, last changed 2020-04-26 02:19 by zach.ware. This issue is now closed.

Messages (7)
msg219423 - (view) Author: Colin Davidson (ColinPDavidson) Date: 2014-05-30 19:55
On windows 7, if a Python app is run (under python 2.7.6) and the invoking filename is capitalized differently from the source file itself, a subsequent attempt to use the multiprocessing module to "fork" a process will fail in the "forking" module. This happens because the fopen at line 1559 in import.c (in the "imp" module find_module method) uses the invoking capitalization, rather than the file capitalization and fails to open the file.

The traceback gives lines 380 then 489 in forking.py, but as noted, the problem is in import.c (or in the Python startup, which could convert the capitalization...).
msg228115 - (view) Author: Nathan McCorkle (nmz787) Date: 2014-10-01 20:46
I've just got done experiencing this bug. It would be much more helpful if the error message was a bit more helpful (I had no idea where to start looking with the "ImportError: No module named [moduleStartingMultiprocess]" exception message)
msg228140 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-01 23:49
Can you try with Python 3.x? The import machinery is different there.
msg231217 - (view) Author: Ben Yelsey (inlinestyle) Date: 2014-11-15 20:49
Hi, could you please list more exact steps to reproduce, e.g. directory/file layout and import statements?
msg231740 - (view) Author: James Burke (James.Burke) Date: 2014-11-27 00:47
I'm also getting this error.

It appears to me to be caused by the length of the module name rather than case.
mp_bug.py will run fine for me, but if I change it to mp_bug_longname.py then I will get this error.

import multiprocessing
import time

class MP_Bug(multiprocessing.Process):
    def __init__(self):
        multiprocessing.Process.__init__(self)
    def run(test):
        for x in range(1, 10):
            print x
            time.sleep(1)

if __name__ == '__main__':
    p = MP_Bug()
    p.start()
msg231741 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-11-27 01:36
(Adding Richard as the multiprocessing maintainer, and Christian as the creator of the backport to 2.4/2.5)

Note that on a case-insensitive filesystem, the fopen() call is likely succeeding, and it's the subsequent case_ok() check that's failing. The assumption of case sensitivity is embedded fairly deeply in the import system, as otherwise it makes it pretty easy to accidentally import the same module multiple times under different names.

However, if importing the same module multiple times under different names isn't a concern, then setting PYTHONCASEOK should allow multiprocessing to import the module using the incorrect capitalisation.

More significant changes to the way the standard library's multiprocessing module starts subprocesses on Windows won't be implemented for Python 2.7 - actually fixing the subprocess spawning to work as intended in all cases (as was done in Python 3.4) relies on the import system changes defined in PEP 451. 

In theory, I expect a multiprocessing2 backport could be written that depends on importib2 (to enable Python 3.4+ import semantics in Python 2.7), but I'm not aware of anyone currently working on such a project.

James's comment sounds like a potentially different problem (e.g. there are some hardcoded platform dependent limits on absolute path lengths for module filenames - 255 in the case of Windows)
msg367290 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-04-26 02:19
With no activity here in over 5 years and no confirmation on any version but 2.7, I'm closing the issue.  If it can be confirmed on 3.8, the issue can be reopened.
History
Date User Action Args
2020-04-26 02:19:18zach.waresetstatus: open -> closed

nosy: + zach.ware
messages: + msg367290

resolution: out of date
stage: resolved
2020-03-18 18:21:52brett.cannonsetnosy: - brett.cannon
2014-11-27 02:34:16ncoghlansettype: crash -> behavior
2014-11-27 01:36:17ncoghlansetnosy: + christian.heimes, sbt
messages: + msg231741
2014-11-27 00:47:54James.Burkesetnosy: + James.Burke
messages: + msg231740
2014-11-15 20:50:00inlinestylesetnosy: + inlinestyle
messages: + msg231217
2014-10-01 23:49:56pitrousetnosy: + pitrou, ncoghlan
messages: + msg228140
2014-10-01 20:46:06nmz787setnosy: + nmz787
messages: + msg228115
2014-06-04 14:42:38r.david.murraysetnosy: + brett.cannon, eric.snow
2014-05-30 19:55:49ColinPDavidsoncreate