Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IOError or OSError in test_multiprocessing #53781

Closed
florentx mannequin opened this issue Aug 11, 2010 · 11 comments
Closed

IOError or OSError in test_multiprocessing #53781

florentx mannequin opened this issue Aug 11, 2010 · 11 comments
Assignees
@brettcannon
Labels
stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@florentx
Copy link
Mannequin

florentx mannequin commented Aug 11, 2010

BPO 9572
Nosy @warsaw, @brettcannon, @florentx
Files
  • issue9572_oserror.diff: Patch, apply to 3.x
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/brettcannon'
closed_at = <Date 2010-08-26.21:09:01.328>
created_at = <Date 2010-08-11.20:09:13.629>
labels = ['tests', 'type-bug', 'library']
title = 'IOError or OSError in test_multiprocessing'
updated_at = <Date 2010-08-26.21:09:01.326>
user = 'https://github.com/florentx'

    bugs.python.org fields:

    activity = <Date 2010-08-26.21:09:01.326>
actor = 'brett.cannon'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2010-08-26.21:09:01.328>
closer = 'brett.cannon'
components = ['Library (Lib)', 'Tests']
creation = <Date 2010-08-11.20:09:13.629>
creator = 'flox'
dependencies = []
files = ['18649']
hgrepos = []
issue_num = 9572
keywords = ['patch', 'buildbot']
message_count = 11.0
messages = ['113626', '113628', '114472', '114713', '114771', '114828', '114946', '114955', '114960', '114961', '115025']
nosy_count = 4.0
nosy_names = ['barry', 'brett.cannon', 'flox', 'sniffer']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9572'
versions = ['Python 3.2']
    @florentx
    Copy link
    Mannequin Author

    florentx mannequin commented Aug 11, 2010

    This error occurred on "x86 Ubuntu 3.x" buildbot.
    This is the 1st test on this run.
    http://www.python.org/dev/buildbot/all/builders/x86%20Ubuntu%203.x/builds/1699/steps/test/logs/stdio

    ./python -Wd -E -bb ./Lib/test/regrtest.py -uall -rwW -l 
== CPython 3.2a1+ (py3k:83951, Aug 11 2010, 15:26:40) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)]
==   Linux-2.6.31.5-linode21-i686-with-debian-lenny-sid little-endian
==   /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/build/test_python_8258
Using random seed 426296
[  1/346] test_multiprocessing
Process Process-22:
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/multiprocessing/process.py", line 233, in _bootstrap
    self.run()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_multiprocessing.py", line 1253, in _putter
    manager.connect()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/multiprocessing/managers.py", line 478, in connect
    dispatch(conn, None, 'dummy')
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/multiprocessing/managers.py", line 79, in dispatch
    kind, result = c.recv()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/multiprocessing/connection.py", line 408, in recv
    s = self._conn.recv_bytes()
EOFError
test test_multiprocessing failed -- Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/importlib/_bootstrap.py", line 486, in set_data
    with _closing(_io.FileIO(path, 'wb')) as file:
IOError: [Errno 2] No such file or directory: '/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/xmlrpc/__pycache__/__init__.cpython-32.pyc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_multiprocessing.py", line 1266, in test_rapid_restart
    queue = manager.get_queue()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/multiprocessing/managers.py", line 644, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/multiprocessing/managers.py", line 542, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/multiprocessing/connection.py", line 427, in XmlClient
    import xmlrpc.client as xmlrpclib
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/importlib/_bootstrap.py", line 450, in load_module
    return self._load_module(fullname)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/importlib/_bootstrap.py", line 155, in decorated
    return fxn(self, module, *args, **kwargs)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/importlib/_bootstrap.py", line 344, in _load_module
    code_object = self.get_code(name)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/importlib/_bootstrap.py", line 437, in get_code
    self.set_data(bytecode_path, data)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/importlib/_bootstrap.py", line 498, in set_data
    _os.mkdir(directory)
OSError: [Errno 17] File exists: '/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/xmlrpc/__pycache__'

    Re-running test test_multiprocessing in verbose mode
    ...
    ----------------------------------------------------------------------
    Ran 127 tests in 34.091s

    OK (skipped=9)

    @florentx florentx mannequin added stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Aug 11, 2010
    @florentx
    Copy link
    Mannequin Author

    florentx mannequin commented Aug 11, 2010

    I succeeded to reproduce one time on Debian Lenny (64 bits).

    ~ $ make pycremoval
    find . -name '*.py[co]' -exec rm -f {} ';'
    find . -name '__pycache__' -exec rmdir {} '+'

    ~ $ ./python -Wd -E -bb ./Lib/test/regrtest.py -uall -rwW -l test_multiprocessing
Using random seed 8192992
[1/1] test_multiprocessing
Process Process-22:
Traceback (most recent call last):
  File "./Lib/multiprocessing/process.py", line 233, in _bootstrap
    self.run()
  File "./Lib/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "./Lib/test/test_multiprocessing.py", line 1253, in _putter
    manager.connect()
  File "./Lib/multiprocessing/managers.py", line 478, in connect
    dispatch(conn, None, 'dummy')
  File "./Lib/multiprocessing/managers.py", line 79, in dispatch
    kind, result = c.recv()
  File "./Lib/multiprocessing/connection.py", line 408, in recv
    s = self._conn.recv_bytes()
EOFError
test test_multiprocessing failed -- Traceback (most recent call last):
  File "./Lib/importlib/_bootstrap.py", line 486, in set_data
    with _closing(_io.FileIO(path, 'wb')) as file:
IOError: [Errno 2] No such file or directory: './Lib/email/__pycache__/__init__.cpython-32.pyc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./Lib/test/test_multiprocessing.py", line 1266, in test_rapid_restart
    queue = manager.get_queue()
  File "./Lib/multiprocessing/managers.py", line 644, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "./Lib/multiprocessing/managers.py", line 542, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "./Lib/multiprocessing/connection.py", line 427, in XmlClient
    import xmlrpc.client as xmlrpclib
  File "./Lib/xmlrpc/client.py", line 137, in <module>
    import http.client
  File "./Lib/http/client.py", line 69, in <module>
    import email.parser
  File "./Lib/importlib/_bootstrap.py", line 450, in load_module
    return self._load_module(fullname)
  File "./Lib/importlib/_bootstrap.py", line 155, in decorated
    return fxn(self, module, *args, **kwargs)
  File "./Lib/importlib/_bootstrap.py", line 344, in _load_module
    code_object = self.get_code(name)
  File "./Lib/importlib/_bootstrap.py", line 437, in get_code
    self.set_data(bytecode_path, data)
  File "./Lib/importlib/_bootstrap.py", line 498, in set_data
    _os.mkdir(directory)
OSError: [Errno 17] File exists: './Lib/email/__pycache__'

    ...

    But next attempts were successful.

    @brettcannon brettcannon self-assigned this Aug 12, 2010
    @brettcannon
    Copy link
    Member

    I think what I need to do is greatly simplify the directory creation code in set_data and make it much more robust against potential race conditions against other Python processes. I think as long as I just stop trying to make a directory when it is found to exist this should be okay. Might also want to file lock the bytecode file as import.c does through open_exclusive.

    @brettcannon
    Copy link
    Member

    Fixed in r84271. I will open another issue for the locking stuff.

    @sniffer
    Copy link
    Mannequin

    sniffer mannequin commented Aug 24, 2010

    I was just playing around with the changes you made and it seems you've overlooked a small error/typo around line 502:

    502 : n elif errno == errno.EACCES:

    quick fix:

    502 : n elif exc.errno == errno.EACCES:

    Hope this helps.

    @brettcannon
    Copy link
    Member

    Fixed in r84304. Thanks for catching that, Łukasz.

    @florentx
    Copy link
    Mannequin Author

    florentx mannequin commented Aug 25, 2010

    This one looks very similar (r84315 on i386 Ubuntu 3.x buildbot):

    ./python -Wd -E -bb ./Lib/test/regrtest.py -uall -rwW -l 
== CPython 3.2a1+ (py3k:84315, Aug 25 2010, 17:07:54) [GCC 4.6.0 20100814 (experimental) [trunk revision 163239]]
==   Linux-2.6.32-24-server-i686-with-debian-squeeze-sid little-endian
==   /scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/build/test_python_17577
Using random seed 4479416
[  1/346] test_thread
[  2/346] test_hash
[  3/346] test_zipfile64
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
[  4/346] test_dbm_gnu
test_dbm_gnu skipped -- No module named _gdbm
[  5/346] test_funcattrs
[  6/346] test_http_cookies
[  7/346] test_multiprocessing
Process Process-24:
Traceback (most recent call last):
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/multiprocessing/process.py", line 233, in _bootstrap
    self.run()
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/test/test_multiprocessing.py", line 1260, in _putter
    manager.connect()
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/multiprocessing/managers.py", line 477, in connect
    conn = Client(self._address, authkey=self._authkey)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/multiprocessing/connection.py", line 427, in XmlClient
    import xmlrpc.client as xmlrpclib
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 450, in load_module
    return self._load_module(fullname)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 155, in decorated
    return fxn(self, module, *args, **kwargs)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 344, in _load_module
    code_object = self.get_code(name)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 437, in get_code
    self.set_data(bytecode_path, data)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 495, in set_data
    _os.mkdir(parent)
OSError: [Errno 17] File exists: '/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/xmlrpc/__pycache__'
test test_multiprocessing failed -- Traceback (most recent call last):
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/test/test_multiprocessing.py", line 1273, in test_rapid_restart
    queue = manager.get_queue()
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/multiprocessing/managers.py", line 644, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/multiprocessing/managers.py", line 542, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/multiprocessing/connection.py", line 427, in XmlClient
    import xmlrpc.client as xmlrpclib
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 450, in load_module
    return self._load_module(fullname)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 155, in decorated
    return fxn(self, module, *args, **kwargs)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 344, in _load_module
    code_object = self.get_code(name)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 437, in get_code
    self.set_data(bytecode_path, data)
  File "/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/importlib/_bootstrap.py", line 495, in set_data
    _os.mkdir(parent)
OSError: [Errno 17] File exists: '/scratch/pybot-buildarea/3.x.klose-ubuntu-i386/build/Lib/xmlrpc/__pycache__'

    http://www.python.org/dev/buildbot/builders/i386 Ubuntu 3.x/builds/2005

    @florentx florentx mannequin reopened this Aug 25, 2010
    @florentx florentx mannequin changed the title IOError in test_multiprocessing IOError or OSError in test_multiprocessing Aug 25, 2010
    @sniffer
    Copy link
    Mannequin

    sniffer mannequin commented Aug 26, 2010

    I Had the same issue while using the lib. I had to replace every: IOError with (IOError, OSError) to catch both and just then compare exc.errno.

    Also in your r84271 commit, on line 507 you did:

    507 : with _io.FileIO(path, 'wb') as file:

    shouldn't it be:

    507 : with _closing(_io.FileIO(path, 'wb')) as file:

    as in previous commit? In case of an error, you're not closing the file properly. Or am I reading that wrong?

    Cheers.

    @florentx
    Copy link
    Mannequin Author

    florentx mannequin commented Aug 26, 2010

    Łukasz,

    _closing is not necessary on FileIO instances. The class already declares an __exit__ method which takes care of closing file.

    >>> import io
>>> f = io.FileIO('/tmp/test_closing', 'wb')
>>> f.closed
False

>>> f.__exit__()
>>> f.closed
True

    Since both IOError and OSError are direct subclasses of EnvironmentError, we can use this in the except clause.

    @florentx
    Copy link
    Mannequin Author

    florentx mannequin commented Aug 26, 2010

    Patch.

    @brettcannon
    Copy link
    Member

    r84329 has the fix. I decided to catch IOError and OSError separately rather than EnvironmentError as a blanket 'except' clause; explicit is better than implicit and all.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @brettcannon brettcannon

    Labels
    stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @brettcannon