classification
Title: Writing a pyc file is not atomic under Windows
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, haypo, ncoghlan, neologix, pitrou, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2011-11-13 01:20 by pitrou, last changed 2011-11-15 19:48 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
winimport.patch pitrou, 2011-11-13 01:20 review
Messages (5)
msg147540 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-13 01:20
#13146 solved the issue of writing pyc files under POSIX. Under Windows, the problem still exists, as the following buildbot failure shows:

[317/360] test_multiprocessing
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\forking.py", line 373, in main
    prepare(preparation_data)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\forking.py", line 499, in prepare
    '__parents_main__', file, path_name, etc
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\regrtest.py", line 175, in <module>
    import packaging.command
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\packaging\command\__init__.py", line 4, in <module>
    from packaging.util import resolve_name
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\packaging\util.py", line 5, in <module>
    import csv
EOFError: EOF read where not expected
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\forking.py", line 373, in main
    prepare(preparation_data)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\forking.py", line 499, in prepare
    '__parents_main__', file, path_name, etc
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\regrtest.py", line 175, in <module>
    import packaging.command
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\packaging\command\__init__.py", line 4, in <module>
    from packaging.util import resolve_name
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\packaging\util.py", line 5, in <module>
    import csv
EOFError: EOF read where not expected
[etc.]

(from http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/5551/steps/test/logs/stdio)

Attached patch uses MoveFileEx to perform a (hopefully atomic) rename from a temporary file when creating the pyc file.
The same strategy cannot be created for importlib since MoveFileEx isn't exposed at the Python level (we could delete the tmp file if renaming fails, though; it simply means another process beat us to it, which shouldn't be a problem here).
msg147690 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-11-15 17:05
I say go with the deletion for importlib. If the failure is a sign of success, then just go with it and not worry about the failure. __pycache__ guarantees that we are not losing out by clobbering some other Python version's pyc file.
msg147693 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-15 18:20
New changeset b75b41237380 by Antoine Pitrou in branch 'default':
Issue #13392: Writing a pyc file should now be atomic under Windows as well.
http://hg.python.org/cpython/rev/b75b41237380
msg147694 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-15 18:21
Ok, done. Let's see if that fixes the sporadic failures on the buildbots.
msg147700 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-15 19:48
New changeset 12940d9f8031 by Antoine Pitrou in branch 'default':
Fix regression under Windows following b75b41237380 (from issue #13392)
http://hg.python.org/cpython/rev/12940d9f8031
History
Date User Action Args
2011-11-15 19:48:03python-devsetmessages: + msg147700
2011-11-15 18:21:20pitrousetstatus: open -> closed
resolution: fixed
messages: + msg147694

stage: patch review -> resolved
2011-11-15 18:20:13python-devsetmessages: + msg147693
2011-11-15 17:05:35brett.cannonsetmessages: + msg147690
2011-11-13 01:20:52pitroucreate