classification
Title: importlib/_bootstrap.py write_bytecode raises an IOError if it can't open the .pyc file for writing
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 3.1
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, dmalcolm, pitrou
Priority: normal Keywords: patch

Created on 2009-10-22 20:37 by dmalcolm, last changed 2009-11-07 23:57 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
python-3.1.1-importlib-fix-handling-of-readonly-pyc-files.patch dmalcolm, 2009-10-22 20:37 Patch to extend exception handling to also cover failure to open the .pyc for writing
Messages (5)
msg94371 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2009-10-22 20:37
I'm working on an RPM package of Python 3.1.1 for Fedora [1].  Within my RPMs are 
precompiled .pyc and .pyo files that are not writable by users, and are hardlinked 
together if they have equal content.  For example:
$ ls -al /usr/lib/python3.1/tkinter/__init__.*
-rw-r--r--. 1 root root 157859 2009-10-22 15:43 
/usr/lib/python3.1/tkinter/__init__.py
-rw-r--r--. 2 root root 226817 2009-10-22 15:42 
/usr/lib/python3.1/tkinter/__init__.pyc
-rw-r--r--. 2 root root 226817 2009-10-22 15:42 
/usr/lib/python3.1/tkinter/__init__.pyo

I see numerous IOError permission errors running the regression test suite, with 
importlib/_bootstrap.py trying and failing to open the .pyc files for writing.  
The IOError bubbles up each time and causes the test to fail.

Here's a minimal reproducer (assuming such an install as above)
>>> from tkinter import Tcl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 151, in decorated
    return fxn(self, module)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 399, in load_module
    return self._load_module(module)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 324, in _load_module
    code_object = self.get_code(module.__name__)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 443, in get_code
    self.write_bytecode(fullname, data)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 171, in inner
    return method(self, name, *args, **kwargs)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 525, in write_bytecode
    file = _io.FileIO(bytecode_path, 'w')  # Assuming bytes.
IOError: [Errno 13] Permission denied: '/usr/lib/python3.1/tkinter/__init__.pyc'

Upon investigating Lib/importlib/_bootstrap.py, the logic for handling permission 
errors in "write_bytecode" appears to be wrong:
  - the initializer for _io.FileIO can raise an "IOError" if it can't open the 
.pyc file for writing
    - see Modules/_io/fileio.c:fileio_init, the exception is indeed raised if the 
call to:
        self->fd = open(name, flags, 0666);
    fails.

So it looks like the try/except clause needs to be extended to also cover opening 
the file for writing, as well as writing the bytes to it.

I'm attaching a patch that makes this change.

With this change, the above failing import succeeds, and an invocation of 
/usr/lib/python3.1/test/regrtest.py improves from:
276 tests OK.
39 tests failed:
    test_cmd_line test_codeccallbacks test_codecencodings_cn
    test_codecencodings_hk test_codecencodings_jp
    test_codecencodings_kr test_codecencodings_tw test_cprofile
    test_distutils test_docxmlrpc test_email test_heapq
    test_htmlparser test_httpservers test_imp test_lib2to3
    test_linecache test_modulefinder test_multiprocessing test_osx_env
    test_plistlib test_pyclbr test_pydoc test_runpy test_socket
    test_sqlite test_sundry test_tcl test_threading_local test_tk
    test_ttk_guionly test_ttk_textonly test_uuid test_warnings
    test_wsgiref test_xml_etree_c test_xmlrpc test_xmlrpc_net
    test_zipfile
to:
304 tests OK.
10 tests failed:
    test_email test_httpservers test_imp test_lib2to3 test_linecache
    test_socket test_tk test_ttk_guionly test_ttk_textonly
    test_zipfile

(I'm working on these other failures; some of them are due to errors in my 
packaging)

FWIW the initial checkin of py3k/Lib/importlib/_bootstrap.py appears to have the 
initialization of the FileIO outside the try/except block:
http://svn.python.org/view/python/branches/py3k/Lib/importlib/_bootstrap.py?
view=markup&pathrev=68698

though at that time it was _fileio._FileIO, rather than _io.FileIO.  (did FileIO 
only test for perms upon writing, and change behavior to test upon opening?  or 
has this case always led to this failure?  or am I misreading this?)

Thanks
Dave

[1] https://bugzilla.redhat.com/show_bug.cgi?id=526126
msg94374 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2009-10-23 04:40
Thanks for the report and potential fix, Dave. I'm swamped at the moment 
but I will get to this.
msg94381 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-23 12:45
Wouldn't the problem disappear, though, if tkinter/__init__.py didn't
claim to be fresher than tkinter/__init__.pyc? Also, it would make the
pyc/pyo files actually useful, rather than having Python recompile the
source files again and again.
(see the modification dates in your listing)
msg94388 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2009-10-23 15:24
Aha!  Thanks pitrou, looks like I managed to mess up my packaging (I was
wondering why no no-one else had run into this, looks like no-one else
made this mistake!).
msg95028 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2009-11-07 23:57
Committed to py3k in r76146 and r76147 for 3.1.

Thanks for the detailed report, Dave!
History
Date User Action Args
2009-11-07 23:57:58brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg95028
2009-10-23 15:24:08dmalcolmsetmessages: + msg94388
2009-10-23 12:45:05pitrousetnosy: + pitrou
messages: + msg94381
2009-10-23 04:41:20brett.cannonsetpriority: normal
stage: test needed
2009-10-23 04:40:08brett.cannonsetmessages: + msg94374
2009-10-23 00:04:42benjamin.petersonsetassignee: brett.cannon

nosy: + brett.cannon
2009-10-22 20:43:31dmalcolmsettitle: importlib/_bootstrap.py write_bytecode fails if it can't open the .pyc file for writing -> importlib/_bootstrap.py write_bytecode raises an IOError if it can't open the .pyc file for writing
2009-10-22 20:37:06dmalcolmcreate