classification
Title: most failures to write byte-compiled file no longer suppressed
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, brett.cannon, eric.snow, haypo, neologix, pitrou, python-dev, rpetrov, sbt, trent
Priority: normal Keywords: 3.3regression, patch

Created on 2012-08-31 16:57 by sbt, last changed 2012-10-16 14:51 by neologix. This issue is now closed.

Files
File name Uploaded Description Edit
0016-CROSS-reload-may-fail-with-operation-on-closed-file-.patch rpetrov, 2012-10-06 00:00
import_error.diff neologix, 2012-10-07 09:51 review
Messages (20)
msg169564 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-08-31 16:57
As discussed in

    http://bugs.python.org/issue15819

trying to run python built outside a read-only source directory fails for me because OSError(EIO, ...) is raised when importlib tries to write the byte compiled file.  (I built python in a Linux VM using VMware Player which only had read-only access to the source directory.)

Currently importlib only ignores PermissionError (EACCES, EPERM) and FileExistsError (EEXIST):

    http://hg.python.org/cpython/file/d54f047312a8/Lib/importlib/_bootstrap.py#l1080

Under Python 3.2 all failures are ignored:

    http://hg.python.org/cpython/file/69952b5599b5/Python/import.c#l905

so this seems to be a regression.
msg169584 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-31 21:12
Raising EIO in a read-only dir sounds weird.
msg169588 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-08-31 22:40
So it might be a change, but I don't know if I would claim it is a regression that needs changing. If you can't write bytecode files for some reason other than permissions or the file already exists then there is something wrong with your setup that you might not be expecting.

I don't know what this EIO error code is, but it sounds non-standard for the situation and thus not something we should blindly be ignoring.
msg169594 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-08-31 23:58
I was thinking along the same lines, though it might make sense for EROFS and a few others.  (http://aplawrence.com/Unixart/errors.html)
msg172120 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-10-05 19:22
We should probably catch all OSErrors, and log a warning in verbose mode, as suggested by Antoine.
Catching individual errnos is tedious and error-prone.
msg172121 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-10-05 19:24
Issue #16139 has been marked as a duplicate of this issue.
msg172155 - (view) Author: Roumen Petrov (rpetrov) * Date: 2012-10-06 00:00
Did 0016-CROSS-reload-may-fail-with-operation-on-closed-file-.patch fix issue ?
msg172285 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-10-07 09:51
Here's a patch.
msg172286 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-10-07 09:53
Roumen, are you sure you submitted your patch to the right issue?
msg172289 - (view) Author: Roumen Petrov (rpetrov) * Date: 2012-10-07 10:47
Yes .
Charles, lest assume that all other issues with build system, are resolved and source tree is ready for use. So in this situation I could run python , I could build all and I could run tests with an additional patch TEMPDIR is relative to build tree (patch 0014-* from recent archive uploaded to issue #3754) . Note without you patch. 

In my build environment test_imp fail on test_source.
So what is result with you patch on this test ?
msg172364 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-10-08 07:51
> Yes .
> Charles, lest assume that all other issues with build system, are resolved and source tree is ready for use. So in this situation I could run python , I could build all and I could run tests with an additional patch TEMPDIR is relative to build tree (patch 0014-* from recent archive uploaded to issue #3754) . Note without you patch.
>
> In my build environment test_imp fail on test_source.
> So what is result with you patch on this test ?

Sorry, I'm not a native English speaker, and I don't understand what
you're talking about.
The issue at hand is that bytecode creation only catches EPERM and
EEXIST, but other errors can be raised on a read-only filesystem
(EROFS, EIO in some strange cases).
msg172405 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-08 19:13
Charles-François's patch looks ok to me. I don't know if this warrants adding a test.
msg172597 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-10-10 17:59
> Charles-François's patch looks ok to me. I don't know if this warrants adding a test.

Yes, and triggering a failure other than a permission error (which is
probably already tested) can be difficult.

I'm abroad and won't be able to commit it until at least a week, so if
someone's willing to commit it before go ahead :-)
msg172818 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-13 17:36
FWIW, I just ran into the following on Solaris 10:

% dbx python
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.9' in your .dbxrc
Reading python
Reading ld.so.1
Reading libsocket.so.1
Reading libnsl.so.1
Reading libintl.so.1
Reading librt.so.1
Reading libdl.so.1
Reading libsendfile.so.1
Reading libm.so.2
Reading libthread.so.1
Reading libc.so.1
Reading libaio.so.1
Reading libmd.so.1
(dbx) run                                                                    
Running: python 
(process id 17819)
Reading libc_psr.so.1
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1558, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 854, in _load_module
  File "<frozen importlib._bootstrap>", line 990, in get_code
  File "<frozen importlib._bootstrap>", line 1051, in _cache_bytecode
  File "<frozen importlib._bootstrap>", line 1065, in set_data
OSError: [Errno 30] Read-only file system: '/home/cpython/nfs/src/3.x/Lib/encodings/__pycache__'
t@1 (l@1) signal ABRT (Abort) in __lwp_kill at 0xffffffff7e2dc2c0
0xffffffff7e2dc2c0: __lwp_kill+0x0008:  bcc,a,pt  %icc,__lwp_kill+0x18  ! 0xffffffff7e2dc2d0
Current function is Py_FatalError
 2360       abort();
(dbx) q 
q: not found
(dbx) quit


Testing the 'import_error.diff' patch now...
msg172820 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-13 17:52
That... didn't work.  Also applied rpetrov's patch and that made no difference:

% ./python Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1558, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 854, in _load_module
  File "<frozen importlib._bootstrap>", line 990, in get_code
  File "<frozen importlib._bootstrap>", line 1051, in _cache_bytecode
  File "<frozen importlib._bootstrap>", line 1065, in set_data
OSError: [Errno 30] Read-only file system: '/home/cpython/nfs/src/3.x/Lib/encodings/__pycache__'
zsh: IOT instruction (core dumped)  ./python
msg172823 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-13 18:14
Charles: your patch is fine.  +1.

My Solaris failures can be traced back to http://bugs.python.org/issue15819.  Sorry for the noise.
msg173030 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-16 11:49
New changeset 53ce30b62de2 by Trent Nelson in branch '3.3':
Issue #15833: don't raise an exception if importlib can't write byte-compiled
http://hg.python.org/cpython/rev/53ce30b62de2
msg173031 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-16 12:03
New changeset 36b2ca7dc893 by Trent Nelson in branch 'default':
Merge issue #15833: don't raise an exception if importlib can't write
http://hg.python.org/cpython/rev/36b2ca7dc893
msg173032 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-16 12:05
Charles' patch applied in 3.3 and merged to 3.x.  Thanks Charles.
msg173045 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-10-16 14:51
> Charles' patch applied in 3.3 and merged to 3.x.  Thanks Charles.

Thanks for taking care of this!
History
Date User Action Args
2012-10-16 14:51:26neologixsetmessages: + msg173045
2012-10-16 12:05:06trentsetstatus: open -> closed
resolution: fixed
messages: + msg173032

stage: resolved
2012-10-16 12:03:57python-devsetmessages: + msg173031
2012-10-16 11:49:14python-devsetnosy: + python-dev
messages: + msg173030
2012-10-13 18:14:36trentsetmessages: + msg172823
2012-10-13 17:52:11trentsetmessages: + msg172820
2012-10-13 17:36:30trentsetnosy: + trent
messages: + msg172818
2012-10-10 17:59:32neologixsetmessages: + msg172597
2012-10-08 19:13:15pitrousetmessages: + msg172405
2012-10-08 07:51:58neologixsetmessages: + msg172364
2012-10-07 10:47:39rpetrovsetmessages: + msg172289
2012-10-07 09:53:26neologixsetmessages: + msg172286
2012-10-07 09:51:51neologixsetfiles: + import_error.diff

messages: + msg172285
2012-10-06 00:00:13rpetrovsetfiles: + 0016-CROSS-reload-may-fail-with-operation-on-closed-file-.patch

nosy: + rpetrov
messages: + msg172155

keywords: + patch
2012-10-05 19:24:09hayposetnosy: + haypo
messages: + msg172121
2012-10-05 19:22:35neologixsetnosy: + neologix
messages: + msg172120
2012-10-05 19:18:36neologixlinkissue16139 superseder
2012-09-04 07:49:14Arfreversetnosy: + Arfrever
2012-08-31 23:58:34eric.snowsetnosy: + eric.snow
messages: + msg169594
2012-08-31 22:40:04brett.cannonsetmessages: + msg169588
2012-08-31 21:12:26pitrousetnosy: + pitrou
messages: + msg169584
2012-08-31 16:57:26sbtcreate