classification
Title: ImportError silences low-level OS errors
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, brett.cannon, eric.snow, ncoghlan, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2011-10-16 20:25 by pitrou, last changed 2013-02-02 07:21 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
fopen_err.patch vstinner, 2011-10-16 21:09 review
Messages (10)
msg145639 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-16 20:25
Not sure this is by design or not, but I wanted to report this issue (it's rather hard to diagnose). Here the file descriptor limit is reached, and "import" raises an ImportError while it would be more helpful to let the original OS error slip through (which, here, is EMFILE):

>>> import resource
>>> resource.setrlimit(resource.RLIMIT_NOFILE, (2, 100))
>>> import encodings.idna
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'idna'
>>> resource.setrlimit(resource.RLIMIT_NOFILE, (100, 100))
>>> import encodings.idna
>>>
msg145640 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-10-16 21:09
_Py_fopen() returns NULL without raising an exception if fopen() fails.

Attached patch raises a OSError if fopen() if errno is different than ENOENT. I don't know if ENOENT is the right error number on Windows. Should we ignore more error numbers, like EACCESS?
msg145642 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-16 21:19
> _Py_fopen() returns NULL without raising an exception if fopen() fails.
> 
> Attached patch raises a OSError if fopen() if errno is different than
> ENOENT. I don't know if ENOENT is the right error number on Windows.

It seems so.

>  Should we ignore more error numbers, like EACCESS?

I suggest also ignoring ENOTDIR and EISDIR. Not sure about EACCES.
msg145645 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-10-16 21:36
> I suggest also ignoring ENOTDIR and EISDIR. Not sure about EACCES.

We should maybe mimick the io module:


>>> open('Lib')
IsADirectoryError: [Errno 21] Is a directory: 'Lib'

In the io module, opening a directory raises an exception. In the import machinery, an explicit check is done (stat+ISDIR) before opening the file. The file is ignored if it is a directory. We can maybe remove the test before opening the file, and handle IsADirectoryError exception instead.

--

I don't think that fopen() can fails with ENOTDIR. rmdir() can fail with ENOTDIR.

--

For EACCES: Python ignores currently these errors. I was already surprised by this behaviour. My use case: I have two users (haypo and fusil). I "installed" a module of haypo user into /usr/lib/python... using symlink, but fusil doesn't have access to /home/haypo. When fusil tries to load a module, import fails without any warning or error.

Fail with an error is maybe not a good idea because it's not because a directory of sys.path cannot be read, than the import will fail. The module can be found in another (accessible) directory.
msg145646 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-16 21:37
> I don't think that fopen() can fails with ENOTDIR.

According to POSIX:

[ENOTDIR]
    A component of the path prefix is not a directory, or the filename
argument contains at least one non- <slash> character and ends with one
or more trailing <slash> characters and the last pathname component
names an existing file that is neither a directory nor a symbolic link
to a directory.

http://pubs.opengroup.org/onlinepubs/9699919799/functions/fopen.html
msg145729 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2011-10-17 17:02
Are you suggesting raising the OSError (or something else) rather than an ImportError?  If so, would it make sense to chain the exception instead.  That way the existing code that expects ImportError continues to work, while still allowing access to the original error through __cause__.
msg145732 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-17 17:17
> Are you suggesting raising the OSError (or something else) rather than
> an ImportError?  If so, would it make sense to chain the exception
> instead.  That way the existing code that expects ImportError
> continues to work

Does any existing code rely on ImportError being raised when there's a
rare OS error of some sort?
msg181116 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-02-01 21:53
Is this still an issue in (at least) 3.4 with the new I/O exceptions? And I feel like there was another bug about this.
msg181135 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-02-02 06:36
Are you thinking of issue 15833?
msg181138 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-02-02 07:21
In 3.3 and 3.4, exceptions are not silenced anymore:

>>> import resource
>>> resource.setrlimit(resource.RLIMIT_NOFILE, (2, 100))
>>> import encodings.idna
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1584, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1551, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1049, in load_module
  File "<frozen importlib._bootstrap>", line 1030, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 883, in _load_module
  File "<frozen importlib._bootstrap>", line 1008, in get_code
  File "<frozen importlib._bootstrap>", line 1058, in get_data
OSError: [Errno 24] Too many open files: '/home/antoine/cpython/default/Lib/encodings/idna.py'


Given that fixing the issue in bugfix branches would be a slight behaviour change, I think we can close this issue.
History
Date User Action Args
2013-02-02 07:21:04pitrousetstatus: open -> closed
versions: + Python 2.7, - Python 3.3
messages: + msg181138

resolution: out of date
stage: resolved
2013-02-02 06:36:45eric.snowsetmessages: + msg181135
2013-02-01 21:53:38brett.cannonsetmessages: + msg181116
2011-10-17 17:17:40pitrousetmessages: + msg145732
2011-10-17 17:02:25eric.snowsetmessages: + msg145729
2011-10-17 14:05:22Arfreversetnosy: + Arfrever
2011-10-17 02:19:44eric.snowsetnosy: + eric.snow
2011-10-16 21:37:47pitrousetmessages: + msg145646
2011-10-16 21:36:23vstinnersetmessages: + msg145645
2011-10-16 21:19:13pitrousetmessages: + msg145642
2011-10-16 21:09:25vstinnersetfiles: + fopen_err.patch

nosy: + vstinner
messages: + msg145640

keywords: + patch
2011-10-16 20:25:16pitroucreate