classification
Title: ImportError hides real error when there too many open files during an import
Type: behavior Stage: resolved
Components: Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.snow, jinty, r.david.murray
Priority: normal Keywords: patch

Created on 2013-01-16 16:18 by jinty, last changed 2020-03-06 19:54 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
patch-Python-import.c.diff jinty, 2013-01-16 16:18 Patch used during debugging
Messages (4)
msg180092 - (view) Author: Brian Sutherland (jinty) Date: 2013-01-16 16:18
When running Python inside PostgreSQL using plpython on OSX 10.7.5 I started coming across very strange and apparently random ImportErrors. For example, failing to find the stat module while importing site:

     Traceback (most recent call last):
       File "/Users/jinty/.buildout/eggs/setuptools-0.6c11-py2.7.egg/site.py", line 73, in <module>
         __boot()
       File "/Users/jinty/.buildout/eggs/setuptools-0.6c11-py2.7.egg/site.py", line 2, in __boot
         import sys, imp, os, os.path
       File "/Users/jinty/src/mp/lib/python2.7/os.py", line 49, in <module>
         import posixpath as path
       File "/Users/jinty/src/mp/lib/python2.7/posixpath.py", line 15, in <module>
         import stat
     ImportError: No module named stat

I debugged this by using PYTHONVERBOSE and modifying import.c with the attached patch. I found that stat.py was not found because fopen() failed with "Too many open files". There were not enough open files because OSX has insanely low limits and PostgreSQL was using a large chunk of that. ulimit -n 4096 resolved the errors. I spent a LOT of time trying to figure that out (see the thread at http://www.postgresql.org/message-id/20130114163014.GA600@Brians-MacBook-Air.local).

The bug I wish to report is that the real error (Too many open files) is hidden by "ImportError: No module named stat". For anyone who does not want to modify import.c and rebuild python, it is almost impossible to figure out what is really happening.
msg180093 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-16 16:49
I wonder if importlib replicates this behavior, it may need fixing as well.
msg180095 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-01-16 18:27
A quick trace through importlib._bootstrap through hg.python.org would suggest that the OSError would propagate when accessing source (OSError is swallowed when you try and write bytecode, but that's legitimate semantics).
msg363536 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-03-06 19:54
Now that importlib is import and it would raise OSError I'm closing this as fixed.
History
Date User Action Args
2020-03-06 19:54:02brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg363536

stage: resolved
2013-01-16 18:27:07brett.cannonsetmessages: + msg180095
2013-01-16 16:49:42r.david.murraysetnosy: + eric.snow, r.david.murray, brett.cannon
messages: + msg180093
2013-01-16 16:18:05jintycreate