classification
Title: Unexpected FileNotFoundError when current directory is removed
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: Arfrever, Christopher Meng, brett.cannon, eric.snow, martin.panter, python-dev, r.david.murray, serhiy.storchaka, terry.reedy
Priority: normal Keywords:

Created on 2014-11-10 03:56 by martin.panter, last changed 2015-09-23 04:42 by Christopher Meng. This issue is now closed.

Messages (18)
msg230933 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-11-10 03:56
I encountered this when I added a unit test case that invoked os.chdir() with a temporary directory on Linux. After the directory was removed, some of the subsequent test cases failed, although I don’t think they should depend on a particular CWD.

I suspect the main problem might be code that is trying to dynamically import modules, and the interpreter is trying to search for modules in the current directory. I would expect it to happily go on to the other standard module directories or raise ImportError, just like if the current directory is valid but empty, or an nonexistent directory is in the module search path list.

Code to set up missing CWD:

import os
from tempfile import TemporaryDirectory
with TemporaryDirectory() as dir:
    os.chdir(dir)

Quick recovery:

os.chdir("/")

Examples of failures:

>>> "\N{COPYRIGHT SIGN}"
  File "<stdin>", line 1
SyntaxError: (unicode error) \N escapes not supported (can't load unicodedata module)
>>> datetime.strptime("", "")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2222, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 2164, in _find_spec
  File "<frozen importlib._bootstrap>", line 1940, in find_spec
  File "<frozen importlib._bootstrap>", line 1911, in _get_spec
  File "<frozen importlib._bootstrap>", line 1879, in _path_importer_cache
FileNotFoundError: [Errno 2] No such file or directory
>>> HTTPConnection("localhost").request("GET", "/")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/http/client.py", line 1090, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.4/http/client.py", line 1086, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.4/http/client.py", line 924, in _send_output
    self.send(msg)
  File "/usr/lib/python3.4/http/client.py", line 859, in send
    self.connect()
  File "/usr/lib/python3.4/http/client.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python3.4/socket.py", line 491, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.4/socket.py", line 530, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
  File "/usr/lib/python3.4/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/usr/lib/python3.4/encodings/idna.py", line 3, in <module>
    import stringprep, re, codecs
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2222, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 2164, in _find_spec
  File "<frozen importlib._bootstrap>", line 1940, in find_spec
  File "<frozen importlib._bootstrap>", line 1911, in _get_spec
  File "<frozen importlib._bootstrap>", line 1879, in _path_importer_cache
FileNotFoundError: [Errno 2] No such file or directory

>>> from datetime import datetime
>>> from http.client import HTTPConnection
These two also generate the FileNotFoundError

My workaround is to add this to my test case:
self.addCleanup(os.chdir, os.getcwd())
msg230963 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-11-10 15:02
Looks like importlib doesn't handle the case of a directory on the path being deleted?  If so, I'm surprised this hasn't been reported before.
msg231161 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-11-14 14:38
So it isn't about importlib not handling missing directories on sys.path directly, it has to do with the fact that os.getcwd() raises FileNotFoundError when CWD is no longer valid and that is in a fundamental part of importlib that isn't worrying about non-existent directories.

We could be robust and simply have instances of os.getcwd() failing just move on, e.g. when '' is hit in sys.path and os.getcwd() raises an exception just give up. The other option is leaving this to raise FileNotFoundError but throwing a new copy with a better error message mentioning this is because the current working directory no longer exists. That would lead to easier debugging since paths on sys.path are typically obvious -- since site.py makes them absolute -- but I'm willing to bet people don't bother checking CWD is still good. So it's robustness vs. debugging when you make this mistake.

Any opinions on the matter?
msg231170 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-11-14 19:18
Well, once I've launched a program, regardless of whether or not I expected some stuff to come from the CWD, I generally don't think about whether or not the CWD might go away, and in a complex setup it would most likely be getting deleted by some other process.  So I think I would prefer an import that is of something that wasn't ever in the CWD to succeed if the CWD goes away.  I'm not even sure I'd want a warning.

What happens if a directory on the path disappears?  If that is treated as a non-error, then I think a missing CWD on the path should also be treated as a non-error.
msg231173 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-11-14 19:33
How about issue a verbose warning for possible debugging and continue for robustness?
msg231192 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-11-14 22:13
The only time I see a warning would be useful is if you intended to override a standard module with a module of the same name in the current directory. In all other cases I think it would be better to either generate an ImportError if the module is not found, or import it from wherever it is found. So I think a warning would not be useful in most cases.

Having any other non-existant directory in the search path is not an error and there is no warning either:
$ python3 -btWall
Python 3.4.2 (default, Oct  8 2014, 14:33:30) 
[GCC 4.9.1 20140903 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.insert(0, "/blaua")
>>> import sadface
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'sadface'
>>> import urllib
>>> # Interpreter = happy
...
msg231477 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-11-21 16:12
I have a patch to silence the exception and I'm running the test suite now.

I'm planning to keep this to a 3.5 fix and not changing the semantics in Python 3.4 as the fix is a little different from the standard "directory in sys.path is invalid" since '' is dynamic and I can see someone relying on the current exception bubbling out somehow and not caching the dead directory in sys.path_importer_cache.
msg231479 - (view) Author: Roundup Robot (python-dev) Date: 2014-11-21 17:19
New changeset 8558fff73032 by Brett Cannon in branch 'default':
Issue #22834: Have import suppress FileNotFoundError when the current
https://hg.python.org/cpython/rev/8558fff73032
msg231480 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-11-21 17:20
Along with fixing this I also updated the import reference to mention how the current working directory is handled.

Thanks to Martin for the report and everyone for their input!
msg231521 - (view) Author: Roundup Robot (python-dev) Date: 2014-11-22 17:04
New changeset d065e6474b67 by Zachary Ware in branch 'default':
Issue #22834: cwd can't not exist on Windows, skip the test
https://hg.python.org/cpython/rev/d065e6474b67
msg236092 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-16 06:52
Tests failed on Solaris:
http://buildbot.python.org/all/builders/AMD64%20Solaris%2011%20%5BSB%5D%203.x/builds/3895/steps/test/logs/stdio

======================================================================
ERROR: test_deleted_cwd (test.test_importlib.import_.test_path.Source_FinderTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/test/test_importlib/import_/test_path.py", line 167, in test_deleted_cwd
    os.chdir(path)
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/tempfile.py", line 711, in __exit__
    self.cleanup()
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/tempfile.py", line 715, in cleanup
    _shutil.rmtree(self.name)
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/shutil.py", line 474, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/shutil.py", line 472, in rmtree
    os.rmdir(path)
OSError: [Errno 22] Invalid argument: '/tmp/tmpsx3fm0t4'

----------------------------------------------------------------------
msg236122 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-02-17 03:23
I don’t have a Solaris to test this, but maybe changing the first half of the test to the following would work:

dir = tempfile.TemporaryDirectory()
self.addCleanup(dir.cleanup)  # In case removal after chdir() fails
self.addCleanup(os.chdir, os.getcwd())
os.chdir(dir.name)
try:
    dir.cleanup()
except OSError as err:  # Invalid argument on Solaris
    self.skipTest("Couldn't remove current directory: {}".format(err))

with util.import_state(...)
msg236289 - (view) Author: Roundup Robot (python-dev) Date: 2015-02-20 14:48
New changeset f4f2096ab6f8 by Brett Cannon in branch 'default':
Issue #22834: Fix a failing test under Solaris due to the platform not
https://hg.python.org/cpython/rev/f4f2096ab6f8
msg236292 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-02-20 15:26
Thanks for the suggestion, Martin. Went with a variant of what you proposed.
msg236294 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-20 15:32
May be better use errno.EINVAL?
msg236336 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-02-20 22:10
+1 to EINVAL, also the bug reference comment is redundant with the one at the top of the test case :)
msg236795 - (view) Author: Roundup Robot (python-dev) Date: 2015-02-27 17:13
New changeset 38c503c2c066 by Brett Cannon in branch 'default':
Issue #22834: Drop a redundant comment and use errno instead of an
https://hg.python.org/cpython/rev/38c503c2c066
msg236796 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-02-27 17:14
Thanks for catches the mistakes, guys!
History
Date User Action Args
2015-09-23 04:42:08Christopher Mengsetnosy: + Christopher Meng
2015-02-27 17:14:11brett.cannonsetstatus: open -> closed

messages: + msg236796
2015-02-27 17:13:52python-devsetmessages: + msg236795
2015-02-21 18:04:29brett.cannonsetstatus: closed -> open
2015-02-20 22:10:33martin.pantersetmessages: + msg236336
2015-02-20 15:32:07serhiy.storchakasetmessages: + msg236294
2015-02-20 15:26:41brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg236292

stage: resolved
2015-02-20 14:48:41python-devsetmessages: + msg236289
2015-02-17 03:23:11martin.pantersetmessages: + msg236122
2015-02-16 06:52:13serhiy.storchakasetstatus: closed -> open

nosy: + serhiy.storchaka
messages: + msg236092

resolution: fixed -> (no value)
stage: resolved -> (no value)
2014-11-22 17:04:36python-devsetmessages: + msg231521
2014-11-21 17:20:30brett.cannonsetstatus: open -> closed
versions: - Python 3.4
messages: + msg231480

resolution: fixed
stage: resolved
2014-11-21 17:19:41python-devsetnosy: + python-dev
messages: + msg231479
2014-11-21 16:12:24brett.cannonsetmessages: + msg231477
versions: + Python 3.5
2014-11-14 22:13:09martin.pantersetmessages: + msg231192
2014-11-14 19:33:41terry.reedysetnosy: + terry.reedy
messages: + msg231173
2014-11-14 19:18:16r.david.murraysetstatus: pending -> open

messages: + msg231170
2014-11-14 14:38:48brett.cannonsetstatus: open -> pending

messages: + msg231161
2014-11-14 02:02:53Arfreversetnosy: + Arfrever
2014-11-11 15:25:52brett.cannonsetassignee: brett.cannon
2014-11-10 15:02:48r.david.murraysetnosy: + eric.snow, r.david.murray, brett.cannon
messages: + msg230963
2014-11-10 03:56:59martin.pantercreate