classification
Title: Undocumented KeyError from os.path.expanduser
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Python startup should not require passwd entry
View: 10496
Assigned To: Nosy List: acdha, docs@python, serhiy.storchaka, terry.reedy, vstinner, zaytsev
Priority: normal Keywords:

Created on 2014-01-07 16:15 by acdha, last changed 2018-12-16 21:59 by vstinner. This issue is now closed.

Messages (7)
msg207552 - (view) Author: Chris Adams (acdha) Date: 2014-01-07 16:15
This is a more general version of #10496: os.path.expanduser is documented as returning the unmodified string if it cannot be expanded (http://docs.python.org/3/library/os.path.html#os.path.expanduser) but there's one edge case where this can fail: when running without HOME in the environment:

https://github.com/python/cpython/blob/3.3/Lib/posixpath.py#L259-L263

In this case, pwd.getpwuid is called and although it's not documented as such, it raises a KeyError anytime the underlying POSIX getpwuid() call fails:

https://github.com/python/cpython/blob/3.3/Modules/pwdmodule.c#L119-L120

Reproducing this deliberately:

cadams@Io:~ $ sudo -H python
Python 2.7.5 (default, Aug 25 2013, 00:04:04) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import posix
>>> posix.setuid(1001)
>>> posix.seteuid(1001)
>>> os.environ.pop('HOME')
'/var/root'
>>> os.path.expanduser("~")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 269, in expanduser
    userhome = pwd.getpwuid(os.getuid()).pw_dir
KeyError: 'getpwuid(): uid not found: 1001'

Context:
* https://github.com/toastdriven/django-haystack/issues/924
* https://github.com/kennethreitz/requests/issues/1846.
msg207555 - (view) Author: Yury V. Zaytsev (zaytsev) Date: 2014-01-07 16:38
Amusingly, it's also the case on BG/Q compute nodes. Only this morning, I cooked up the following patch:

--- a/Lib/posixpath.py
+++ b/Lib/posixpath.py
@@ -237,7 +237,11 @@ def expanduser(path):
     if i == 1:
         if 'HOME' not in os.environ:
             import pwd
-            userhome = pwd.getpwuid(os.getuid()).pw_dir
+            try:
+                userhome = pwd.getpwuid(os.getuid()).pw_dir
+            except KeyError:
+                import warnings
+                warnings.warn('Unable to auto-detect $HOME, export it or functionality such as user-installed modules will not work!', RuntimeWarning)
         else:
             userhome = os.environ['HOME']
     else:

I think it's a bit harsh to fail completely, but returning silently also doesn't sound like a good idea, so I decided to issue a warning instead; the wording should be probably tweaked by a native speaker of English though.

What do you think?
msg207579 - (view) Author: Chris Adams (acdha) Date: 2014-01-07 18:29
Other than hoisting the warnings import to the top (PEP-8) that seems entirely reasonable to me. The user report which we got was confusing because it was non-obvious where it came from - a warning or other pointer would have helped the original reporter get an idea as to what was failing earlier.

The other two changes I'd suggest would be documentation updates:

os.path.expanduser noting that KeyError may be raised when HOME is undefined and pwd.getpwuid fails for the active user

pwd.getpwuid noting that KeyErrors will be raised when the underlying POSIX call fails
msg227746 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-09-28 01:49
I think the doc should be considered correct and posixpath buggy.  The ntpath code checks for the existence of HOME and substitutes before returning the arg unchanged, as documented.

    if 'HOME' in os.environ:
        userhome = os.environ['HOME']
    elif 'USERPROFILE' in os.environ:
        userhome = os.environ['USERPROFILE']
    elif not 'HOMEPATH' in os.environ:
        return path
    <create userhome from HOMEDRIVE and HOMEPATH

--
The pwd doc already says " KeyError is raised if the entry asked for cannot be found.", so this is not an issue.
msg227953 - (view) Author: Chris Adams (acdha) Date: 2014-09-30 17:16
I agree that making the code match the docs would be preferable – an unexpected KeyError might be easier to track down that way but it'd still surprise most developers.

re:pwd docs, the formatting in https://hg.python.org/cpython/file/8e9df3414185/Doc/library/pwd.rst makes it easy to miss the KeyError note – it's at the end of a paragraph after a table and since "entry" is a generic term it's possible to misread it as applying to fields within the record rather than the record itself. I think it would at least be worth adding a newline and perhaps explicitly noting that it applies to any function which returns a password database entry. 

Given that there are only two functions which it applies to, it might be best to simply add a note to each of them for people who use "pydoc pwd.getpwduid" or an IDE and don't see the module-level docs but that's getting a bit contrived.
msg331924 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-16 11:34
The behavior was changed in f2f4555d8287ad217a1dba7bbd93103ad4daf3a8 as a part of issue10496.
msg331938 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-16 21:59
Even if it's not exactly a duplicate of bpo-10496, the reported bug has been fixed. I close the issue.
History
Date User Action Args
2018-12-16 21:59:17vstinnersetstatus: pending -> closed
superseder: Python startup should not require passwd entry
messages: + msg331938

resolution: duplicate
stage: test needed -> resolved
2018-12-16 11:34:46serhiy.storchakasetstatus: open -> pending
nosy: + serhiy.storchaka, vstinner
messages: + msg331924

2014-09-30 17:16:21acdhasetmessages: + msg227953
2014-09-28 01:49:08terry.reedysetnosy: + terry.reedy
messages: + msg227746

assignee: docs@python ->
components: - Documentation
stage: needs patch -> test needed
2014-09-27 14:29:14serhiy.storchakasetversions: + Python 3.5, - Python 3.3
nosy: + docs@python

assignee: docs@python
components: + Documentation
stage: needs patch
2014-01-07 18:29:08acdhasetmessages: + msg207579
2014-01-07 16:38:31zaytsevsetversions: + Python 3.4
nosy: + zaytsev

messages: + msg207555

type: behavior
2014-01-07 16:15:27acdhacreate