classification
Title: Python startup should not require passwd entry
Type: behavior Stage: patch review
Components: None Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: merwok Nosy List: Chi Hsuan Yen, Denis.Barmenkov, Jonathon Reinhart, Melissa Chang, Nicholas Brown, aikinci, arekm, bbi5291, christian.heimes, cinerar, georg.brandl, grahamd, haypo, loewis, matrixise, merwok, nadeem.vawda, ned.deily, neologix, r.david.murray, serhiy.storchaka, surajd, tarek, xuanji, zaytsev
Priority: normal Keywords: needs review, patch

Created on 2010-11-22 00:59 by bbi5291, last changed 2017-10-11 12:33 by haypo.

Files
File name Uploaded Description Edit
bug.c bbi5291, 2010-11-22 00:59
nonexistent_user.patch haypo, 2011-05-05 12:49 review
user.py haypo, 2017-10-11 12:33
Pull Requests
URL Status Linked Edit
PR 3403 open cinerar, 2017-09-16 03:19
Messages (33)
msg122086 - (view) Author: Brian Bi (bbi5291) Date: 2010-11-22 03:57
This bug is Linux-specific.

When Python cannot find the home directory of the user invoking it, it prints "'import site' failed; use -v for traceback".

This occurs only when both of these conditions are met:
1. /etc/passwd contains no entry for the current real UID
2. the HOME environment variable does not exist

To duplicate this bug, compile and run the attached program as root.
(This program assumes that 12345 is not a legitimate user on your system, and that Python is at /usr/bin/python ; if this is not the case then edit it accordingly.)
msg122272 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-11-24 11:06
The problem is reproducible on a current Debian Linux system although with different results for current versions of Python (only security issues are accepted against 2.6).  Unlike with 2.6, 3.1 reports no error.  2.7 and 3.2 both fail with an exception:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site.py", line 548, in <module>
    main()
  File "/usr/lib/python2.7/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/lib/python2.7/site.py", line 257, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/lib/python2.7/site.py", line 232, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/lib/python2.7/site.py", line 222, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/lib/python2.7/sysconfig.py", line 541, in get_config_var
    return get_config_vars().get(name)
  File "/usr/lib/python2.7/sysconfig.py", line 449, in get_config_vars
    _CONFIG_VARS['userbase'] = _getuserbase()
  File "/usr/lib/python2.7/sysconfig.py", line 198, in _getuserbase
    return env_base if env_base else joinuser("~", ".local")
  File "/usr/lib/python2.7/sysconfig.py", line 185, in joinuser
    return os.path.expanduser(os.path.join(*args))
  File "/usr/lib/python2.7/posixpath.py", line 256, in expanduser
    userhome = pwd.getpwuid(os.getuid()).pw_dir
KeyError: 'getpwuid(): uid not found: 12345'
msg122640 - (view) Author: Xuanji Li (xuanji) Date: 2010-11-28 08:54
I tried running bug.c using the svn head of python and got this:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding
Aborted
msg132771 - (view) Author: Denis Barmenkov (Denis.Barmenkov) Date: 2011-04-01 22:28
I saw similar error on Python 2.6 installed on freeBSD.
I had test SVN server and wrote pre-commit hook using python. When remote developer commited his changes to repository, hook called os.path.expanduser and exception was raised:
        #  File "/usr/local/lib/python2.6/posixpath.py", line 259, in expanduser
        #    userhome = pwd.getpwuid(os.getuid()).pw_dir
        #KeyError: 'getpwuid(): uid not found: 12345'

So its not a just-Linux issue.
msg133626 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-04-12 23:57
This issue remembers me the issue #6612 (failure if the current directory was removed): the fix was to ignore os.getcwd().

Attached patch ignores os.path.expanduser() error (KeyError) and keeps ~ in the path. Example without HOME var and with an non existent user (uid 12345):

----------------------
$ env -i ./python
>>> import sysconfig
>>> sysconfig.get_config_var('userbase')
'~/.local'

>>> sysconfig.get_paths(scheme='posix_user', expand=False)
{'platstdlib': '{userbase}/lib/python{py_version_short}', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'stdlib': '{userbase}/lib/python{py_version_short}', 'scripts': '{userbase}/bin', 'include': '{userbase}/include/python{py_version_short}', 'data': '{userbase}'}

>>> sysconfig.get_paths(scheme='posix_user')              
{'platstdlib': '~/.local/lib/python3.3', 'platlib': '~/.local/lib/python3.3/site-packages', 'purelib': '~/.local/lib/python3.3/site-packages', 'stdlib': '~/.local/lib/python3.3', 'scripts': '~/.local/bin', 'include': '~/.local/include/python3.3', 'data': '~/.local'}
----------------------

Example with an existant user but without HOME var:
----------------------
marge$ env -i ./python
>>> import sysconfig
>>> sysconfig.get_config_var('userbase')
'/home/haypo/.local'
>>> sysconfig.get_paths(scheme='posix_user')      
{'platstdlib': '/home/haypo/.local/lib/python3.3', 'platlib': '/home/haypo/.local/lib/python3.3/site-packages', 'purelib': '/home/haypo/.local/lib/python3.3/site-packages', 'stdlib': '/home/haypo/.local/lib/python3.3', 'scripts': '/home/haypo/.local/bin', 'include': '/home/haypo/.local/include/python3.3', 'data': '/home/haypo/.local'}
----------------------
msg133674 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-04-13 15:43
Can someone explain how it can happen that a user has no home directory?
msg133675 - (view) Author: Brian Bi (bbi5291) Date: 2011-04-13 15:50
I discovered this while I was implementing and testing a sandbox for automatic evaluation of programs.
msg133676 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-04-13 16:21
Can you be more precise?  IOW, why is this a Python bug rather than a system misconfiguration?  Note that I don’t know a lot about POSIX, so I’m open to change my mind.
msg133742 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-04-14 15:08
I'm not sure whether POSIX warrants anything about this behavior, but nothing prevents a process from running with a UID not listed in /etc/passwd (or NIS, whatever). For example, sudo allows running a command with a UID not listed in the password database, see http://linux.die.net/man/5/sudoers :
"""
targetpw

If set, sudo will prompt for the password of the user specified by the -u flag (defaults to root) instead of the password of the invoking user. Note that this precludes the use of a uid not listed in the passwd database as an argument to the -u flag. This flag is off by default.
"""

UIDs not backed by users are useful for example if you're working with a sandbox, or virtual users such as in some FTP servers http://www.proftpd.org/docs/howto/VirtualUsers.html :
"""
Question: What makes a user "virtual", then?
Answer: A virtual user is, quite simply, a user that is not defined in the system /etc/passwd file. This file associates a user name, given by the system administrator, to a user ID (commonly shortened to UID) and a group ID (GID), among other details. The Unix kernel does not deal with users in terms of their user names; it only "knows" about UIDs and GIDs. This means that an application like proftpd can look up the IDs to use for a given user name however it sees fit. Using /etc/passwd is not strictly required.
"""
msg133743 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-04-14 15:32
Because the patch is simple (just add a try/except), I think that it doesn't matter if only few people use users without entry in /etc/passwd and we should fix this issue.
msg133830 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-04-15 14:58
It’s not just a try/except, it’s a behavior change: after the patch, paths returned by sysconfig may not be fully expanded paths.  I would like Tarek to make a call on this.
msg134984 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-02 15:30
> I would like Tarek to make a call on this.

So Tarek, what do you think?
msg134987 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-02 15:43
As discussed w/ Victor, a process should be able to run Python even if its user does not have a home.

So the call to _getuserbase() should be protected. 

But then we have to control that all the code that uses  CONFIG_VARS['userbase'] is protected when the value is not set.

I am thinking about per-user installation and such things: we need to make sure everything is checking this.
msg135044 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-05-03 15:54
This is the one thing about which I wanted a call: “after the patch, paths returned by sysconfig may not be fully expanded paths” (i.e. they may start with '~').
msg135047 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-03 16:08
Paths that are starting with ~ should be extended with the right value with the user base. If the user base cannot be calculated, paths starting with ~ should not exist or be used at all in this context.

Maybe we need to completely reset them to None like userbase. We need to list all use cases within the stdlib and come up with a general rule.
msg135195 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-05 12:49
> If the user base cannot be calculated, paths
> starting with ~ should not exist or be used at all in this context.

It's not "~" but "{userbase}" substitution variable.

Here is a new patch implementing this idea: don't create the variables using the user directory if the user doesn't exist (if we cannot get the user directory). I didn't test distutils, I just try to start Python (it does work using bug.c).
msg135551 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-08 23:11
@eric.araujo, @tarek: do you prefer nonexistent_user.patch? I removed sysconfig_getuserbase.patch, because I agree that an expanded path containing "~" is a bug.
msg135592 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-05-09 14:40
In the absence of tests or doc update, can you tell in English what the new behavior is?  IIUC, when the home dir is not found, all the variables that depend on it would not exist, right?  Or would they be set to None?
msg135596 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-09 15:10
Main changes of the patch, if the current user has no home directory (no entry in /etc/passwd) and there is HOME environment variable:

 - sysconfig.get_config_vars() doesn't have a 'userbase' variable. sysconfig.get_config_var('userbase') returns None as any other nonexistent key.
 - sysconfig.get_paths() doesn't create a path if expand fails (without raising an error or emiting a warning). For example, sysconfig.get_paths(scheme='posix_user') returns an empty dict.
msg155530 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2012-03-13 00:06
Adding people from the duplicate report #14238.

Georg said:
> there should be a guard here that just doesn't add user site directories if the lookup fails.

I concurred and added:
> we also need to fix sysconfig; I think the right thing to do would be not to have a config var
> named 'userbase' when run from a UID without passwd entry.  This implies that the functions
> listed under http://docs.python.org/library/sysconfig#installation-paths would also need to omit
> *user schemes.
msg155534 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2012-03-13 00:12
I reviewed haypo’s patch.  I’d be happy to update it to address my remarks if you want me to.
msg156373 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-03-20 00:34
> I’d be happy to update it to address my remarks if you want me to.

Please write your own patch addressing your concerns.
msg193987 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-07-31 08:14
Is there a hope to get this fixed?
It can be needed to e.g. run python as nobody user (used for example for CGI scripts, etc).
msg257513 - (view) Author: Jonathon Reinhart (Jonathon Reinhart) Date: 2016-01-05 03:10
I have another scenario where this happens: Running Python in a Docker container, passing the --user option to 'docker run'. This sets the UID in the container, which has its own /etc/passwd. Couple this with the fact that $HOME might not be set (e.g. when Python is invoked from SCons which deliberately clears the environment for sub-processes), and *boom*, Python is non-functional.

Regardless of where the fix is done, the ideal behavior is straightforward: if Python can't determine the home directory, don't try to add user site packages.

This bug is now over 5 years old, and people have identified real-life various scenarios where this bug manifests itself, and submitted patches. Could we please address this?
msg270771 - (view) Author: Stéphane Wirtel (matrixise) * Date: 2016-07-18 16:10
Hi @victor I have compiled the bug.c file with my fedora 24.
there is python 2.7.11 and 3.5.1 and I don't have the bug with these versions.

What can we do with this issue ?

Thanks,

Stephane
msg272115 - (view) Author: Yury V. Zaytsev (zaytsev) Date: 2016-08-07 09:36
Another scenario how this could happen is when Python interpreter is run from a scheduled job on a BlueGene/Q node by the CNK; this is a minimalist execution environment where getpwuid() is unable to resolve home directories for UIDs running the jobs.
msg282502 - (view) Author: Suraj Deshmukh (surajd) Date: 2016-12-06 05:47
I did not encounter this issue in fedora24 based container on docker but when I run it in openshift I see errors as:

That container has `Python 2.7.12` installed. While `flask` and `redis` python libs are installed via pip.

```
bash: /usr/local/bin/wait-for-it.sh: Permission denied
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
Traceback (most recent call last):
  File "app.py", line 18, in <module>
    app.run(host="0.0.0.0", port=5000, debug=True)
  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 843, in run
    run_simple(host, port, self, **options)
  File "/usr/lib/python2.7/site-packages/werkzeug/serving.py", line 633, in run_simple
    application = DebuggedApplication(application, use_evalex)
  File "/usr/lib/python2.7/site-packages/werkzeug/debug/__init__.py", line 254, in __init__
    if self.pin is None:
  File "/usr/lib/python2.7/site-packages/werkzeug/debug/__init__.py", line 264, in _get_pin
    self._pin, self._pin_cookie = get_pin_and_cookie_name(self.app)
  File "/usr/lib/python2.7/site-packages/werkzeug/debug/__init__.py", line 144, in get_pin_and_cookie_name
    username = getpass.getuser()
  File "/usr/lib64/python2.7/getpass.py", line 158, in getuser
    return pwd.getpwuid(os.getuid())[0]
KeyError: 'getpwuid(): uid not found: 1000050000'
```
msg282503 - (view) Author: Graham Dumpleton (grahamd) Date: 2016-12-06 05:54
@surajd Why aren't you using the Python S2I builders for OpenShift?

When you run anything under OpenShift, it will assign your project a UID which any containers are forced to run under. The OpenShift containers include a mechanism using a package call nss_wrapper so that correct passwd entries are reported for the assigned UID. If your own image doesn't have a similar provision, and in general aren't designed to run as an arbitrary assigned UID, you will encounter various issues.

Read through the latter section of posts in:

* http://blog.dscpl.com.au/p/using-python-with-docker.html

for further details.

Anyway, this isn't the place to discuss OpenShift, use instead:

* https://groups.google.com/forum/#!forum/openshift

if you have questions about running Python under OpenShift.
msg282504 - (view) Author: Suraj Deshmukh (surajd) Date: 2016-12-06 06:01
@grahamd thanks this helps.
msg302176 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-09-14 15:35
Unless I'm mistaken, this has come up again in issue 31469.
msg302355 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-09-16 21:56
Dmitriy: you will note from the discussion on this issue that your "simple patch" was not considered sufficient.  There were additional concerns voiced about haypo's patch, which is why I guess it didn't get applied.  However, can you review that and confirm that it solves your problem?  Maybe that will encourage a reconsideration of this issue.
msg303943 - (view) Author: Nicholas Brown (Nicholas Brown) Date: 2017-10-09 12:11
This is also a problem when using the DynamicUser=yes feature available in systemd 232 onward with a service that's implemented in python.
See http://0pointer.net/blog/dynamic-users-with-systemd.html for details of the DynamicUser= systemd feature.
msg304128 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-10-11 12:33
user.py: short Python script to reproduce the bug. It runs Python as user 12345, group 12345 and unset the HOME environment variable to reproduce the bug.

The python binary must be readable and executable by any user, especially uid 12345 gid 12345. On my Fedora, /home/haypo is only accessible by the usr haypo. I cloned Python in /opt/cpython to work around the issue.

I confirm that Python 3.7 still has the bug.
History
Date User Action Args
2017-10-11 12:33:02hayposetfiles: + user.py

messages: + msg304128
2017-10-09 12:11:17Nicholas Brownsetnosy: + Nicholas Brown
messages: + msg303943
2017-09-18 14:33:57Chi Hsuan Yensetnosy: + Chi Hsuan Yen
2017-09-16 21:56:25r.david.murraysetmessages: + msg302355
2017-09-16 03:19:32cinerarsetpull_requests: + pull_request3602
2017-09-14 15:35:21r.david.murraysetnosy: + cinerar, r.david.murray
messages: + msg302176
2017-09-14 15:34:28r.david.murraylinkissue31469 superseder
2016-12-06 06:01:05surajdsetmessages: + msg282504
2016-12-06 05:54:26grahamdsetmessages: + msg282503
2016-12-06 05:47:53surajdsetnosy: + surajd
messages: + msg282502
2016-08-07 09:37:55matrixisesetversions: + Python 3.6, - Python 3.4
2016-08-07 09:36:38zaytsevsetmessages: + msg272115
2016-07-18 16:10:48matrixisesetnosy: + matrixise
messages: + msg270771
2016-05-16 23:25:08Melissa Changsetnosy: + Melissa Chang
2016-01-05 03:10:55Jonathon Reinhartsetnosy: + Jonathon Reinhart
messages: + msg257513
2014-10-12 00:30:49grahamdsetnosy: + grahamd
2014-09-28 01:43:30terry.reedysetstage: patch review
versions: + Python 3.5, - Python 3.2, Python 3.3
2014-01-07 17:11:26zaytsevsetnosy: + zaytsev
2013-08-02 07:08:28neologixsetkeywords: + needs review
2013-08-02 06:56:00neologixlinkissue16945 dependencies
2013-07-31 19:43:59christian.heimessetnosy: + christian.heimes
2013-07-31 15:32:00serhiy.storchakasetnosy: + serhiy.storchaka
2013-07-31 08:14:12neologixsetmessages: + msg193987
2012-11-18 19:58:05ezio.melottisetversions: + Python 3.4
2012-03-20 00:34:21hayposetmessages: + msg156373
2012-03-13 00:12:28merwoksetmessages: + msg155534
2012-03-13 00:06:15merwoksettitle: "import site failed" when Python can't find home directory (sysconfig._getuserbase) -> Python startup should not require passwd entry
nosy: + loewis, georg.brandl, arekm, nadeem.vawda, aikinci

messages: + msg155530

assignee: tarek -> merwok
versions: - Python 3.1
2012-03-13 00:04:10merwoklinkissue14238 superseder
2011-05-09 15:10:48hayposetmessages: + msg135596
2011-05-09 14:40:02merwoksetmessages: + msg135592
2011-05-08 23:11:34hayposetmessages: + msg135551
2011-05-08 23:10:19hayposetfiles: - sysconfig_getuserbase.patch
2011-05-05 12:49:46hayposetfiles: + nonexistent_user.patch

messages: + msg135195
2011-05-03 16:08:10tareksetmessages: + msg135047
2011-05-03 15:54:56merwoksetmessages: + msg135044
2011-05-02 15:43:19tareksetmessages: + msg134987
2011-05-02 15:30:15hayposetmessages: + msg134984
2011-04-15 14:58:52merwoksetassignee: tarek
messages: + msg133830
2011-04-14 15:32:31hayposetmessages: + msg133743
2011-04-14 15:08:30neologixsetnosy: + neologix
messages: + msg133742
2011-04-13 16:21:37merwoksetstage: needs patch -> (no value)
messages: + msg133676
versions: + Python 3.3
2011-04-13 15:50:02bbi5291setmessages: + msg133675
2011-04-13 15:43:34merwoksetmessages: + msg133674
2011-04-12 23:59:49hayposetfiles: + sysconfig_getuserbase.patch
2011-04-12 23:59:41hayposetfiles: - sysconfig_getuserbase.patch
2011-04-12 23:58:59hayposetfiles: + sysconfig_getuserbase.patch
keywords: + patch
2011-04-12 23:57:32hayposetnosy: + tarek

messages: + msg133626
title: "import site failed" when Python can't find home directory -> "import site failed" when Python can't find home directory (sysconfig._getuserbase)
2011-04-03 19:35:02merwoksetnosy: + haypo, merwok
2011-04-01 22:28:46Denis.Barmenkovsetnosy: + Denis.Barmenkov
messages: + msg132771
2010-11-28 08:54:22xuanjisetnosy: + xuanji
messages: + msg122640
2010-11-24 11:07:05ned.deilysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2010-11-24 11:06:28ned.deilysetnosy: + ned.deily

messages: + msg122272
stage: needs patch
2010-11-24 11:05:32ned.deilysetmessages: - msg122051
2010-11-22 03:57:59bbi5291setmessages: + msg122086
2010-11-22 00:59:48bbi5291create