New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python startup should not require passwd entry #54705
Comments
This bug is Linux-specific. When Python cannot find the home directory of the user invoking it, it prints "'import site' failed; use -v for traceback". This occurs only when both of these conditions are met:
To duplicate this bug, compile and run the attached program as root. |
The problem is reproducible on a current Debian Linux system although with different results for current versions of Python (only security issues are accepted against 2.6). Unlike with 2.6, 3.1 reports no error. 2.7 and 3.2 both fail with an exception: Traceback (most recent call last):
File "/usr/lib/python2.7/site.py", line 548, in <module>
main()
File "/usr/lib/python2.7/site.py", line 530, in main
known_paths = addusersitepackages(known_paths)
File "/usr/lib/python2.7/site.py", line 257, in addusersitepackages
user_site = getusersitepackages()
File "/usr/lib/python2.7/site.py", line 232, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/usr/lib/python2.7/site.py", line 222, in getuserbase
USER_BASE = get_config_var('userbase')
File "/usr/lib/python2.7/sysconfig.py", line 541, in get_config_var
return get_config_vars().get(name)
File "/usr/lib/python2.7/sysconfig.py", line 449, in get_config_vars
_CONFIG_VARS['userbase'] = _getuserbase()
File "/usr/lib/python2.7/sysconfig.py", line 198, in _getuserbase
return env_base if env_base else joinuser("~", ".local")
File "/usr/lib/python2.7/sysconfig.py", line 185, in joinuser
return os.path.expanduser(os.path.join(*args))
File "/usr/lib/python2.7/posixpath.py", line 256, in expanduser
userhome = pwd.getpwuid(os.getuid()).pw_dir
KeyError: 'getpwuid(): uid not found: 12345' |
I tried running bug.c using the svn head of python and got this: Could not find platform independent libraries <prefix> |
I saw similar error on Python 2.6 installed on freeBSD. So its not a just-Linux issue. |
This issue remembers me the issue bpo-6612 (failure if the current directory was removed): the fix was to ignore os.getcwd(). Attached patch ignores os.path.expanduser() error (KeyError) and keeps ~ in the path. Example without HOME var and with an non existent user (uid 12345): ---------------------- $ env -i ./python
>>> import sysconfig
>>> sysconfig.get_config_var('userbase')
'~/.local' >>> sysconfig.get_paths(scheme='posix_user', expand=False)
{'platstdlib': '{userbase}/lib/python{py_version_short}', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'stdlib': '{userbase}/lib/python{py_version_short}', 'scripts': '{userbase}/bin', 'include': '{userbase}/include/python{py_version_short}', 'data': '{userbase}'}
>>> sysconfig.get_paths(scheme='posix_user')
{'platstdlib': '~/.local/lib/python3.3', 'platlib': '~/.local/lib/python3.3/site-packages', 'purelib': '~/.local/lib/python3.3/site-packages', 'stdlib': '~/.local/lib/python3.3', 'scripts': '~/.local/bin', 'include': '~/.local/include/python3.3', 'data': '~/.local'} Example with an existant user but without HOME var: marge$ env -i ./python
>>> import sysconfig
>>> sysconfig.get_config_var('userbase')
'/home/haypo/.local'
>>> sysconfig.get_paths(scheme='posix_user')
{'platstdlib': '/home/haypo/.local/lib/python3.3', 'platlib': '/home/haypo/.local/lib/python3.3/site-packages', 'purelib': '/home/haypo/.local/lib/python3.3/site-packages', 'stdlib': '/home/haypo/.local/lib/python3.3', 'scripts': '/home/haypo/.local/bin', 'include': '/home/haypo/.local/include/python3.3', 'data': '/home/haypo/.local'} |
Can someone explain how it can happen that a user has no home directory? |
I discovered this while I was implementing and testing a sandbox for automatic evaluation of programs. |
Can you be more precise? IOW, why is this a Python bug rather than a system misconfiguration? Note that I don’t know a lot about POSIX, so I’m open to change my mind. |
I'm not sure whether POSIX warrants anything about this behavior, but nothing prevents a process from running with a UID not listed in /etc/passwd (or NIS, whatever). For example, sudo allows running a command with a UID not listed in the password database, see http://linux.die.net/man/5/sudoers : If set, sudo will prompt for the password of the user specified by the -u flag (defaults to root) instead of the password of the invoking user. Note that this precludes the use of a uid not listed in the passwd database as an argument to the -u flag. This flag is off by default. UIDs not backed by users are useful for example if you're working with a sandbox, or virtual users such as in some FTP servers http://www.proftpd.org/docs/howto/VirtualUsers.html : |
Because the patch is simple (just add a try/except), I think that it doesn't matter if only few people use users without entry in /etc/passwd and we should fix this issue. |
It’s not just a try/except, it’s a behavior change: after the patch, paths returned by sysconfig may not be fully expanded paths. I would like Tarek to make a call on this. |
So Tarek, what do you think? |
As discussed w/ Victor, a process should be able to run Python even if its user does not have a home. So the call to _getuserbase() should be protected. But then we have to control that all the code that uses CONFIG_VARS['userbase'] is protected when the value is not set. I am thinking about per-user installation and such things: we need to make sure everything is checking this. |
This is the one thing about which I wanted a call: “after the patch, paths returned by sysconfig may not be fully expanded paths” (i.e. they may start with '~'). |
Paths that are starting with ~ should be extended with the right value with the user base. If the user base cannot be calculated, paths starting with ~ should not exist or be used at all in this context. Maybe we need to completely reset them to None like userbase. We need to list all use cases within the stdlib and come up with a general rule. |
It's not "~" but "{userbase}" substitution variable. Here is a new patch implementing this idea: don't create the variables using the user directory if the user doesn't exist (if we cannot get the user directory). I didn't test distutils, I just try to start Python (it does work using bug.c). |
@eric.araujo, @tarek: do you prefer nonexistent_user.patch? I removed sysconfig_getuserbase.patch, because I agree that an expanded path containing "~" is a bug. |
In the absence of tests or doc update, can you tell in English what the new behavior is? IIUC, when the home dir is not found, all the variables that depend on it would not exist, right? Or would they be set to None? |
Main changes of the patch, if the current user has no home directory (no entry in /etc/passwd) and there is HOME environment variable:
|
Adding people from the duplicate report bpo-14238. Georg said:
I concurred and added:
|
I reviewed haypo’s patch. I’d be happy to update it to address my remarks if you want me to. |
Please write your own patch addressing your concerns. |
Is there a hope to get this fixed? |
I have another scenario where this happens: Running Python in a Docker container, passing the --user option to 'docker run'. This sets the UID in the container, which has its own /etc/passwd. Couple this with the fact that $HOME might not be set (e.g. when Python is invoked from SCons which deliberately clears the environment for sub-processes), and *boom*, Python is non-functional. Regardless of where the fix is done, the ideal behavior is straightforward: if Python can't determine the home directory, don't try to add user site packages. This bug is now over 5 years old, and people have identified real-life various scenarios where this bug manifests itself, and submitted patches. Could we please address this? |
I did not encounter this issue in fedora24 based container on docker but when I run it in openshift I see errors as: That container has
|
@surajd Why aren't you using the Python S2I builders for OpenShift? When you run anything under OpenShift, it will assign your project a UID which any containers are forced to run under. The OpenShift containers include a mechanism using a package call nss_wrapper so that correct passwd entries are reported for the assigned UID. If your own image doesn't have a similar provision, and in general aren't designed to run as an arbitrary assigned UID, you will encounter various issues. Read through the latter section of posts in: for further details. Anyway, this isn't the place to discuss OpenShift, use instead: if you have questions about running Python under OpenShift. |
@Grahamd thanks this helps. |
Unless I'm mistaken, this has come up again in bpo-31469. |
Dmitriy: you will note from the discussion on this issue that your "simple patch" was not considered sufficient. There were additional concerns voiced about haypo's patch, which is why I guess it didn't get applied. However, can you review that and confirm that it solves your problem? Maybe that will encourage a reconsideration of this issue. |
This is also a problem when using the DynamicUser=yes feature available in systemd 232 onward with a service that's implemented in python. |
user.py: short Python script to reproduce the bug. It runs Python as user 12345, group 12345 and unset the HOME environment variable to reproduce the bug. The python binary must be readable and executable by any user, especially uid 12345 gid 12345. On my Fedora, /home/haypo is only accessible by the usr haypo. I cloned Python in /opt/cpython to work around the issue. I confirm that Python 3.7 still has the bug. |
According to seirl on IRC, this issue is trigged when a Python application is run by systemd using DynamicUser=yes: "DynamicUser: Takes a boolean parameter. If set, a UNIX user and group pair is allocated dynamically when the unit is started, and released as soon as it is stopped. The user and group will not be added to /etc/passwd or /etc/group, but are managed transiently during runtime. (...)" |
Trivial way to reproduce, run this as root:
|
With my PR 10919, "python3 setup.py install" and "python3 setup.py install --user" still fail with: Traceback (most recent call last):
File "setup.py", line 79, in <module>
main()
File "setup.py", line 75, in main
setup(**options)
File "/tmp/cpython/Lib/distutils/core.py", line 121, in setup
dist.parse_config_files()
File "/tmp/cpython/Lib/distutils/dist.py", line 397, in parse_config_files
filenames = self.find_config_files()
File "/tmp/cpython/Lib/distutils/dist.py", line 349, in find_config_files
check_environ()
File "/tmp/cpython/Lib/distutils/util.py", line 161, in check_environ
os.environ['HOME'] = pwd.getpwuid(os.getuid())[5]
KeyError: 'getpwuid(): uid not found: 12345' I suggest to open a new issue if you want to enhance the error message and/or handle getpwuid() failure in find_config_files(). I prefer to stick to the initial bug report which hasn't been fixed in 8 years:
IMHO PR 10919 fix is straighforward, it respects the contract (documentation) of posixpath.expanduser() ("If user or $HOME is unknown, do nothing."), and expanduser() already handles KeyError on getpwnam() (since the function has been created in 1992 by Guido van Rossum! commit 7ac4878). |
Would it make sense to backport this fix in 3.6 and 3.7? As distros increasingly move in the direction of using DynamicUser=yes for most stateless services, it would really help to have that, for instance in Debian Buster (which will probably be on 3.7 if my understanding is correct). FYI the cutoff date for the release candidate of 3.6.8 is 2018-12-07 (in two days). |
I'm interested in fixing this bug too since it bit me in SCons and I had to use a local patch for it. I welcome the upstream fix and don't object to PR 10919. I'm not personally affected by unexpanded paths in sysconfig (at least not in a way I know of), so I just want to make sure that all effects of the PR are considered, given the previous discussions. Thanks, Victor! |
Yes, I plan to try to backport the fix to stable branches. |
I'd like to see it there, given that this bug surfaced in many use cases not involving any modern features or systemd at all. |
A few minor tests are still failing when the uid doesn't exist in the password database, but the main issue in pwdhas been fixed and distutils should work more or lesss. I close this 8 years old issue. Thanks to everyone who has been involved in helping to fix it! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: