classification
Title: importing yields unexpected results when initial script is a symbolic link
Type: enhancement Stage: needs patch
Components: Documentation Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: Arfrever, brett.cannon, eric.snow, jamadagni, jbeulich, matejcik, ncoghlan, python-dev, r.david.murray, senko
Priority: normal Keywords: patch

Created on 2009-06-30 10:05 by jbeulich, last changed 2014-02-06 14:24 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
tutorial-symlink-syspath-note.diff senko, 2013-07-06 16:04 review
Messages (13)
msg89914 - (view) Author: (jbeulich) Date: 2009-06-30 10:05
Due to the way PySys_SetArgv() works, when the initial script is a
symbolic link, importing from normal files in the same directory does
not work. This is particularly surprising when the work tree is a
symlinked clone (cp -s) of an original (i.e. snapshot) tree with a few
modifications (patches) applied to one or more of the modules imported
from: Due the the erratum, the modifications made will appear to not
take effect until one realizes that the wrong module is being imported from.

The solution would in my opinion be to not only add the path left after
the readlink()/realpath() processing to the import search path list, but
also any intermediately encountered ones (in the order processed) as
long as the leaf component continues to be a symlink.

(Note: It seems pointless to use readlink() in the current [3.1 and
earlier] implementation, since the result gets passed to realpath()
anyway. Also, the documentation doesn't seem to mention this behavior,
and from the sources it remains unclear why this processing is needed at
all.)
msg187280 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2013-04-18 18:48
It will be difficult to take this forward owing to the problem description in msg89914 being limited to "importing from normal files in the same directory does not work".  Plus any problems back then may well have been fixed due to the reworking of the import mechanism by somebody who apparently has spent 10 years as a Python core developer.  Congratulations :)
msg187281 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-04-18 19:23
In case someone wants to reproduce:

  mkdir pkg
  echo "import tester" > pkg/symlinked.py
  ln -s pkg/symlinked.py linked.py
  echo "print('HIT')" > tester.py

That fails because Python assumes you are in the pkg directory, not the directory you started execution. This makes sense to me. If you used a hard link then this isn't a problem. Python treats a symlink as a redirect, which means it works where the redirect tells it to and doesn't try to confuse things by considering 2 different locations to be the cwd for imports.

Closing as "won't fix" since I think it would be more confusing to support both a symlink directory and the cwd.
msg190145 - (view) Author: Shriramana Sharma (jamadagni) Date: 2013-05-27 16:05
I'm sorry but I don't get why this is a WONTFIX. I reported what is (now) apparently a dup: issue 18067. Just like the OP of this bug, I feel that in doing testing and such, one would naturally symlink and expect the library in the *current* directory to be imported. 

And about the CWD, I have demonstrated in issue 18067 how the CWD is in fact reported to be the directory of the *source* of the symlink (i.e. the dir containing the symlink inode) and not the *target* of the symlink. This is precisely what is frustrating about this bug: the fact that Python does not import something from a directory which it reports to be the current directory as per os.getcwd(). 

While I myself lack the internal CPython code knowledge to fix this, I can't imagine this would be too difficult to fix, given that os.getcwd() already reports the correct current directory -- in setting up the import path list, you just have to use that i.o. whatever else you are using now.

Thanks.
msg190146 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-05-27 16:50
When a script is executed by python, it does *not* import from the CWD, it imports from the *location of the script*.  From this, then, you can see that there are two possible interpretations of "the location of the script" when the script is a symlink, and we have chosen the more secure (and practical) one: the location of the real script file.
msg190158 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-27 22:44
The current behaviour is also needed to sanely support Python scripts
symlinked from Linux /bin directories.
msg190167 - (view) Author: Shriramana Sharma (jamadagni) Date: 2013-05-28 01:56
> The current behaviour is also needed to sanely support Python 
> scripts symlinked from Linux /bin directories.

OK that clinched it for me -- I can't argue against that! And obviously it is not meaningful to copy/symlink *all* the current-directory modules a particular script depends upon to the symlink directory as well. And searching both directories (containing the source and target of the symlink) is not good for security I guess.

And I also checked the contents of sys.path with my test case -- and sure enough the directory corresponding to the actual output was printed. Just that sys.path is different from os.getcwd() needs some effort to bring into mind.

So I think someone should please clearly mention this behaviour in the documentation under http://docs.python.org/3/tutorial/modules.html#the-module-search-path and the Py2 equivalent. Specifically this point needs clarification:

""the directory containing the input script (or the current directory).""

It would be best to remove the text in parantheses (which immediately makes one think of os.getcwd()) and add a clarification like:

Note that on filesystems that support symlinks, this means the directory containing the actual script file. Symlinks to the script may be present elsewhere and may be used to invoke the script, but the directories containing those symlinks will *not* be searched for dependency modules.

Thank you very much for these clarifications and for your work on Python! Please do add the above documentation clarification, though.
msg190171 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-28 02:17
That's fair - reopening this as a docs bug.
msg192456 - (view) Author: Senko Rasic (senko) * Date: 2013-07-06 16:04
Patch for modifying the modules part of tutorial with the changes suggested by jamadagni (reworded slightly so the note is outside the itemized list).
msg192476 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-07-06 17:42
Senko, could you sign the contributor agreement (http://python.org/psf/contrib/contrib-form/) so we can use your patch?
msg192514 - (view) Author: Senko Rasic (senko) * Date: 2013-07-07 06:53
Yep, signed.
msg210384 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-06 14:23
New changeset 47c31e7d3779 by Brett Cannon in branch 'default':
Issue #6386: When executing a script that's a symlink, the directory
http://hg.python.org/cpython/rev/47c31e7d3779
msg210385 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-02-06 14:24
Thanks for the path, Senko! Tweaked the wording a bit as the "current directory" part was accurate for when you're in the REPL.
History
Date User Action Args
2014-02-06 14:24:12brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg210385
2014-02-06 14:23:03python-devsetnosy: + python-dev
messages: + msg210384
2014-02-03 19:27:27Arfreversetnosy: + Arfrever
2014-02-03 17:45:57brett.cannonsetassignee: brett.cannon
2014-02-03 17:06:08BreamoreBoysetnosy: - BreamoreBoy
2013-07-07 06:53:17senkosetmessages: + msg192514
2013-07-06 17:42:11brett.cannonsetmessages: + msg192476
2013-07-06 16:04:18senkosetfiles: + tutorial-symlink-syspath-note.diff

nosy: + senko
messages: + msg192456

keywords: + patch
2013-05-28 02:17:12ncoghlansetstatus: closed -> open

type: behavior -> enhancement
assignee: brett.cannon -> (no value)
components: + Documentation
versions: - Python 3.2
messages: + msg190171
resolution: wont fix -> (no value)
stage: test needed -> needs patch
2013-05-28 01:56:05jamadagnisetmessages: + msg190167
2013-05-27 22:44:43ncoghlansetmessages: + msg190158
2013-05-27 16:50:07r.david.murraysetnosy: + r.david.murray
messages: + msg190146
2013-05-27 16:05:33jamadagnisetnosy: + jamadagni
messages: + msg190145
2013-05-26 18:05:13r.david.murraylinkissue18067 superseder
2013-04-18 19:23:17brett.cannonsetstatus: open -> closed
assignee: brett.cannon
resolution: wont fix
messages: + msg187281
2013-04-18 18:48:50BreamoreBoysetnosy: + BreamoreBoy
messages: + msg187280
2012-11-13 02:48:22eric.snowsetnosy: + eric.snow
2012-11-09 14:04:15ezio.melottisetnosy: + brett.cannon, ncoghlan

versions: + Python 3.3, Python 3.4, - Python 3.1
2010-07-10 23:38:34BreamoreBoysetstage: test needed
versions: + Python 3.2, - Python 2.6, Python 2.5, Python 2.4, Python 3.0
2009-06-30 10:12:16matejciksetnosy: + matejcik
2009-06-30 10:05:43jbeulichcreate