classification
Title: Python3 ignores __init__.py that are links to /dev/null
Type: behavior Stage: test needed
Components: Interpreter Core Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, dgreiman, eric.smith, orsenthil, ztane
Priority: normal Keywords:

Created on 2016-10-13 00:18 by dgreiman, last changed 2017-02-28 19:52 by dgreiman. This issue is now closed.

Files
File name Uploaded Description Edit
namespace_package_dev_null.sh dgreiman, 2016-10-13 00:18 Unix shell script to demonstrate bug
Messages (11)
msg278544 - (view) Author: Douglas Greiman (dgreiman) * Date: 2016-10-13 00:18
This manifests in the following way: A package directory containing an __init__.py that is a symlink to /dev/null is treated as a namespace package instead of a regular package.

The Bazel build tool creates many __init__.py files in this way, which is how I even ran into this.

Symlinks to regular files seem fine.
msg278547 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2016-10-13 06:46
Linking __init__.py to /dev/null is very odd. Do you know bazel does that?
msg278548 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2016-10-13 06:46
I wanted to ask, Do you know why bazel does that?
msg278549 - (view) Author: Antti Haapala (ztane) * Date: 2016-10-13 07:50
One question is why doesn't it just try to `open`, but wants to stat first, when the python principle has always been EAFP.
msg278614 - (view) Author: Douglas Greiman (dgreiman) * Date: 2016-10-13 20:54
See associated bug filed against Bazel:

https://github.com/bazelbuild/bazel/issues/1458

As for why Bazel does that, it's related to the sandboxing implementation but I don't know any details beyond that.
msg278668 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-10-14 21:36
Since /dev/null is not a file according to os.path.isfile(), I'm going to close this as not a bug.

And to answer Antti's question, it's because the semantics come from the original C code where EAFP is not pleasant.
msg282727 - (view) Author: Douglas Greiman (dgreiman) * Date: 2016-12-08 18:43
Is there a reasonable place to document that __init__.py (and probably source files in general) must be a "regular file" by the Unix definition, and not a device file, socket, etc?
msg282792 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-12-09 17:32
There's probably a place to mention it in the language spec for import and/or importlib.
msg282812 - (view) Author: Douglas Greiman (dgreiman) * Date: 2016-12-10 00:16
To be thorough, I looked at non __init__.py source files as well, with erratic results.  And who knows what would happen on other OS's.  So a blanket "must be a regular file" seems reasonable.

$ ls -l b.*
lrwxrwxrwx 1 dgreiman eng 9 Dec  9 16:05 b.py -> /dev/null

$ python2.7 -m b             # Success
$ python2.7 -c 'import b'    # Success
$ python2.7 b.py             # Success

$ python3.5 b.py             # Success
$ python3.5 -m b             # Failure
/opt/python3.5/bin/python3.5: No module named b
$ python3.5 -c 'import b'    # Failure
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'b'

$ uname -a
Linux 53334eb940e6 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016 x86_64 GNU/Linux
msg282869 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-12-10 19:53
Those results are expected as passing in a file by path means it's just read by open() and then passed to exec() while the others pass through import itself which has the os.path.isfile() check.

Since this is a change from Python 2.7 I would be willing to look at a patch that loosened things that was fully backwards-compatible, but it would be an enhancement and not a bugfix.
msg288722 - (view) Author: Douglas Greiman (dgreiman) * Date: 2017-02-28 19:52
Bazel has been updated to no longer create symlinks to /dev/null

https://github.com/bazelbuild/bazel/issues/1458
History
Date User Action Args
2017-02-28 19:52:18dgreimansetmessages: + msg288722
2016-12-10 19:53:48brett.cannonsetmessages: + msg282869
2016-12-10 00:16:44dgreimansetmessages: + msg282812
2016-12-09 17:32:08brett.cannonsetmessages: + msg282792
2016-12-08 18:43:58dgreimansetmessages: + msg282727
2016-10-14 21:36:25brett.cannonsetstatus: open -> closed

nosy: + brett.cannon
messages: + msg278668

resolution: not a bug
2016-10-14 21:26:21terry.reedysetstage: test needed
versions: + Python 3.6, Python 3.7, - Python 3.4
2016-10-13 20:54:06dgreimansetmessages: + msg278614
2016-10-13 19:36:03eric.smithsetnosy: + eric.smith
2016-10-13 07:50:49ztanesetnosy: + ztane
messages: + msg278549
2016-10-13 06:46:24orsenthilsetmessages: + msg278548
2016-10-13 06:46:00orsenthilsetnosy: + orsenthil
messages: + msg278547
2016-10-13 00:18:52dgreimancreate