classification
Title: __import__ with fromlist=
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: Aaron.Sterling, brett.cannon, eric.araujo, gsakkis, hauser, mrts, rhettinger
Priority: low Keywords: patch

Created on 2008-02-12 20:50 by hauser, last changed 2012-04-17 23:18 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
empty_import.tgz hauser, 2008-02-12 20:50 simple example of empty import
issue_2090.patch gsakkis, 2010-04-18 11:41
Messages (21)
msg62333 - (view) Author: (hauser) Date: 2008-02-12 20:50
This construction:

__import__( 'pkg', {}, {}, [''] )

Will cause double initialization of package 'pkg', once with name 'pkg'
and second one with name 'pkg.' (trailing dot). Implementation tries to
import subpackage of 'pkg' with empty name, and imports the same package
twice.

This kind of construction is used as a hacky way to obtain exact module
instead of top-level module in return value. It is a hack, but should
not cause this kind of side effects.
msg64079 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-03-19 18:04
As you said, it's a hack, so supporting an abuse of the API is not
reasonable. You don't have to set the fromlist for the import to work.
And if you are doing it to get the tail module, you can write some
simple code to use getattr() to walk down from the root module to the
one you want.

And I plan to add a much simpler API to the imp module for people to use
directly so that these abuses don't continue.
msg64080 - (view) Author: (hauser) Date: 2008-03-19 18:27
There are quite a few projects that use this solution:
http://google.com/codesearch?hl=en&lr=&q=__import__.*%5C%5B%27%27%5C%5D
. I would change it even if it is a hack, but I understand your point.
msg76461 - (view) Author: Mart Sõmermaa (mrts) Date: 2008-11-26 12:01
Just for reference, the simplest workaround is to use:

modname = "foo.bar.baz.baq"
mod = __import__(modname, {}, {}, [modname.rsplit(".", 1)[-1]])
msg76462 - (view) Author: Mart Sõmermaa (mrts) Date: 2008-11-26 12:02
See also http://bugs.python.org/issue4438
msg80987 - (view) Author: Mart Sõmermaa (mrts) Date: 2009-02-02 20:01
A pointer for people who keep referring to this bug -- after
discussions, the following idiom was selected as the "official" way to
import modules by name in 2.x (as seen in latest 2.x docs
http://docs.python.org/dev/library/functions.html#__import__ ).

---

If you simply want to import a module (potentially within a package) by
name, you can get it from sys.modules:

>>> import sys
>>> name = 'foo.bar.baz'
>>> __import__(name)
<module 'foo' from ...>
>>> baz = sys.modules[name]
>>> baz
<module 'foo.bar.baz' from ...>
msg80989 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2009-02-02 20:10
And just some more info, Python 2.7/3.1 have gained the importlib
module/package and its import_module function which gives a much saner
API than __import__.
msg103244 - (view) Author: George Sakkis (gsakkis) Date: 2010-04-15 18:30
Just bitten by this (through a 3rd party library that uses this pattern) and I'm wondering why it was closed as invalid. Passing a non-empty fromlist string also imports the tail module but without the side effect of double import, so it's not generally harmful. More surprisingly, a colleague discovered accidentally that the same behavior happens if you pass one or more slashes: __import__('pkg', fromlist=['', '/', '//']) imports 'pkg', 'pkg.', 'pkg./' and 'pkg.//' !

I'm not arguing that using fromlist to import the tail module is not a hack, but the behavior for empty strings and slashes (and whatever else causes multiple imports) is clearly a bug. Unless someone is actually relying on this double import behavior (very unlikely), I think it should be fixed.
msg103259 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-04-15 21:31
If you want a justification, think of it as undefined behavior. When you use an empty string in fromlist you are essentially simulating ``from pkg import`` which makes absolutely no sense, so no one has cared enough to try to fix this. It's such a hack that I don't think people need to worry about fixing it, especially with a more sanctioned way to do it and with importlib being available in PyPI and running in Python 2.3 and later.

Now if someone bothers to submit a patch to fix the issue that is reasonable then it can be considered for fixing, but I view this as such a nonsensical call signature that I personally don't see the need to have someone burn some time on this unless they really care. As a compromise I have made this a "wont fix" bug, but I still don't see the need to open the bug again.
msg103260 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-04-15 21:47
I concur with Brett.  For the most part, we don't care about implementation artifacts and undefined behaviors (as long as it doesn't segfault).
msg103262 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-04-15 21:56
Since ``from pkg import`` makes no sense, would it be okay if __import__ with an empty fromlist or slashes raised an error?
msg103266 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-04-15 22:10
On Thu, Apr 15, 2010 at 14:56, Éric Araujo <report@bugs.python.org> wrote:

That's fine with me if someone wrote a patch that did that.
msg103268 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-04-15 22:20
Although now that I think about it, there is a slightly sticky situation of someone using '' or some name with a slash for a key in __dict__. The usage in fromlist would then be "reasonable", but the semantics would be somewhat odd as fromlist is really only needed to trigger other imports.

It's probably safe to still make it an error, although it shouldn't be special-cased just for slashes and spaces but specifically the logic triggering the double import.
msg103269 - (view) Author: George Sakkis (gsakkis) Date: 2010-04-15 22:33
> When you use an empty string in fromlist you are essentially simulating 
> ``from pkg import`` which makes absolutely no sense, so no one has
> cared enough to try to fix this.

``from pkg import __bogus__, 123, @$%`` doesn't make sense either and yet the equivalent __import__ call doesn't cause multiple imports neither binds __name__ to bogus strings, it just imports and returns pkg.

> Since ``from pkg import`` makes no sense, would it be okay if
> __import__ with an empty fromlist or slashes raised an error?

No, this would break lots of working code and would be inconsistent anyway with other invalid fromlist inputs. The backwards compatible solution would be to treat the empty string (and slashes) like every other input, i.e. prevent multiple imports.
msg103270 - (view) Author: George Sakkis (gsakkis) Date: 2010-04-15 22:51
More fun findings: dots are special-cased too, but only if they don't appear consecutively (!);

~$ cat pkg/__init__.py
print  __name__

~$ python -c "__import__('pkg', fromlist=['.'])"
pkg
pkg..
~$ python -c "__import__('pkg', fromlist=['..'])"
pkg
~$ python -c "__import__('pkg', fromlist=['...'])"
pkg
~$ python -c "__import__('pkg', fromlist=['././//.'])"
pkg
pkg.././//.
~$ python -c "__import__('pkg', fromlist=['././../'])"
pkg
msg103482 - (view) Author: George Sakkis (gsakkis) Date: 2010-04-18 11:41
FWIW attached is a patch that allows only valid identifiers before calling import_submodule(), and returns silently otherwise (for backwards compatibility).

For the record, the reason that empty strings and some combinations of slashes/dots caused the double import was that they were concatenated to the path, and if the final path was a valid directory and contained an __init__.py it was imported. E.g. __import__('pkg.subpkg', fromlist=['/../.']) ends up looking in "pkg/subpkg//../.". On the surface this seems like a potential directory traversal attack hole, although I couldn't get past 'pkg' by passing '../../../', so I guess there must be other checks before attempting the import.
msg103485 - (view) Author: George Sakkis (gsakkis) Date: 2010-04-18 12:05
> On the surface this seems like a potential directory traversal attack
> hole, although I couldn't get past 'pkg' by passing '../../../', so I 
> guess there must be other checks before attempting the import.

I rushed to post; it turns out one *can* access packages in parent directories, so I think it's accurate to describe it as a directory traversal hole.
msg103512 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-04-18 18:32
Thanks for the patch, George. I will get it when I can.

And this make me even more glad that we removed the file path import from 3.x.
msg116855 - (view) Author: Aaron Sterling (Aaron.Sterling) Date: 2010-09-19 10:31
FWIW, I also get this behavior on 2.6.5 and there are claims that it occurs on 2.6.4 and 3.1.1. see http://stackoverflow.com/questions/3745221/import-calls-init-py-twice/3745273#3745273
msg116882 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-09-19 20:11
I replied to the Stack Overflow question. I should also mention that importlib is on PyPI and compatible back to PYthon 2.3.

I still plan to get to this some day, but I don't view this as a critical fix, just a nice thing to do for folks.
msg158576 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-17 23:18
Importlib does away with this issue.
History
Date User Action Args
2012-04-17 23:18:15brett.cannonsetstatus: open -> closed
resolution: out of date
messages: + msg158576

stage: resolved
2012-03-29 03:51:51eric.araujosetversions: + Python 3.3, - Python 3.1
2010-09-20 22:11:13eric.araujosetversions: + Python 3.2, - Python 2.6
2010-09-19 20:11:34brett.cannonsetmessages: + msg116882
2010-09-19 10:33:54Aaron.Sterlingsetversions: + Python 3.1, Python 2.7
2010-09-19 10:31:43Aaron.Sterlingsetnosy: + Aaron.Sterling

messages: + msg116855
versions: + Python 2.6, - Python 2.7
2010-08-05 00:00:41terry.reedysetversions: - Python 2.6, Python 2.5
2010-05-21 01:41:37brett.cannonsetfiles: - unnamed
2010-05-08 23:57:19brett.cannonsetstatus: closed -> open
resolution: wont fix -> (no value)
2010-04-18 18:32:52brett.cannonsetmessages: + msg103512
2010-04-18 12:05:36gsakkissetmessages: + msg103485
2010-04-18 11:41:14gsakkissetfiles: + issue_2090.patch
keywords: + patch
messages: + msg103482
2010-04-15 22:51:32gsakkissetmessages: + msg103270
2010-04-15 22:33:48gsakkissetmessages: + msg103269
2010-04-15 22:20:31brett.cannonsetmessages: + msg103268
2010-04-15 22:10:31brett.cannonsetfiles: + unnamed

messages: + msg103266
title: __import__ with fromlist=[''] causes double initialization of modules -> __import__ with fromlist=
2010-04-15 21:56:41eric.araujosetnosy: + eric.araujo
messages: + msg103262
2010-04-15 21:47:01rhettingersetnosy: + rhettinger
messages: + msg103260
2010-04-15 21:31:32brett.cannonsetpriority: normal -> low
resolution: not a bug -> wont fix
messages: + msg103259
2010-04-15 18:30:29gsakkissetnosy: + gsakkis

messages: + msg103244
versions: + Python 2.7
2009-02-02 20:10:56brett.cannonsetmessages: + msg80989
2009-02-02 20:01:30mrtssetmessages: + msg80987
2008-11-26 12:02:04mrtssetmessages: + msg76462
2008-11-26 12:01:05mrtssetnosy: + mrts
messages: + msg76461
2008-03-19 18:27:34hausersetmessages: + msg64080
versions: + Python 2.5
2008-03-19 18:04:56brett.cannonsetstatus: open -> closed
resolution: not a bug
messages: + msg64079
2008-03-18 19:52:28jafosetpriority: normal
assignee: brett.cannon
nosy: + brett.cannon
2008-02-12 20:50:35hausercreate