classification
Title: 'collections.abc' is no longer defined when collections is imported
Type: behavior Stage: needs patch
Components: Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Jurko.Gospodnetić, barry, christian.heimes, eric.snow, haypo, larry, ncoghlan, pitrou, python-dev, r.david.murray, rhettinger, serhiy.storchaka
Priority: normal Keywords: 3.4regression

Created on 2014-02-26 18:25 by r.david.murray, last changed 2014-03-13 16:05 by r.david.murray.

Messages (17)
msg212284 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-26 18:25
collections.abc was renamed _collections_abc in issue 19218.  The __init__ file was modified to load all the abc into the collections namespace, but the 'abc' name itself is no longer defined:

Python 3.3.2 (default, Dec 17 2013, 17:24:42) 
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
>>> collections.abc
<module 'collections.abc' from '/usr/lib/python3.3/collections/abc.py'>

Python 3.4.0rc1+ (default:1bc585ba5df2, Feb 24 2014, 15:04:31) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
>>> collections.abc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'abc'

It looks like the import statement for _collections_abc in the __init__ file as has missing "as abc" phrase.

This is probably not important enough to require fixing in the RC, since 'import collections.abc' works fine, but it is a regression.
msg212286 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-02-26 18:34
I'm pretty sure that if you import "x", there are zero guarantees that "x.y" will work.  The offical line is that you must explicitly import all the deepest submodules you use.  So I don't think this is even a bug.
msg212288 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-26 18:58
It is a backward compatibility bug.  Something that used to work doesn't any more.  And it was explicitly *made* to work previously (the original __init__ statement was 'import collections.abc').  And it is is an implementation bug in the original patch because otherwise there would be no point in importing _collections_abc in __init__.

But you are right, it is relatively unlikely that anyone is relying on it.
msg212289 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-26 19:04
If we decide we want this (small) backward compatibility break, to make collections.abc consistent with the other modules (except os.path), then I should mention it in the whatsnew porting section for 3.4, which is really why I opened this issue :)

I that case, IMO, the import for _collections_abc should be removed from the collections __init__ file (in 3.4.1), just to keep things tidy.
msg212291 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-02-26 19:09
> I'm pretty sure that if you import "x", there are zero guarantees that
> "x.y" will work.  The offical line is that you must explicitly import
> all the deepest submodules you use.

I'm not sure why you're saying that. I think it's quite common to only
"import os" and then use os.path.
msg212293 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-02-26 19:13
A quick grep indicates at least the following modules use os.path but only import os:
bdb, binhex, cgitb, compileall, cProfile, doctest, filecmp, fileinput, fnmatch, ftplib, gettext, glob, imghdr, imp, inspect, linecache, mailbox, mimetypes, modulefinder, netrc, optparse, pdb, platform, profile, pstats, pyclbr, pydoc, shlex, site, sndhdr, ssl, subprocess, tabnanny, tarfile, trace, uuid, uu, webbrowser, zipfile.

So, it's a very common idiom.
msg212294 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-26 19:49
May be add temporary replacement for collections.abc?

class _AbcModulePlaceholder(type(_collections_abc)):
    def __warn(self):
        import warnings
        warnings.warn('collections.abc used without importing',
                      DeprecationWarning, 3)
    def __getattr__(self, name):
        self.__warn()
        return getattr(_collections_abc, name)
    def __setattr__(self, name, value):
        self.__warn()
        setattr(_collections_abc, name, value)
    def __delattr__(self, name):
        self.__warn()
        delattr(_collections_abc, name)
    def __dir__(self):
        self.__warn()
        return dir(_collections_abc)

abc = _AbcModulePlaceholder('abc')
msg212302 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-02-26 20:49
> So, it's a very common idiom.

"Common" doesn't imply "correct" or "supported".  There are plenty of other packages/modules who don't import their subpackages/submodules during initialization.  Unless explicitly supported by the module, using a submodule without explicitly importing it is relying on undefined behavior.

I stand by my statement.
msg212305 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-02-26 20:56
> > So, it's a very common idiom.
> 
> "Common" doesn't imply "correct" or "supported".  There are plenty of
> other packages/modules who don't import their subpackages/submodules
> during initialization.  Unless explicitly supported by the module,
> using a submodule without explicitly importing it is relying on
> undefined behavior.

Yes, you are technically right. But this is an idealized view of what
backwards compatibility means in the real world.

Well, let's see if people complain after 3.4 is released, anyway.
msg213126 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2014-03-11 07:45
Item possibly related to this. When packaging a simple HelloWorld-like application like this:

  print("Hello world")
  import configparser

using cx_Freeze and Python 3.4, you get the following error on packaged application startup:

Traceback (most recent call last):
  File "C:\Program Files\Python\Python34rc3\lib\site-packages\cx_freeze-4.3.2-py3.4-win-amd64.egg\cx_Freeze\initscripts\Console3.py", line 27, in <module>
    exec(code, m.__dict__)
  File "hello_world.py", line 4, in <module>
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\configparser.py", line 121, in <module>
    from collections.abc import MutableMapping
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2201, in _find_and_load_unlocked
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'collections.abc'

  As I see it, the config parser module is attempting to import
'collections.abc'.

  The same does not occur with Python 3.3.5 or 3.3.3.

  Not sure if this is due to something cx_Freeze does so it failed to collect some module in the created installation package, or if it's something to look into in Python itself. Just looked similar to this so I bring it up as additional info for anyone looking deeper into this issue.

  Hope this helps.

  Best regards,
    Jurko Gospodnetić
msg213128 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-03-11 08:28
> Item possibly related to this.

I cannot reproduce your issue, it looks like a bug in cx_Freeze. "import collections" doesn't import "collections.abc" by default anymore, but configparser is correct: "from collections.abc import MutableMapping". So it imports explicitly "collections.abc".
msg213129 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2014-03-11 08:59
> I cannot reproduce your issue

  Meaning you do not have the environment set up for this or that you tried it and it worked for you?

  If it 'worked for you', I can send you more detailed environment information when get back to my office in an hour or so.
msg213134 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-03-11 10:01
>> I cannot reproduce your issue
> Meaning you do not have the environment set up for this or that
> you tried it and it worked for you?

I mean that executing the following lines in Python doesn't fail:
---
print("Hello world")
import configparser
---

It's specific to cx_Freeze and unrelated to this issue.
msg213135 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-03-11 10:39
cx-freeze feedback is better added to issue 20884 - I currently still suspect that 3.4 has just uncovered some latent spec non-compliance in the cx-freeze import system emulation (although I need to investigate further to be sure, since it's also possible we accidentally broke backwards compatibility with certain kinds of legacy behaviour when implementing PEP 451).

For this issue, I don't think os.path can be used as a precedent - "os" isn't a package, and "os.path" isn't a normal submodule (instead, the os module does a dance to figure out which top level module to import as path - it's usually either ntpath or posixpath).

Instead, this is just normal submodule import behaviour - "import collections" doesn't necessarily imply "import collections.abc", it just used to do so because that was how the backwards compatibility for the old names happened to be implemented.

Setting the attribute would also shadow the submodule, which would be a little weird (since collections.abc could then refer to either that module or to _collections_abc). Really, the change to collections/__init__.py could just be reverted - the _collections_abc move was just to avoid running collections/__init__.py at startup, so there's no need to use the hack inside collections/__init__.py itself.
msg213136 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-03-11 10:52
Oh, and yes, this is *definitely* a bug: _collections_abc.__name__ is set to "collections.abc", but if "collections.abc" isn't imported anywhere in the program, then cx-freeze and similar tools will miss the fact that "collections.abc" should be bundled.

If "collections" is changed back to implicitly importing the submodule, that problem should go away (although the lie in __name__ for pickle compatibility may still cause problems with freezing, so it's perhaps worth mentioning in the porting notes regardless).
msg213411 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-13 16:01
New changeset d575398d1916 by R David Murray in branch 'default':
whatsnew: collections no longer implicitly imports 'abc' (#20784).
http://hg.python.org/cpython/rev/d575398d1916
msg213412 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-13 16:05
I've documented that 'collections.abc' is no longer implicit, which I presume means we are going to keep this behavior.  (Unless you tell me to revert that and we fix it as a regression in 3.4.1).

As long as an application follows the note and explicitly imports collections.abc, I would think that the freeze tools would do the right thing.  I'd think they'd do the right thing anyway, since _collections_abc appears in an 'import from' in collections.__init__, so I'm not clear what problem is anticipated for freeze tools that is different from the one that any program relying on the implicit import would face.
History
Date User Action Args
2014-03-13 16:05:15r.david.murraysetmessages: + msg213412
2014-03-13 16:01:10python-devsetmessages: + msg213411
2014-03-11 10:52:42ncoghlansetmessages: + msg213136
2014-03-11 10:39:08ncoghlansetnosy: + ncoghlan
messages: + msg213135
2014-03-11 10:01:12hayposetmessages: + msg213134
2014-03-11 08:59:50Jurko.Gospodnetićsetmessages: + msg213129
2014-03-11 08:28:11hayposetmessages: + msg213128
2014-03-11 07:45:51Jurko.Gospodnetićsetnosy: + Jurko.Gospodnetić
messages: + msg213126
2014-02-26 20:56:40pitrousetmessages: + msg212305
2014-02-26 20:49:05larrysetmessages: + msg212302
2014-02-26 19:49:32serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg212294
2014-02-26 19:13:58pitrousetmessages: + msg212293
2014-02-26 19:09:53pitrousetmessages: + msg212291
2014-02-26 19:04:05r.david.murraysetmessages: + msg212289
2014-02-26 18:58:57r.david.murraysetmessages: + msg212288
2014-02-26 18:34:22larrysetmessages: + msg212286
2014-02-26 18:25:38r.david.murraycreate