Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'collections.abc' is no longer defined when collections is imported #64983

Closed
bitdancer opened this issue Feb 26, 2014 · 18 comments
Closed

'collections.abc' is no longer defined when collections is imported #64983

bitdancer opened this issue Feb 26, 2014 · 18 comments
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@bitdancer
Copy link
Member

BPO 20784
Nosy @warsaw, @rhettinger, @ncoghlan, @pitrou, @vstinner, @larryhastings, @tiran, @bitdancer, @ericsnowcurrently, @serhiy-storchaka, @pawciobiel

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-08.22:49:26.872>
created_at = <Date 2014-02-26.18:25:38.898>
labels = ['type-bug', 'library']
title = "'collections.abc' is no longer defined when collections is imported"
updated_at = <Date 2018-04-11.11:05:20.144>
user = 'https://github.com/bitdancer'

bugs.python.org fields:

activity = <Date 2018-04-11.11:05:20.144>
actor = 'pawciobiel'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-08.22:49:26.872>
closer = 'christian.heimes'
components = ['Library (Lib)']
creation = <Date 2014-02-26.18:25:38.898>
creator = 'r.david.murray'
dependencies = []
files = []
hgrepos = []
issue_num = 20784
keywords = ['3.4regression']
message_count = 18.0
messages = ['212284', '212286', '212288', '212289', '212291', '212293', '212294', '212302', '212305', '213126', '213128', '213129', '213134', '213135', '213136', '213411', '213412', '315190']
nosy_count = 14.0
nosy_names = ['barry', 'rhettinger', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'christian.heimes', 'Arfrever', 'r.david.murray', 'Jurko.Gospodneti\xc4\x87', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'pawciobiel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20784'
versions = ['Python 2.7', 'Python 3.4']

@bitdancer
Copy link
Member Author

collections.abc was renamed _collections_abc in bpo-19218. The __init__ file was modified to load all the abc into the collections namespace, but the 'abc' name itself is no longer defined:

Python 3.3.2 (default, Dec 17 2013, 17:24:42) 
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
>>> collections.abc
<module 'collections.abc' from '/usr/lib/python3.3/collections/abc.py'>

Python 3.4.0rc1+ (default:1bc585ba5df2, Feb 24 2014, 15:04:31) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
>>> collections.abc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'abc'

It looks like the import statement for _collections_abc in the __init__ file as has missing "as abc" phrase.

This is probably not important enough to require fixing in the RC, since 'import collections.abc' works fine, but it is a regression.

@bitdancer bitdancer added the type-bug An unexpected behavior, bug, or error label Feb 26, 2014
@larryhastings
Copy link
Contributor

I'm pretty sure that if you import "x", there are zero guarantees that "x.y" will work. The offical line is that you must explicitly import all the deepest submodules you use. So I don't think this is even a bug.

@bitdancer
Copy link
Member Author

It is a backward compatibility bug. Something that used to work doesn't any more. And it was explicitly *made* to work previously (the original __init__ statement was 'import collections.abc'). And it is is an implementation bug in the original patch because otherwise there would be no point in importing _collections_abc in __init__.

But you are right, it is relatively unlikely that anyone is relying on it.

@bitdancer
Copy link
Member Author

If we decide we want this (small) backward compatibility break, to make collections.abc consistent with the other modules (except os.path), then I should mention it in the whatsnew porting section for 3.4, which is really why I opened this issue :)

I that case, IMO, the import for _collections_abc should be removed from the collections __init__ file (in 3.4.1), just to keep things tidy.

@pitrou
Copy link
Member

pitrou commented Feb 26, 2014

I'm pretty sure that if you import "x", there are zero guarantees that
"x.y" will work. The offical line is that you must explicitly import
all the deepest submodules you use.

I'm not sure why you're saying that. I think it's quite common to only
"import os" and then use os.path.

@pitrou
Copy link
Member

pitrou commented Feb 26, 2014

A quick grep indicates at least the following modules use os.path but only import os:
bdb, binhex, cgitb, compileall, cProfile, doctest, filecmp, fileinput, fnmatch, ftplib, gettext, glob, imghdr, imp, inspect, linecache, mailbox, mimetypes, modulefinder, netrc, optparse, pdb, platform, profile, pstats, pyclbr, pydoc, shlex, site, sndhdr, ssl, subprocess, tabnanny, tarfile, trace, uuid, uu, webbrowser, zipfile.

So, it's a very common idiom.

@serhiy-storchaka
Copy link
Member

May be add temporary replacement for collections.abc?

class _AbcModulePlaceholder(type(_collections_abc)):
    def __warn(self):
        import warnings
        warnings.warn('collections.abc used without importing',
                      DeprecationWarning, 3)
    def __getattr__(self, name):
        self.__warn()
        return getattr(_collections_abc, name)
    def __setattr__(self, name, value):
        self.__warn()
        setattr(_collections_abc, name, value)
    def __delattr__(self, name):
        self.__warn()
        delattr(_collections_abc, name)
    def __dir__(self):
        self.__warn()
        return dir(_collections_abc)

abc = _AbcModulePlaceholder('abc')

@larryhastings
Copy link
Contributor

So, it's a very common idiom.

"Common" doesn't imply "correct" or "supported". There are plenty of other packages/modules who don't import their subpackages/submodules during initialization. Unless explicitly supported by the module, using a submodule without explicitly importing it is relying on undefined behavior.

I stand by my statement.

@pitrou
Copy link
Member

pitrou commented Feb 26, 2014

> So, it's a very common idiom.

"Common" doesn't imply "correct" or "supported". There are plenty of
other packages/modules who don't import their subpackages/submodules
during initialization. Unless explicitly supported by the module,
using a submodule without explicitly importing it is relying on
undefined behavior.

Yes, you are technically right. But this is an idealized view of what
backwards compatibility means in the real world.

Well, let's see if people complain after 3.4 is released, anyway.

@JurkoGospodneti
Copy link
Mannequin

JurkoGospodneti mannequin commented Mar 11, 2014

Item possibly related to this. When packaging a simple HelloWorld-like application like this:

  print("Hello world")
  import configparser

using cx_Freeze and Python 3.4, you get the following error on packaged application startup:

Traceback (most recent call last):
  File "C:\Program Files\Python\Python34rc3\lib\site-packages\cx_freeze-4.3.2-py3.4-win-amd64.egg\cx_Freeze\initscripts\Console3.py", line 27, in <module>
    exec(code, m.__dict__)
  File "hello_world.py", line 4, in <module>
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\configparser.py", line 121, in <module>
    from collections.abc import MutableMapping
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2201, in _find_and_load_unlocked
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'collections.abc'

As I see it, the config parser module is attempting to import
'collections.abc'.

The same does not occur with Python 3.3.5 or 3.3.3.

Not sure if this is due to something cx_Freeze does so it failed to collect some module in the created installation package, or if it's something to look into in Python itself. Just looked similar to this so I bring it up as additional info for anyone looking deeper into this issue.

Hope this helps.

Best regards,
Jurko Gospodnetić

@vstinner
Copy link
Member

Item possibly related to this.

I cannot reproduce your issue, it looks like a bug in cx_Freeze. "import collections" doesn't import "collections.abc" by default anymore, but configparser is correct: "from collections.abc import MutableMapping". So it imports explicitly "collections.abc".

@JurkoGospodneti
Copy link
Mannequin

JurkoGospodneti mannequin commented Mar 11, 2014

I cannot reproduce your issue

Meaning you do not have the environment set up for this or that you tried it and it worked for you?

If it 'worked for you', I can send you more detailed environment information when get back to my office in an hour or so.

@vstinner
Copy link
Member

> I cannot reproduce your issue
Meaning you do not have the environment set up for this or that
you tried it and it worked for you?

I mean that executing the following lines in Python doesn't fail:
---

print("Hello world")
import configparser

It's specific to cx_Freeze and unrelated to this issue.

@ncoghlan
Copy link
Contributor

cx-freeze feedback is better added to bpo-20884 - I currently still suspect that 3.4 has just uncovered some latent spec non-compliance in the cx-freeze import system emulation (although I need to investigate further to be sure, since it's also possible we accidentally broke backwards compatibility with certain kinds of legacy behaviour when implementing PEP-451).

For this issue, I don't think os.path can be used as a precedent - "os" isn't a package, and "os.path" isn't a normal submodule (instead, the os module does a dance to figure out which top level module to import as path - it's usually either ntpath or posixpath).

Instead, this is just normal submodule import behaviour - "import collections" doesn't necessarily imply "import collections.abc", it just used to do so because that was how the backwards compatibility for the old names happened to be implemented.

Setting the attribute would also shadow the submodule, which would be a little weird (since collections.abc could then refer to either that module or to _collections_abc). Really, the change to collections/init.py could just be reverted - the _collections_abc move was just to avoid running collections/init.py at startup, so there's no need to use the hack inside collections/init.py itself.

@ncoghlan
Copy link
Contributor

Oh, and yes, this is *definitely* a bug: _collections_abc.__name__ is set to "collections.abc", but if "collections.abc" isn't imported anywhere in the program, then cx-freeze and similar tools will miss the fact that "collections.abc" should be bundled.

If "collections" is changed back to implicitly importing the submodule, that problem should go away (although the lie in __name__ for pickle compatibility may still cause problems with freezing, so it's perhaps worth mentioning in the porting notes regardless).

@python-dev
Copy link
Mannequin

python-dev mannequin commented Mar 13, 2014

New changeset d575398d1916 by R David Murray in branch 'default':
whatsnew: collections no longer implicitly imports 'abc' (bpo-20784).
http://hg.python.org/cpython/rev/d575398d1916

@bitdancer
Copy link
Member Author

I've documented that 'collections.abc' is no longer implicit, which I presume means we are going to keep this behavior. (Unless you tell me to revert that and we fix it as a regression in 3.4.1).

As long as an application follows the note and explicitly imports collections.abc, I would think that the freeze tools would do the right thing. I'd think they'd do the right thing anyway, since _collections_abc appears in an 'import from' in collections.__init__, so I'm not clear what problem is anticipated for freeze tools that is different from the one that any program relying on the implicit import would face.

@tiran tiran closed this as completed Sep 8, 2016
@pawciobiel
Copy link
Mannequin

pawciobiel mannequin commented Apr 11, 2018

I had similar issue and it helped when as a workaround I use
'pip install .' as opposed to 'python setup.py develop' in my project.
I hope this will give you a hint.

@pawciobiel pawciobiel mannequin added the stdlib Python modules in the Lib dir label Apr 11, 2018
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

7 participants