classification
Title: "TypeError: Parameterized generics cannot be used with class or instance checks" in test_functools after importing typing module
Type: behavior Stage: resolved
Components: Library (Lib), Tests Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Wilfred.Hughes, gvanrossum, levkivskyi, ncoghlan, ned.deily, python-dev, rhettinger, yan12125
Priority: normal Keywords: 3.5regression

Created on 2016-10-02 15:21 by Arfrever, last changed 2018-02-18 12:58 by levkivskyi. This issue is now closed.

Messages (21)
msg277896 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2016-10-02 15:21
Commits 09cc43df4509 (3.5), 81f27d3ab214 (3.6), 8f0df4db2b06 (3.7) cause "TypeError: Parameterized generics cannot be used with class or instance checks" errors in 4 tests in Lib/test/test_functools.py file in situation when typing module is already imported.

Examples of steps to reproduce:
$ LD_LIBRARY_PATH="$(pwd)" ./python -m test -v test_typing test_functools
$ LD_LIBRARY_PATH="$(pwd)" ./python -m test -v test___all__ test_functools
$ LD_LIBRARY_PATH="$(pwd)" ./python -c 'import runpy, typing; runpy.run_module("test")' -v test_functools

Errors in test_functools:

======================================================================
ERROR: test_cache_invalidation (test.test_functools.TestSingleDispatch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/cpython/Lib/functools.py", line 776, in dispatch
    impl = dispatch_cache[cls]
  File "/tmp/cpython/Lib/test/test_functools.py", line 1896, in __getitem__
    result = self.data[key]
KeyError: <class 'dict'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/cpython/Lib/functools.py", line 779, in dispatch
    impl = registry[cls]
KeyError: <class 'dict'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/cpython/Lib/test/test_functools.py", line 1955, in test_cache_invalidation
    self.assertEqual(g(d), "sized")
  File "/tmp/cpython/Lib/functools.py", line 801, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/tmp/cpython/Lib/functools.py", line 781, in dispatch
    impl = _find_impl(cls, registry)
  File "/tmp/cpython/Lib/functools.py", line 732, in _find_impl
    mro = _compose_mro(cls, registry.keys())
  File "/tmp/cpython/Lib/functools.py", line 709, in _compose_mro
    if sub not in bases and issubclass(cls, sub):
  File "/tmp/cpython/Lib/typing.py", line 1043, in __subclasscheck__
    raise TypeError("Parameterized generics cannot be used with class "
TypeError: Parameterized generics cannot be used with class or instance checks

======================================================================
ERROR: test_compose_mro (test.test_functools.TestSingleDispatch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/cpython/Lib/test/test_functools.py", line 1594, in test_compose_mro
    m = mro(c.defaultdict, [c.Sized, c.Container, str])
  File "/tmp/cpython/Lib/functools.py", line 709, in _compose_mro
    if sub not in bases and issubclass(cls, sub):
  File "/tmp/cpython/Lib/typing.py", line 1043, in __subclasscheck__
    raise TypeError("Parameterized generics cannot be used with class "
TypeError: Parameterized generics cannot be used with class or instance checks

======================================================================
ERROR: test_mro_conflicts (test.test_functools.TestSingleDispatch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/cpython/Lib/functools.py", line 776, in dispatch
    impl = dispatch_cache[cls]
  File "/tmp/cpython/Lib/test/test_functools.py", line 1896, in __getitem__
    result = self.data[key]
KeyError: <class 'collections.defaultdict'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/cpython/Lib/functools.py", line 779, in dispatch
    impl = registry[cls]
KeyError: <class 'collections.defaultdict'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/cpython/Lib/test/test_functools.py", line 1822, in test_mro_conflicts
    h(c.defaultdict(lambda: 0))
  File "/tmp/cpython/Lib/functools.py", line 801, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/tmp/cpython/Lib/functools.py", line 781, in dispatch
    impl = _find_impl(cls, registry)
  File "/tmp/cpython/Lib/functools.py", line 732, in _find_impl
    mro = _compose_mro(cls, registry.keys())
  File "/tmp/cpython/Lib/functools.py", line 709, in _compose_mro
    if sub not in bases and issubclass(cls, sub):
  File "/tmp/cpython/Lib/typing.py", line 1043, in __subclasscheck__
    raise TypeError("Parameterized generics cannot be used with class "
TypeError: Parameterized generics cannot be used with class or instance checks

======================================================================
ERROR: test_register_abc (test.test_functools.TestSingleDispatch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/cpython/Lib/functools.py", line 776, in dispatch
    impl = dispatch_cache[cls]
  File "/tmp/cpython/Lib/test/test_functools.py", line 1896, in __getitem__
    result = self.data[key]
KeyError: <class 'dict'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/cpython/Lib/functools.py", line 779, in dispatch
    impl = registry[cls]
KeyError: <class 'dict'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/cpython/Lib/test/test_functools.py", line 1641, in test_register_abc
    self.assertEqual(g(d), "sized")
  File "/tmp/cpython/Lib/functools.py", line 801, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/tmp/cpython/Lib/functools.py", line 781, in dispatch
    impl = _find_impl(cls, registry)
  File "/tmp/cpython/Lib/functools.py", line 732, in _find_impl
    mro = _compose_mro(cls, registry.keys())
  File "/tmp/cpython/Lib/functools.py", line 709, in _compose_mro
    if sub not in bases and issubclass(cls, sub):
  File "/tmp/cpython/Lib/typing.py", line 1043, in __subclasscheck__
    raise TypeError("Parameterized generics cannot be used with class "
TypeError: Parameterized generics cannot be used with class or instance checks

----------------------------------------------------------------------
Ran 186 tests in 0.431s

FAILED (errors=4)
msg277897 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-02 15:51
Sorry about that. I haven't looked into this but I suspect this is due to some manipulation of the ABC registry by typing.py that exceed their intended scope. I'll ask Ivan to fix it; if he's not available before b2 goes out I'll roll this back.
msg277928 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-03 04:00
Offline, Ivan and I have discussed a solution. We can make a small incision in typing.py that will fix this, at the cost of still allowing isinstance()/issubclass()). We also have a slightly better quick fix in mind. Ultimately we will have a more complex and complete fix, and if that's not ready by b2, one of the quicker fixes will definitely be.
msg277953 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-10-03 11:41
I have submitted a PR with one of the quick fixes upstream (to python/typing). Also I have played a bit with a more permanent fix. Here is an important observation: it is not easy to avoid adding parameterized generics to __subclasses__. For example, Node[int] should have at least one base, that base will have it in __subclasses__. The former is dynamically updated, so that we cannot "fool" it.

Also making subclass checks for all subclasses is a "deliberate act", so that it should be treated by common rules.

It looks like we have only three options:

1. Abandon the idea of raising TypeError for generics, most users expect True or False, so that some exiting code might break

2. Make __getitem__ for generics return self, so that ``Node[int] is Node`` at runtime (this is a subset of the first option).

3. Still force people not to use issubclass() with parameterized generics (this is quite bad idea and could have misleading consequences), but make an exception for existing stdlib modules abc and functools, all later additions should respect the rule.

4. Similarly to above, but just make tiny patches to abc and functools to use __origin__ in subclass checks if it is present and not None.

Which option is the right one? I would vote for the last one. This could break some (probably very small amount) existing code. So that formally speaking it is a backward incompatible change. We could go with option 1 for 3.5 and with option 4 for 3.6

Also I have found another failure in test suite with latest version from python/typing after importing typing while running ./python -c 'import runpy, typing; runpy.run_module("test")'

test test_collections failed -- Traceback (most recent call last):
  File "/Users/ivan/hg-cpython/Lib/test/test_collections.py", line 1309, in test_ByteString
    self.assertNotIsInstance(memoryview(b""), ByteString)
AssertionError: <memory at 0x113b129b8> is an instance of <class 'collections.abc.ByteString'>

This is because of this line in typing.py

ByteString.register(type(memoryview(b'')))

The fix for this is very simple, we just need to decide whether memoryview should be an instance of ByteString or not, and either remove this line or remove the failing test.

I do not have any strong opinion on this, what do you think?
msg277960 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-03 15:43
I have merged the upstream fix (and some other things) into 3.5, 3.6, 3.7.

changeset:   104262:7f0d27180b6d
tag:         tip
parent:      104259:36b052adf5a7
parent:      104261:0e0189b47291
user:        Guido van Rossum <guido@python.org>
date:        Mon Oct 03 08:42:17 2016 -0700
summary:     More updates from upstream typing.py (3.6->3.7)

changeset:   104261:0e0189b47291
branch:      3.6
parent:      104258:0d948a46c59a
parent:      104260:b24d0f274623
user:        Guido van Rossum <guido@python.org>
date:        Mon Oct 03 08:41:37 2016 -0700
summary:     More updates from upstream typing.py (3.5->3.6)

changeset:   104260:b24d0f274623
branch:      3.5
parent:      104255:ac838bf5499d
user:        Guido van Rossum <guido@python.org>
date:        Mon Oct 03 08:40:50 2016 -0700
summary:     More updates from upstream typing.py
msg278280 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-10-07 22:31
Since the quick fix is now applied, I think it should not be a release blocker any more.
msg278338 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-09 03:23
Making it a deferred release blocker so we're reminded to do the more thorough fix before 3.6.0 rc1.
msg278345 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2016-10-09 05:29
All tests from test_functools.py now pass.

test.test_collections.TestCollectionABCs.test_ByteString() from test_collections.py still fails (as reported in msg277953).

LD_LIBRARY_PATH="$(pwd)" ./python -c 'import runpy, typing; runpy.run_module("test")' -v test_collections
msg278375 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-09 17:01
Oh the line

ByteString.register(type(memoryview(b'')))

should be removed from typing.py.
msg278376 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-09 17:05
New changeset 69fe5f2e5aae by Guido van Rossum in branch '3.5':
Issue #28339: Remove ByteString.register(memoryview(...)) from typing.py.
https://hg.python.org/cpython/rev/69fe5f2e5aae

New changeset 8958836a2c89 by Guido van Rossum in branch '3.6':
Issue #28339: Remove ByteString.register(memoryview(...)) from typing.py. (merge 3.5->3.6)
https://hg.python.org/cpython/rev/8958836a2c89

New changeset def461406c70 by Guido van Rossum in branch 'default':
Issue #28339: Remove ByteString.register(memoryview(...)) from typing.py. (merge 3.6->3.7)
https://hg.python.org/cpython/rev/def461406c70
msg278377 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-09 17:07
Maybe this fix helps? I tried your repro and it no longer fails.
msg278378 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-10-09 17:16
I think this is the right way to fix this, as discussed on python-dev, memoryview is not considered consistent with ByteString.
msg278384 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-09 18:59
So is a larger fix not necessary? If so we can close this (assuming
the tests now pass).
msg278385 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-10-09 19:08
I tried and all tests pass on 3.7a also with prior import of typing.

A larger fix is not _necessary_, but I would _prefer_ to go with the option 4 that I proposed above, i.e.:

Instead of special casing abc and functools in __subclasshook__ in typing via sys._getframe, I would rather add small changes to abc and functools (they should use __origin__ in subclass checks).

In general, I think we should document __origin__, it could be useful at runtime (especially that we prohibit certain things like class checks for parameterized generics).
msg278386 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-10-09 19:45
Hm. I agree that _getframe() is horrible. But I'm not sure I want code outside typing.py to be aware of __origin__.
msg282575 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-06 23:06
This is still marked as "deferred blocker". Is there anything more that needs to be done for 3.6.0?  I'm assuming not.
msg282673 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-12-07 22:07
The failure is fixed now, but it is fixed not in an elegant way (sys._getframe is used).

I think this is not urgent. If there are no objections, then I would propose to change priority to normal.
msg285701 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-01-18 04:50
@levkivskyi Do you want to attempt a better fix in tome for 3.6.1?
msg285718 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-01-18 11:19
The problem is the only good way to fix this I see now is patching two lines in functools and abc using the public ``typing_inspect`` API proposed in http://bugs.python.org/issue29262 (I added it as a dependency). The latter could take some time, so that I am not 100% sure about 3.6.1.
msg304589 - (view) Author: Wilfred Hughes (Wilfred.Hughes) * Date: 2017-10-18 16:54
Note that this also affects the singledispatch library that backports singledispatch to Python 2: https://github.com/python/typing/issues/484
msg312308 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-02-18 12:58
FWIW, this is fixed in 3.7 by PEP 560. I don't think we will be able to get rid of `sys._getframe` workaround on 3.6, so I propose to just close this.
History
Date User Action Args
2018-02-18 12:58:26levkivskyisetstatus: open -> closed
messages: + msg312308

dependencies: - Provide a way to check for *real* typing.Union instances
resolution: fixed
stage: resolved
2017-10-18 16:54:55Wilfred.Hughessetnosy: + Wilfred.Hughes
messages: + msg304589
2017-01-18 11:19:17levkivskyisettype: behavior
dependencies: + Provide a way to check for *real* typing.Union instances
messages: + msg285718
2017-01-18 05:17:56larrysetnosy: - larry
2017-01-18 04:50:20gvanrossumsetmessages: + msg285701
2016-12-07 22:26:12gvanrossumsetpriority: deferred blocker -> normal
2016-12-07 22:07:29levkivskyisetmessages: + msg282673
2016-12-06 23:06:39ned.deilysetmessages: + msg282575
2016-10-09 19:45:28gvanrossumsetpriority: release blocker -> deferred blocker

messages: + msg278386
2016-10-09 19:08:51levkivskyisetmessages: + msg278385
2016-10-09 18:59:16gvanrossumsetmessages: + msg278384
2016-10-09 17:16:59levkivskyisetmessages: + msg278378
2016-10-09 17:07:25gvanrossumsetmessages: + msg278377
2016-10-09 17:05:08python-devsetnosy: + python-dev
messages: + msg278376
2016-10-09 17:01:07gvanrossumsetpriority: deferred blocker -> release blocker

messages: + msg278375
2016-10-09 05:29:11Arfreversetmessages: + msg278345
2016-10-09 05:29:06Arfreversetmessages: - msg278344
2016-10-09 05:21:42Arfreversetmessages: + msg278344
2016-10-09 03:23:05gvanrossumsetpriority: release blocker -> deferred blocker

messages: + msg278338
2016-10-07 22:31:29levkivskyisetmessages: + msg278280
2016-10-03 15:43:17gvanrossumsetmessages: + msg277960
2016-10-03 11:41:18levkivskyisetmessages: + msg277953
2016-10-03 06:16:34yan12125setnosy: + yan12125
2016-10-03 04:00:10gvanrossumsetmessages: + msg277928
2016-10-02 15:51:14gvanrossumsetpriority: normal -> release blocker

nosy: + larry, levkivskyi, ned.deily
messages: + msg277897

keywords: + 3.5regression
2016-10-02 15:21:33Arfrevercreate