test_typing is flaky #68823

bitdancer · 2015-07-15T01:34:05Z

BPO	24635
Nosy	@gvanrossum, @vstinner, @larryhastings, @bitdancer, @ambv, @gvanrossum

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gvanrossum'
closed_at = <Date 2015-09-04.19:49:21.019>
created_at = <Date 2015-07-15.01:34:05.095>
labels = ['type-bug', 'tests', 'release-blocker']
title = 'test_typing is flaky'
updated_at = <Date 2015-09-04.19:57:35.514>
user = 'https://github.com/bitdancer'

bugs.python.org fields:

activity = <Date 2015-09-04.19:57:35.514>
actor = 'gvanrossum'
assignee = 'gvanrossum'
closed = True
closed_date = <Date 2015-09-04.19:49:21.019>
closer = 'larry'
components = ['Tests']
creation = <Date 2015-07-15.01:34:05.095>
creator = 'r.david.murray'
dependencies = []
files = []
hgrepos = []
issue_num = 24635
keywords = []
message_count = 18.0
messages = ['246750', '246854', '247394', '249720', '249721', '249730', '249731', '249735', '249769', '249785', '249809', '249811', '249812', '249813', '249814', '249816', '249820', '249821']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'vstinner', 'larry', 'r.david.murray', 'lukasz.langa', 'python-dev', 'Guido.van.Rossum']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24635'
versions = ['Python 3.5', 'Python 3.6']

bitdancer · 2015-07-15T01:34:04Z

See for example:

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Non-Debug%203.x/builds/106

It might be a test order dependency, but see

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/11931

where the test passed when re-run at the end.

I'm making this a release blocker because it would be pretty sad to ship a new feature with a flaky test. Larry can of course overrule me on that :)

I note that no one is yet listed as the subject expert for the typing module. I've at least added a line for the module to experts.rst.

gvanrossum · 2015-07-17T14:10:55Z

You can list me as the expert for typing.py, since I wrote it. :-) (However, until mid August I have limited availability since I'm on vacation.)

This looks indeed like a test order dependency. The three failures are all basic failures where an empty set, dict or list is expected to be an instance of the corresponding ABC (AbstractSet, Mapping, Sequence) defined in typing.py. Those ABCs in turn derive (in a slightly sneaky way) from the corresponding ABCs in collections.abc.

My hunch is that some other test messes with the ABC registration cache and doesn't restore it -- or typing.py's sneaky way of deriving from collections.abc has a subtle bug in it. Maybe something is calling reload(collections.abc)?

I haven't done any research to confirm or deny this theory. It shouldn't be too hard to find the (probably only a handful of) tests that mess with the registration cache.

larryhastings · 2015-07-25T21:08:06Z

Deferring for beta 4. We should still fix before release. Preferably before RC1 depending on the availability of Certain Persons.

vstinner · 2015-09-04T06:58:30Z

Is there any progress on this issue? It's still failing randomly:
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.5/builds/221/steps/test/logs/stdio

vstinner · 2015-09-04T06:59:51Z

======================================================================
FAIL: test_abstractset (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 955, in test_abstractset
    assert isinstance(set(), typing.AbstractSet)
AssertionError

======================================================================
FAIL: test_bytestring (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 979, in test_bytestring
    assert isinstance(b'', typing.ByteString)
AssertionError

======================================================================
FAIL: test_container (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 951, in test_container
    assert isinstance([], typing.Container)
AssertionError

======================================================================
FAIL: test_iterable (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 936, in test_iterable
    assert isinstance([], typing.Iterable)
AssertionError

======================================================================
FAIL: test_iterator (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 942, in test_iterator
    assert isinstance(it, typing.Iterator)
AssertionError

======================================================================
FAIL: test_mapping (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 963, in test_mapping
    assert isinstance({}, typing.Mapping)
AssertionError

======================================================================
FAIL: test_mutablemapping (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 967, in test_mutablemapping
    assert isinstance({}, typing.MutableMapping)
AssertionError

======================================================================
FAIL: test_mutablesequence (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 975, in test_mutablesequence
    assert isinstance([], typing.MutableSequence)
AssertionError

======================================================================
FAIL: test_mutableset (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 959, in test_mutableset
    assert isinstance(set(), typing.MutableSet)
AssertionError

======================================================================
FAIL: test_sequence (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 971, in test_sequence
    assert isinstance([], typing.Sequence)
AssertionError

larryhastings · 2015-09-04T07:12:15Z

Anything happening with this? We tag 3.5.0rc3 in about 36 hours.

vstinner · 2015-09-04T07:12:56Z

Anything happening with this? We tag 3.5.0rc3 in about 36 hours.

I believe that it's a bug in test_typing, not in the typing module. So it must not block the release.

vstinner · 2015-09-04T07:52:54Z

Ok, I found a short script to reproduce the bug:

import typing:
assert isinstance([], typing.Container)
assert isinstance([], typing.Container)

The first check calls typing.GenericMeta.__subclasscheck__(list) which calls abc.ABCMeta.__subclasscheck__(typing.Container, list). The problem is that not only abc.ABCMeta.__subclasscheck__() returns False, but it adds also list to typing.Container._abc_negative_cache.

The next check returns False without calling typing.GenericMeta.__subclasscheck__(list), in fact ABCMeta.__instancecheck__(typing.GenericMeta, list) returns immediatly False because list is in typing.Container._abc_negative_cache.

The problem: GenericMeta calls ABCMeta.__subclasscheck__() which returns False, but ABCMeta.__subclasscheck__() also stores the result "False" in cache, whereas it's not the expected result.

Should we override __instancecheck__ in GenericMeta?

@guido: are you interested to fix this complex metaclass/type issue? I don't have time to fix it right now.

gvanrossum · 2015-09-04T14:41:46Z

Sorry, this is a bug in typing.py. python/typing#155

It's a subtle bug but the first reporter has done some good research; I will try to find time to fix it today.

vstinner · 2015-09-04T16:47:46Z

It's a subtle bug but the first reporter has done some good research; I will try to find time to fix it today.

Oh, it's already known and investigated by someone else. Great. I
didn't know this github project.

python-dev · 2015-09-04T19:00:40Z

New changeset d1f41c614e62 by Guido van Rossum in branch '3.5':
Issue bpo-24635: Fixed flakiness in test_typing.py.
https://hg.python.org/cpython/rev/d1f41c614e62

gvanrossum · 2015-09-04T19:02:19Z

Fixed now on the 3.5 branch. I still have to merge to default and create the pull request for Larry.

gvanrossum · 2015-09-04T19:19:10Z

Pull request for Larry: https://bitbucket.org/larry/cpython350/pull-requests/14/fix-issue-24635/diff

gvanrossum · 2015-09-04T19:21:38Z

BTW, I screwed up and committed this into the public 3.5 repo first (and merged from there into default). I guess once Larry has merged the fix into his special closed 3.5 bitbucket repo I should do a null merge from there back to public 3.5 and forward the null merge to 3.6 as well.

gvanrossum · 2015-09-04T19:22:26Z

Assigning to Larry since the next step is his.

larryhastings · 2015-09-04T19:49:21Z

Pull request accepted. Please forward merge, thanks!

And, yes, this will be a null merge because you already separately committed it to 3.5.

python-dev · 2015-09-04T19:57:25Z

New changeset 0f37918440c9 by Guido van Rossum in branch 'default':
Issue bpo-24635: Fixed flakiness in test_typing.py. (Merge from 3.5.)
https://hg.python.org/cpython/rev/0f37918440c9

New changeset 438dde69871d by Guido van Rossum in branch '3.5':
Fix issue bpo-24635.
https://hg.python.org/cpython/rev/438dde69871d

gvanrossum · 2015-09-04T19:57:35Z

OK, done.

bitdancer added release-blocker tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Jul 15, 2015

larryhastings added deferred-blocker and removed release-blocker labels Jul 25, 2015

larryhastings removed the deferred-blocker label Sep 4, 2015

gvanrossum self-assigned this Sep 4, 2015

gvanrossum assigned larryhastings and unassigned gvanrossum Sep 4, 2015

gvanrossum added the release-blocker label Sep 4, 2015

larryhastings closed this as completed Sep 4, 2015

larryhastings assigned gvanrossum and unassigned larryhastings Sep 4, 2015

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_typing is flaky #68823

test_typing is flaky #68823

bitdancer commented Jul 15, 2015

bitdancer commented Jul 15, 2015

gvanrossum commented Jul 17, 2015

larryhastings commented Jul 25, 2015

vstinner commented Sep 4, 2015

vstinner commented Sep 4, 2015

larryhastings commented Sep 4, 2015

vstinner commented Sep 4, 2015

vstinner commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

vstinner commented Sep 4, 2015

python-dev mannequin commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

larryhastings commented Sep 4, 2015

python-dev mannequin commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

test_typing is flaky #68823

test_typing is flaky #68823

Comments

bitdancer commented Jul 15, 2015

bitdancer commented Jul 15, 2015

gvanrossum commented Jul 17, 2015

larryhastings commented Jul 25, 2015

vstinner commented Sep 4, 2015

vstinner commented Sep 4, 2015

larryhastings commented Sep 4, 2015

vstinner commented Sep 4, 2015

vstinner commented Sep 4, 2015

Ok, I found a short script to reproduce the bug:

import typing: assert isinstance([], typing.Container) assert isinstance([], typing.Container)

gvanrossum commented Sep 4, 2015

vstinner commented Sep 4, 2015

python-dev mannequin commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

larryhastings commented Sep 4, 2015

python-dev mannequin commented Sep 4, 2015

gvanrossum commented Sep 4, 2015

import typing:
assert isinstance([], typing.Container)
assert isinstance([], typing.Container)