classification
Title: test_typing is flaky
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: Guido.van.Rossum, gvanrossum, larry, lukasz.langa, python-dev, r.david.murray, vstinner
Priority: release blocker Keywords:

Created on 2015-07-15 01:34 by r.david.murray, last changed 2015-09-04 19:57 by gvanrossum. This issue is now closed.

Messages (18)
msg246750 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-07-15 01:34
See for example:

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Non-Debug%203.x/builds/106

It might be a test order dependency, but see

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/11931

where the test passed when re-run at the end.

I'm making this a release blocker because it would be pretty sad to ship a new feature with a flaky test.  Larry can of course overrule me on that :)

I note that no one is yet listed as the subject expert for the typing module.  I've at least added a line for the module to experts.rst.
msg246854 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-07-17 14:10
You can list me as the expert for typing.py, since I wrote it. :-)  (However, until mid August I have limited availability since I'm on vacation.)

This looks indeed like a test order dependency.  The three failures are all basic failures where an empty set, dict or list is expected to be an instance of the corresponding ABC (AbstractSet, Mapping, Sequence) defined in typing.py. Those ABCs in turn derive (in a slightly sneaky way) from the corresponding ABCs in collections.abc.

My hunch is that some other test messes with the ABC registration cache and doesn't restore it -- or typing.py's sneaky way of deriving from collections.abc has a subtle bug in it.  Maybe something is calling reload(collections.abc)?

I haven't done any research to confirm or deny this theory.  It shouldn't be too hard to find the (probably only a handful of) tests that mess with the registration cache.
msg247394 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-07-25 21:08
Deferring for beta 4.  We should still fix before release.  Preferably before RC1 depending on the availability of Certain Persons.
msg249720 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 06:58
Is there any progress on this issue? It's still failing randomly:
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.5/builds/221/steps/test/logs/stdio
msg249721 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 06:59
======================================================================
FAIL: test_abstractset (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 955, in test_abstractset
    assert isinstance(set(), typing.AbstractSet)
AssertionError

======================================================================
FAIL: test_bytestring (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 979, in test_bytestring
    assert isinstance(b'', typing.ByteString)
AssertionError

======================================================================
FAIL: test_container (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 951, in test_container
    assert isinstance([], typing.Container)
AssertionError

======================================================================
FAIL: test_iterable (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 936, in test_iterable
    assert isinstance([], typing.Iterable)
AssertionError

======================================================================
FAIL: test_iterator (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 942, in test_iterator
    assert isinstance(it, typing.Iterator)
AssertionError

======================================================================
FAIL: test_mapping (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 963, in test_mapping
    assert isinstance({}, typing.Mapping)
AssertionError

======================================================================
FAIL: test_mutablemapping (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 967, in test_mutablemapping
    assert isinstance({}, typing.MutableMapping)
AssertionError

======================================================================
FAIL: test_mutablesequence (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 975, in test_mutablesequence
    assert isinstance([], typing.MutableSequence)
AssertionError

======================================================================
FAIL: test_mutableset (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 959, in test_mutableset
    assert isinstance(set(), typing.MutableSet)
AssertionError

======================================================================
FAIL: test_sequence (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 971, in test_sequence
    assert isinstance([], typing.Sequence)
AssertionError

----------------------------------------------------------------------
msg249730 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-04 07:12
Anything happening with this?  We tag 3.5.0rc3 in about 36 hours.
msg249731 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 07:12
> Anything happening with this?  We tag 3.5.0rc3 in about 36 hours.

I believe that it's a bug in test_typing, not in the typing module. So it must not block the release.
msg249735 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 07:52
Ok, I found a short script to reproduce the bug:
--
import typing:
assert isinstance([], typing.Container)
assert isinstance([], typing.Container)
--

The first check calls typing.GenericMeta.__subclasscheck__(list) which calls abc.ABCMeta.__subclasscheck__(typing.Container, list). The problem is that not only abc.ABCMeta.__subclasscheck__() returns False, but it adds also list to typing.Container._abc_negative_cache.

The next check returns False without calling typing.GenericMeta.__subclasscheck__(list), in fact ABCMeta.__instancecheck__(typing.GenericMeta, list) returns immediatly False because list is in typing.Container._abc_negative_cache.

The problem: GenericMeta calls ABCMeta.__subclasscheck__() which returns False, but ABCMeta.__subclasscheck__() also stores the result "False" in cache, whereas it's not the expected result.

Should we override __instancecheck__ in GenericMeta?

@Guido: are you interested to fix this complex metaclass/type issue? I don't have time to fix it right now.
msg249769 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 14:41
Sorry, this is a bug in typing.py.  https://github.com/ambv/typehinting/issues/155

It's a subtle bug but the first reporter has done some good research; I will try to find time to fix it today.
msg249785 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 16:47
> It's a subtle bug but the first reporter has done some good research; I will try to find time to fix it today.

Oh, it's already known and investigated by someone else. Great. I
didn't know this github project.
msg249809 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-04 19:00
New changeset d1f41c614e62 by Guido van Rossum in branch '3.5':
Issue #24635: Fixed flakiness in test_typing.py.
https://hg.python.org/cpython/rev/d1f41c614e62
msg249811 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 19:02
Fixed now on the 3.5 branch. I still have to merge to default and create the pull request for Larry.
msg249812 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 19:19
Pull request for Larry: https://bitbucket.org/larry/cpython350/pull-requests/14/fix-issue-24635/diff
msg249813 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 19:21
BTW, I screwed up and committed this into the public 3.5 repo first (and merged from there into default). I guess once Larry has merged the fix into his special closed 3.5 bitbucket repo I should do a null merge from there back to public 3.5 and forward the null merge to 3.6 as well.
msg249814 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 19:22
Assigning to Larry since the next step is his.
msg249816 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-04 19:49
Pull request accepted.  Please forward merge, thanks!

And, yes, this will be a null merge because you already separately committed it to 3.5.
msg249820 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-04 19:57
New changeset 0f37918440c9 by Guido van Rossum in branch 'default':
Issue #24635: Fixed flakiness in test_typing.py. (Merge from 3.5.)
https://hg.python.org/cpython/rev/0f37918440c9

New changeset 438dde69871d by Guido van Rossum in branch '3.5':
Fix issue #24635.
https://hg.python.org/cpython/rev/438dde69871d
msg249821 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-04 19:57
OK, done.
History
Date User Action Args
2015-09-04 19:57:35gvanrossumsetmessages: + msg249821
2015-09-04 19:57:25python-devsetmessages: + msg249820
2015-09-04 19:49:21larrysetstatus: open -> closed
assignee: larry -> gvanrossum
messages: + msg249816

stage: needs patch -> resolved
2015-09-04 19:22:26gvanrossumsetpriority: high -> release blocker
assignee: gvanrossum -> larry
messages: + msg249814
2015-09-04 19:21:38gvanrossumsetmessages: + msg249813
2015-09-04 19:19:10gvanrossumsetmessages: + msg249812
2015-09-04 19:02:19gvanrossumsetassignee: gvanrossum
resolution: fixed
messages: + msg249811
2015-09-04 19:00:40python-devsetnosy: + python-dev
messages: + msg249809
2015-09-04 16:47:45vstinnersetmessages: + msg249785
2015-09-04 14:41:46gvanrossumsetmessages: + msg249769
2015-09-04 07:52:54vstinnersetmessages: + msg249735
2015-09-04 07:14:20larrysetpriority: deferred blocker -> high
2015-09-04 07:12:55vstinnersetmessages: + msg249731
2015-09-04 07:12:15larrysetmessages: + msg249730
2015-09-04 06:59:51vstinnersetmessages: + msg249721
2015-09-04 06:58:30vstinnersetmessages: + msg249720
2015-07-25 22:15:55vstinnersetnosy: + vstinner
2015-07-25 21:08:05larrysetpriority: release blocker -> deferred blocker
nosy: + larry
messages: + msg247394

2015-07-17 14:10:55gvanrossumsetnosy: + gvanrossum
messages: + msg246854
2015-07-16 04:33:12rhettingersetnosy: + lukasz.langa
2015-07-15 01:34:05r.david.murraycreate