See for example:

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Non-Debug%203.x/builds/106

It might be a test order dependency, but see

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/11931

where the test passed when re-run at the end.

I'm making this a release blocker because it would be pretty sad to ship a new feature with a flaky test.  Larry can of course overrule me on that :)

I note that no one is yet listed as the subject expert for the typing module.  I've at least added a line for the module to experts.rst.

You can list me as the expert for typing.py, since I wrote it. :-)  (However, until mid August I have limited availability since I'm on vacation.)

This looks indeed like a test order dependency.  The three failures are all basic failures where an empty set, dict or list is expected to be an instance of the corresponding ABC (AbstractSet, Mapping, Sequence) defined in typing.py. Those ABCs in turn derive (in a slightly sneaky way) from the corresponding ABCs in collections.abc.

My hunch is that some other test messes with the ABC registration cache and doesn't restore it -- or typing.py's sneaky way of deriving from collections.abc has a subtle bug in it.  Maybe something is calling reload(collections.abc)?

I haven't done any research to confirm or deny this theory.  It shouldn't be too hard to find the (probably only a handful of) tests that mess with the registration cache.

Deferring for beta 4.  We should still fix before release.  Preferably before RC1 depending on the availability of Certain Persons.

Is there any progress on this issue? It's still failing randomly:
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.5/builds/221/steps/test/logs/stdio

======================================================================
FAIL: test_abstractset (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 955, in test_abstractset
    assert isinstance(set(), typing.AbstractSet)
AssertionError

======================================================================
FAIL: test_bytestring (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 979, in test_bytestring
    assert isinstance(b'', typing.ByteString)
AssertionError

======================================================================
FAIL: test_container (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 951, in test_container
    assert isinstance([], typing.Container)
AssertionError

======================================================================
FAIL: test_iterable (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 936, in test_iterable
    assert isinstance([], typing.Iterable)
AssertionError

======================================================================
FAIL: test_iterator (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 942, in test_iterator
    assert isinstance(it, typing.Iterator)
AssertionError

======================================================================
FAIL: test_mapping (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 963, in test_mapping
    assert isinstance({}, typing.Mapping)
AssertionError

======================================================================
FAIL: test_mutablemapping (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 967, in test_mutablemapping
    assert isinstance({}, typing.MutableMapping)
AssertionError

======================================================================
FAIL: test_mutablesequence (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 975, in test_mutablesequence
    assert isinstance([], typing.MutableSequence)
AssertionError

======================================================================
FAIL: test_mutableset (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 959, in test_mutableset
    assert isinstance(set(), typing.MutableSet)
AssertionError

======================================================================
FAIL: test_sequence (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 971, in test_sequence
    assert isinstance([], typing.Sequence)
AssertionError

----------------------------------------------------------------------

Anything happening with this?  We tag 3.5.0rc3 in about 36 hours.

> Anything happening with this?  We tag 3.5.0rc3 in about 36 hours.

I believe that it's a bug in test_typing, not in the typing module. So it must not block the release.

Ok, I found a short script to reproduce the bug:
--
import typing:
assert isinstance([], typing.Container)
assert isinstance([], typing.Container)
--

The first check calls typing.GenericMeta.__subclasscheck__(list) which calls abc.ABCMeta.__subclasscheck__(typing.Container, list). The problem is that not only abc.ABCMeta.__subclasscheck__() returns False, but it adds also list to typing.Container._abc_negative_cache.

The next check returns False without calling typing.GenericMeta.__subclasscheck__(list), in fact ABCMeta.__instancecheck__(typing.GenericMeta, list) returns immediatly False because list is in typing.Container._abc_negative_cache.

The problem: GenericMeta calls ABCMeta.__subclasscheck__() which returns False, but ABCMeta.__subclasscheck__() also stores the result "False" in cache, whereas it's not the expected result.

Should we override __instancecheck__ in GenericMeta?

@Guido: are you interested to fix this complex metaclass/type issue? I don't have time to fix it right now.

Sorry, this is a bug in typing.py.  https://github.com/ambv/typehinting/issues/155

It's a subtle bug but the first reporter has done some good research; I will try to find time to fix it today.

> It's a subtle bug but the first reporter has done some good research; I will try to find time to fix it today.

Oh, it's already known and investigated by someone else. Great. I
didn't know this github project.

New changeset d1f41c614e62 by Guido van Rossum in branch '3.5':
Issue #24635: Fixed flakiness in test_typing.py.
https://hg.python.org/cpython/rev/d1f41c614e62

Fixed now on the 3.5 branch. I still have to merge to default and create the pull request for Larry.

Pull request for Larry: https://bitbucket.org/larry/cpython350/pull-requests/14/fix-issue-24635/diff

BTW, I screwed up and committed this into the public 3.5 repo first (and merged from there into default). I guess once Larry has merged the fix into his special closed 3.5 bitbucket repo I should do a null merge from there back to public 3.5 and forward the null merge to 3.6 as well.

Assigning to Larry since the next step is his.

Pull request accepted.  Please forward merge, thanks!

And, yes, this will be a null merge because you already separately committed it to 3.5.

New changeset 0f37918440c9 by Guido van Rossum in branch 'default':
Issue #24635: Fixed flakiness in test_typing.py. (Merge from 3.5.)
https://hg.python.org/cpython/rev/0f37918440c9

New changeset 438dde69871d by Guido van Rossum in branch '3.5':
Fix issue #24635.
https://hg.python.org/cpython/rev/438dde69871d

OK, done.