Issue #24975 shows that we can easily end up with holes in support from the ast module. We should probably make sure we have a test that goes through the entire stdlib, Does an ast.parse(), and then passes the result to compile() to verify no exceptions are raised. We should either avoid the test/ directory or blacklist specific files which will fail because they are designed to be syntactically incorrect. The test can also be behind a -u flag if it turns out to be an expensive test.

+1 This is a great idea.

The attached recurses one level deep in Lib. Catching UnicodeDecodeError on file read is needed at least for "test/test_source_encoding.py 'utf-8' codec can't decode byte 0xf0 in position 267: invalid continuation byte".  Catching "badxxx.py" is needed for several test/ modules.

This test may have found another hole:  It prints

 "C:/programs/Python35/Lib\distutils _msvccompiler.py None disallowed in expression list"

whereas

>>> compile(open('C:/programs/Python35/Lib\distutils\_msvccompiler.py').read(), '', 'exec')
<code object <module> at 0x00000000034A3F60, file "", line 8>

Forgot to mention: runtime 4 seconds.

Ran with installed 3.5.0rc1.

Two things about the UTF-8 decode issue. One is you don't need to decode the source for it to be compiled; ast.parse()/compile() will work from bytes objects and thus handle the decoded for you. Two, if you want to decode source files into text, use importlib.util.decode_source().

Since Terry's script shows a complete compilation of the stdlib is very fast, we should make this a proper test. Any resulting test should probably should go into test_compile(). We can have it read all files and those that raise a a SyntaxError or some such exception can simply be skipped as the test is about AST -> code and not source -> AST (although we could theoretically have the test validate the failure with a source -> code compilation as well and verify the same exception is raised). We can also double-check that any AST -> code failure passes under source -> code and then report a failure (basically I'm trying to avoid blacklisting files that are test cases for the compiler overall and are known to fail).

The distutils error is a ValueError, which was printed as such before I added 'Exception' to the except statement.

I did not initially catch SyntaxError separately because some are caught during the compile phase. Revised file corrects 'pass' to 'continue' and continues with parse errors also.

Many test/badsyntax files have compile-time syntax errors.

SyntaxError 'await' outside function (badsyntax_async1.py, line 1)
SyntaxError 'yield' inside async function (badsyntax_async6.py, line 2)
SyntaxError from __future__ imports must occur at the beginning of the file (badsyntax_future10.py, line 3)
SyntaxError future feature rested_snopes is not defined (badsyntax_future3.py, line 3)
SyntaxError not a chance (badsyntax_future9.py, line 3)

Attached is a fully recursive version that only report compile errors for non-'badsyntax' files.  It takes about 2 seconds more. Output:

C:/programs/Python35/Lib\distutils _msvccompiler.py ValueError None disallowed in expression list
1776 (files compiled)

I excluded site-packages after testing on 3.4 because pylint has bad syntax examples giving a page of error messages.

I think there is something like this here: 

https://github.com/python/cpython/blob/22fe0eb13c3441b71b60aaea0e7fe289a29783da/Lib/test/test_ast.py#L1498