A user reported this error with `make -j${high_value}`:

========================
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/_warnings.o Python/_warnings.c
/bin/mkdir -p Include
python3.3 ./Parser/asdl_c.py -h Include ./Parser/Python.asdl
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/asdl.o Python/asdl.c
make Parser/pgen
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/bltinmodule.o Python/bltinmodule.c
make[1]: Entering directory '/var/tmp/portage/dev-lang/python-3.3.3-r1000/work/Python-3.3.3'
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/dynamic_annotations.o Python/dynamic_annotations.c
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/ceval.o Python/ceval.c
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/pyctype.o Python/pyctype.c
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o Parser/tokenizer_pgen.c
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/codecs.o Python/codecs.c
Python/bltinmodule.c:4:24: warning: Include/Python-ast.h is shorter than expected [enabled by default]
 #include "Python-ast.h"
                        ^
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/errors.o Python/errors.c
In file included from Python/bltinmodule.c:10:0:
Include/ast.h:7:1: warning: parameter names (without types) in function declaration [enabled by default]
 PyAPI_FUNC(int) PyAST_Validate(mod_ty);
 ^
In file included from Include/Python.h:50:0,
                 from Python/bltinmodule.c:3:
Include/ast.h:8:12: error: unknown type name ‘mod_ty’
 PyAPI_FUNC(mod_ty) PyAST_FromNode(
            ^
Include/pyport.h:777:34: note: in definition of macro ‘PyAPI_FUNC’
 #       define PyAPI_FUNC(RTYPE) RTYPE
                                  ^
Python/bltinmodule.c: In function ‘builtin_compile’:
Python/bltinmodule.c:641:13: error: unknown type name ‘mod_ty’
             mod_ty mod;
             ^
Python/bltinmodule.c:647:21: warning: comparison between pointer and integer [enabled by default]
             if (mod == NULL) {
                     ^
Python/bltinmodule.c:656:49: warning: passing argument 1 of ‘PyAST_CompileEx’ makes pointer from integer without a cast [enabled by default]
                                                 &cf, optimize, arena);
                                                 ^
In file included from Include/Python.h:122:0,
                 from Python/bltinmodule.c:3:
Include/compile.h:33:28: note: expected ‘struct _mod *’ but argument is of type ‘int’
 PyAPI_FUNC(PyCodeObject *) PyAST_CompileEx(
                            ^
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Python/frozenmain.o Python/frozenmain.c
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c
Makefile:1362: recipe for target 'Python/bltinmodule.o' failed
make: *** [Python/bltinmodule.o] Error 1
make: *** Waiting for unfinished jobs....
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Parser/parsetok_pgen.o Parser/parsetok_pgen.c
x86_64-pc-linux-gnu-gcc -pthread -c -Wno-unused-result -DNDEBUG  -O2 -march=atom -pipe -fwrapv    -I. -IInclude -I./Include   -fPIC -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c
x86_64-pc-linux-gnu-gcc -pthread -DNDEBUG  -Wl,-O1 -Wl,--sort-common -Wl,--as-needed  Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain.o -lpthread -ldl  -lutil -o Parser/pgen
make[1]: Leaving directory '/tmp/Python-3.3.3'
Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
========================

I attach a patch (for default branch). This bug should be also fixed in 2.7 and 3.3 branches.

I think os.replace() should be used instead os.rename(). Or pair os.unlink()/os.rename().

And perhaps for decreasing chance of race between creating .c and .h files, both renames should be done after writing both files.

'p' variable in one part of code has different value than in other part.

We can save them in a list of created files.

Since the patch doesn't use O_EXCL for the temporary file, it suffers
from the exact same race condition as the current code. It should be
using the "x" open mode.

> Since the patch doesn't use O_EXCL for the temporary file, it suffers
> from the exact same race condition as the current code.

This does not make race condition. Only one thread writes .tmp files.

The problem is that other threads can read *-ast.[ch] files while they are written.

>
>
> > Since the patch doesn't use O_EXCL for the temporary file, it suffers
> > from the exact same race condition as the current code.
>
> This does not make race condition. Only one thread writes .tmp files.
>
> The problem is that other threads can read *-ast.[ch] files while they are
> written.
>

Ah, I thought it was written by several processes. It's OK then.

Actually no, I don't understand this patch: if the makefile was correct, C
files depending on Include/Python-ast.h and Python/Python-ast.c shouldn't
be built before those files have been generated by asdl_c.py.
The problem doesn't lie in the non-atomicity, but in the make file
dependency graph.

Here's a patch.

LGTM.

New changeset 874813a3523d by Charles-François Natali in branch '2.7':
Issue #19965: Make sure that Python-ast.h is properly taken into account in the
http://hg.python.org/cpython/rev/874813a3523d

New changeset cfe0a293551f by Charles-François Natali in branch '3.3':
Issue #19965: Make sure that Python-ast.h is properly taken into account in the
http://hg.python.org/cpython/rev/cfe0a293551f

New changeset ad42ea70668e by Charles-François Natali in branch 'default':
Issue #19965: Make sure that Python-ast.h is properly taken into account in the
http://hg.python.org/cpython/rev/ad42ea70668e