With the recent activity around making out-of-tree builds work, while investigating Issue15822 I realized that the lib2to3 grammar pickle files are being built in the source directory by the libinstall target step of the main Makefile rather than in the build directory or just in the install location.  The fixes for Issue15645 moved the pickle building to the start of the target recipes:

libinstall:	build_all $(srcdir)/Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
		$(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"


On reflection, I think the PYTHONPATH definition is bogus here since LIBDEST is populated immediately after this step. More importantly, lib2to3/pgen2/driver:load_grammar ends up writing the pickle files into the source directory.  Unlike some other cases in the Makefile, if the source directory is read-only, the failure to create the pickle files is not an error during installation but it may likely cause problems for later users of the installed lib2to3 (like in the Issue15822 scenario), i.e. when the missing pickles are attempted to be created in the read-only install directories.

FWIW the patch is issue15645 was a patch that fixed the issue with minimal changes, as it was created close to the first 3.3 release candidate I didn't want to change code when that could be avoided.

A fix that writes the generated file into the build tree instead of the source tree requires changes to lib2to3.pgen.driver:load_grammar: it assumes it can derive the path of the pickle file from the path of the grammar file, which wouldn't be true when storing the pickle file in the build directory and the build directory is not the same as the source directory.

Now that I look at this code again: the patch is issue15645 was probably make necessary by the introduction of $(PYTHON_FOR_BUILD), that expands to './$(BUILD_PYTHON) -E' on my machine, and that means the PYTHONPATH setting is ignored which is why the file is created in the source directory instead of the installation directory.

This means it could be fairly easy to properly fix this issue:

- Move the line back to the end of the libinstall recipe
- Add "import sys, os; sys.path.insert(0, os.environ['PYTHONPATH']"
  to the start of the python scriptlet (before "import lib2to3....")

I haven't tested yet if that actually works though.

This problem is resolved by the fixes for Issue15822 which implements a more explicit way of installing the grammar pickles.