classification
Title: make install tries to create lib2to3 grammar pickles in source directory
Type: Stage: resolved
Components: 2to3 (2.x to 3.x conversion tool), Build, Installation Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles
View: 15822
Assigned To: Nosy List: Arfrever, benjamin.peterson, ned.deily, ronaldoussoren, trent
Priority: normal Keywords:

Created on 2012-09-01 11:26 by ned.deily, last changed 2012-09-09 02:38 by ned.deily. This issue is now closed.

Messages (3)
msg169623 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-09-01 11:26
With the recent activity around making out-of-tree builds work, while investigating Issue15822 I realized that the lib2to3 grammar pickle files are being built in the source directory by the libinstall target step of the main Makefile rather than in the build directory or just in the install location.  The fixes for Issue15645 moved the pickle building to the start of the target recipes:

libinstall:	build_all $(srcdir)/Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
		$(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"


On reflection, I think the PYTHONPATH definition is bogus here since LIBDEST is populated immediately after this step. More importantly, lib2to3/pgen2/driver:load_grammar ends up writing the pickle files into the source directory.  Unlike some other cases in the Makefile, if the source directory is read-only, the failure to create the pickle files is not an error during installation but it may likely cause problems for later users of the installed lib2to3 (like in the Issue15822 scenario), i.e. when the missing pickles are attempted to be created in the read-only install directories.
msg169818 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-09-04 08:11
FWIW the patch is issue15645 was a patch that fixed the issue with minimal changes, as it was created close to the first 3.3 release candidate I didn't want to change code when that could be avoided.

A fix that writes the generated file into the build tree instead of the source tree requires changes to lib2to3.pgen.driver:load_grammar: it assumes it can derive the path of the pickle file from the path of the grammar file, which wouldn't be true when storing the pickle file in the build directory and the build directory is not the same as the source directory.

Now that I look at this code again: the patch is issue15645 was probably make necessary by the introduction of $(PYTHON_FOR_BUILD), that expands to './$(BUILD_PYTHON) -E' on my machine, and that means the PYTHONPATH setting is ignored which is why the file is created in the source directory instead of the installation directory.

This means it could be fairly easy to properly fix this issue:

- Move the line back to the end of the libinstall recipe
- Add "import sys, os; sys.path.insert(0, os.environ['PYTHONPATH']"
  to the start of the python scriptlet (before "import lib2to3....")

I haven't tested yet if that actually works though.
msg170073 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-09-09 02:38
This problem is resolved by the fixes for Issue15822 which implements a more explicit way of installing the grammar pickles.
History
Date User Action Args
2012-09-09 02:38:25ned.deilysetstatus: open -> closed
superseder: installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles
messages: + msg170073

resolution: duplicate
stage: needs patch -> resolved
2012-09-04 08:11:54ronaldoussorensetmessages: + msg169818
2012-09-04 07:49:32Arfreversetnosy: + Arfrever
2012-09-01 11:26:44ned.deilycreate