classification
Title: 2to3 Grammar pickles not created when upgrading to 3.3.0b2
Type: Stage: resolved
Components: 2to3 (2.x to 3.x conversion tool), Build Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, asvetlov, benjamin.peterson, hynek, lregebro, ned.deily, python-dev, ronaldoussoren, stefanholek, terry.reedy
Priority: normal Keywords: needs review, patch

Created on 2012-08-14 08:16 by stefanholek, last changed 2012-09-09 02:48 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
issue15645.txt ronaldoussoren, 2012-08-20 07:07 review
Messages (19)
msg168180 - (view) Author: Stefan Holek (stefanholek) Date: 2012-08-14 08:16
After upgrading 3.3.0b1 to b2 (make install to same location) I noticed that the Grammar pickles for beta.2 were missing (the beta.1 pickles still being there).

The result is permission errors and/or virtualenv SandboxViolations, when 2to3 is used for the first time. E.g.:

SandboxViolation: open('/usr/local/python3.3/lib/python3.3/lib2to3/Grammar3.3.0.beta.2.pickle', 'wb') {}

This is because users of Python (as opposed to root) may of course not write to the library dir.

The pickles are created fine on a fresh install of Python 3.3.0b2 (make install to new location).
msg168465 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-08-17 18:42
Your OS and distribution/version might be relevant information.
msg168480 - (view) Author: Stefan Holek (stefanholek) Date: 2012-08-17 21:21
Mac OS X 10.6.8 (Snow Leopard)
Xcode Tools 3.2.6

./configure --prefix=/usr/local/python3.3
make
sudo make install
msg168483 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-08-17 22:45
I believe there are known problems with 3.3 and mac, so I added the mac experts.
msg168484 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-17 22:59
I took a quick look at the issue and I could reproduce by doing a "make install" of 3.3.0b1 then a build and "make install" of 3.3.0b2 in the same locations.  Doing a clean 3.3.0b2 build and install produce the expected results.  It appears to be a unix Makefile issue that shouldn't be unique to OS X.  I don't have time to look further into it right at the moment, though.  (And I'm not sure what "known problems with 3.3 and mac" means; the buildbots are happy.)
msg168629 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-08-20 07:07
I'm pretty sure this is a generic Makefile bug.  I could reproduce this problem with a clean install into a new prefix, after cleaning up the source tree: 'make clean' won't remove the generated file when srcdir != builddir.

This line in the Makefile seems to generate the pickle file:

        -PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
                $(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram,lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"

This uses $(PYTHON_FOR_BUILD) to generate the cache file, and that which means the cache file is generated in the *source* tree, not in the installation tree.  This command is at the end of the libinstall target, which means the generatd file is not copied into the installation prefix.

Running 'make install' twice will therefore install the file.

The attached patch seems to fix the issue by moving the command to the start of the libinstall target. The patch also fixes "make clean", by explicitly targetting the pickle files in $(srcdir)/Lib/lib2to3.


BTW. What "known OSX problems"?

BTW2. This should affect all platforms that use Makefiles (basicly anything but Windows)
msg168677 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-08-20 16:24
I had the impression that there *were* problems with compiling 3.3 on the new OSX and Xcode or something. If that is wrong or all fixed, great. Sorry for the noise.
msg168678 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-08-20 16:31
There were some problems building using llvm-gcc, those have been resolved by detecting falling back to clang when llvm-gcc is detected. That's the only real problem I know of.
msg168679 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-08-20 16:33
BTW. I wouldn't mind a patch review, particularly by someone that can easily test on Linux. I'm not too keen on checking in this patch without a test on Linux, not this close to 3.3rc1.
msg168703 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-08-20 21:37
If necessary, ask on pydev.
msg168737 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-21 05:00
Patch LGTM and I verified that it does indeed install the pickles properly on both a vanilla Linux build/install and with an OS X framework build/install. (Note that, as expected, previously installed versioned pickles in the install library directory are not removed. They will harmlessly accumulate until you do a clean install or manually remove them.)
msg168858 - (view) Author: Lennart Regebro (lregebro) Date: 2012-08-22 08:29
I can confirm that this is an issue on Ubuntu 12.04 as well, and that removing the existing pre 3.3b2 install before installing solves the problem.
msg168924 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-23 06:36
New changeset 787ed9b03ef9 by Ned Deily in branch '2.7':
Issue #15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/787ed9b03ef9

New changeset a377a4298b4e by Ned Deily in branch '3.2':
Issue #15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/a377a4298b4e

New changeset b48cd7045909 by Ned Deily in branch 'default':
issue #15645: null merge
http://hg.python.org/cpython/rev/b48cd7045909

New changeset 995e58439b59 by Ned Deily in branch 'default':
Issue #15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/995e58439b59
msg168926 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-23 06:41
Fixes applied for release in 2.7.4, 3.2.4, and 3.3.0.
msg169678 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2012-09-01 23:04
libinstall target now contains:

	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
		$(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"
	...
	@for d in $(LIBSUBDIRS); \
	do \
		a=$(srcdir)/Lib/$$d; \
		if test ! -d $$a; then continue; else true; fi; \
		if test `ls $$a | wc -l` -lt 1; then continue; fi; \
		b=$(LIBDEST)/$$d; \
		for i in $$a/*; \

There is a small chance that with high parallelization the wildcard expansion in the last line will occur for $$a == lib2to3 before creation of pickles has finished, resulting in not installation of pickles.
The solution would be to move creation of pickles to a separate target and make it a dependency of libinstall target.
msg169679 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2012-09-01 23:07
I meant $$d == lib2to3 and $$a == $(srcdir)/Lib/lib2to3
msg169817 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-09-04 07:59
Is it really possible that the pickle is created after calculation of the wildcard expansion? The GNU make manual says that make will run multiple recipes in parallel when the -j option is used (that is, multiple targets are build at the same time), not that the commands in a recipe are run at the same time. That latter would break a lot of assumptions.
msg170025 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-09-08 05:34
See also Issue15838
msg170074 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-09-09 02:48
>Is it really possible that the pickle is created after calculation of the wildcard expansion?

With the current GNUmake, it seems it is not possible. GNUmake appears to serializes the individual steps in a target's recipe although I did not find an explicit guarantee in its documentation that that could never happen.  (I did find at least one proprietary "make" replacement that does try to parallelize the individual steps in one target.)

In any case, there were other problems with the approach implemented here and these changes have been superseded by those of Issue15822.
History
Date User Action Args
2012-09-09 02:48:16ned.deilysetmessages: + msg170074
2012-09-08 05:34:20ronaldoussorensetmessages: + msg170025
2012-09-04 07:59:57ronaldoussorensetmessages: + msg169817
2012-09-01 23:07:55Arfreversetmessages: + msg169679
2012-09-01 23:04:10Arfreversetmessages: + msg169678
2012-08-23 06:41:25ned.deilysetstatus: open -> closed
versions: + Python 2.7, Python 3.2
type: compile error ->
messages: + msg168926

resolution: fixed
stage: commit review -> resolved
2012-08-23 06:36:54python-devsetnosy: + python-dev
messages: + msg168924
2012-08-22 08:29:06lregebrosetnosy: + lregebro
messages: + msg168858
2012-08-21 05:00:39ned.deilysetmessages: + msg168737
stage: patch review -> commit review
2012-08-20 21:37:36terry.reedysetmessages: + msg168703
2012-08-20 16:33:12ronaldoussorensetmessages: + msg168679
2012-08-20 16:31:11ronaldoussorensetmessages: + msg168678
2012-08-20 16:24:01terry.reedysetmessages: + msg168677
2012-08-20 07:07:12ronaldoussorensetfiles: + issue15645.txt
messages: + msg168629

components: + Build
keywords: + patch, needs review
type: compile error
stage: patch review
2012-08-18 13:36:51Arfreversetnosy: + Arfrever
2012-08-17 22:59:31ned.deilysetmessages: + msg168484
2012-08-17 22:45:57terry.reedysetnosy: + ronaldoussoren, ned.deily, hynek
messages: + msg168483
2012-08-17 21:21:12stefanholeksetmessages: + msg168480
2012-08-17 18:42:38terry.reedysetnosy: + terry.reedy
messages: + msg168465
2012-08-17 16:36:00asvetlovsetnosy: + asvetlov
2012-08-14 11:54:13r.david.murraysetnosy: + benjamin.peterson
2012-08-14 08:16:19stefanholekcreate