After upgrading 3.3.0b1 to b2 (make install to same location) I noticed that the Grammar pickles for beta.2 were missing (the beta.1 pickles still being there).

The result is permission errors and/or virtualenv SandboxViolations, when 2to3 is used for the first time. E.g.:

SandboxViolation: open('/usr/local/python3.3/lib/python3.3/lib2to3/Grammar3.3.0.beta.2.pickle', 'wb') {}

This is because users of Python (as opposed to root) may of course not write to the library dir.

The pickles are created fine on a fresh install of Python 3.3.0b2 (make install to new location).

Your OS and distribution/version might be relevant information.

Mac OS X 10.6.8 (Snow Leopard)
Xcode Tools 3.2.6

./configure --prefix=/usr/local/python3.3
make
sudo make install

I believe there are known problems with 3.3 and mac, so I added the mac experts.

I took a quick look at the issue and I could reproduce by doing a "make install" of 3.3.0b1 then a build and "make install" of 3.3.0b2 in the same locations.  Doing a clean 3.3.0b2 build and install produce the expected results.  It appears to be a unix Makefile issue that shouldn't be unique to OS X.  I don't have time to look further into it right at the moment, though.  (And I'm not sure what "known problems with 3.3 and mac" means; the buildbots are happy.)

I'm pretty sure this is a generic Makefile bug.  I could reproduce this problem with a clean install into a new prefix, after cleaning up the source tree: 'make clean' won't remove the generated file when srcdir != builddir.

This line in the Makefile seems to generate the pickle file:

        -PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
                $(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram,lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"

This uses $(PYTHON_FOR_BUILD) to generate the cache file, and that which means the cache file is generated in the *source* tree, not in the installation tree.  This command is at the end of the libinstall target, which means the generatd file is not copied into the installation prefix.

Running 'make install' twice will therefore install the file.

The attached patch seems to fix the issue by moving the command to the start of the libinstall target. The patch also fixes "make clean", by explicitly targetting the pickle files in $(srcdir)/Lib/lib2to3.


BTW. What "known OSX problems"?

BTW2. This should affect all platforms that use Makefiles (basicly anything but Windows)

I had the impression that there *were* problems with compiling 3.3 on the new OSX and Xcode or something. If that is wrong or all fixed, great. Sorry for the noise.

There were some problems building using llvm-gcc, those have been resolved by detecting falling back to clang when llvm-gcc is detected. That's the only real problem I know of.

BTW. I wouldn't mind a patch review, particularly by someone that can easily test on Linux. I'm not too keen on checking in this patch without a test on Linux, not this close to 3.3rc1.

If necessary, ask on pydev.

Patch LGTM and I verified that it does indeed install the pickles properly on both a vanilla Linux build/install and with an OS X framework build/install. (Note that, as expected, previously installed versioned pickles in the install library directory are not removed. They will harmlessly accumulate until you do a clean install or manually remove them.)

I can confirm that this is an issue on Ubuntu 12.04 as well, and that removing the existing pre 3.3b2 install before installing solves the problem.

New changeset 787ed9b03ef9 by Ned Deily in branch '2.7':
Issue #15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/787ed9b03ef9

New changeset a377a4298b4e by Ned Deily in branch '3.2':
Issue #15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/a377a4298b4e

New changeset b48cd7045909 by Ned Deily in branch 'default':
issue #15645: null merge
http://hg.python.org/cpython/rev/b48cd7045909

New changeset 995e58439b59 by Ned Deily in branch 'default':
Issue #15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/995e58439b59

Fixes applied for release in 2.7.4, 3.2.4, and 3.3.0.

libinstall target now contains:

	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
		$(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"
	...
	@for d in $(LIBSUBDIRS); \
	do \
		a=$(srcdir)/Lib/$$d; \
		if test ! -d $$a; then continue; else true; fi; \
		if test `ls $$a | wc -l` -lt 1; then continue; fi; \
		b=$(LIBDEST)/$$d; \
		for i in $$a/*; \

There is a small chance that with high parallelization the wildcard expansion in the last line will occur for $$a == lib2to3 before creation of pickles has finished, resulting in not installation of pickles.
The solution would be to move creation of pickles to a separate target and make it a dependency of libinstall target.

I meant $$d == lib2to3 and $$a == $(srcdir)/Lib/lib2to3

Is it really possible that the pickle is created after calculation of the wildcard expansion? The GNU make manual says that make will run multiple recipes in parallel when the -j option is used (that is, multiple targets are build at the same time), not that the commands in a recipe are run at the same time. That latter would break a lot of assumptions.

See also Issue15838

>Is it really possible that the pickle is created after calculation of the wildcard expansion?

With the current GNUmake, it seems it is not possible. GNUmake appears to serializes the individual steps in a target's recipe although I did not find an explicit guarantee in its documentation that that could never happen.  (I did find at least one proprietary "make" replacement that does try to parallelize the individual steps in one target.)

In any case, there were other problems with the approach implemented here and these changes have been superseded by those of Issue15822.