Python 3.3rc1 source install modifies/installs lib2to3/Grammar.txt after modifying Grammar3.3.0.candidate.1.pickle:

-rw-r--r--. 1 root root  6589 2012-08-27 13:25:33.190414570 +0300 Grammar.txt
-rw-r--r--. 1 root root 20622 2012-08-27 13:25:33.150477192 +0300 Grammar3.3.0.candidate.1.pickle

This causes lib2to3/pgen2/driver:load_grammar to try to recreate the pickle (the _newer call on line 119 returns false), which triggers SandboxViolation in when using easy_install/distribute (https://bitbucket.org/tarek/distribute/issue/317/easy_install-fails-with-sandboxviolation) see .

Can you give the steps to reproduce this issue?  Also, what platform and file system are in use?  I can guess what an underlying problem might be but I haven't been able to reproduce the failure myself.

Platform: RHEL 6.3 x86_64.

0. (remove old python 3 install, though this doesn't change anything)
1. Unpack 3.3rc1 source, cd Python-3.3.0rc1/
2. ./configure; make
3. sudo make install
4. ls -ltc --full-time /usr/local/lib/python3.3/lib2to3/

This shows that Grammar.txt is just a tiny bit newer than the pickle.

To produce the SandboxViolation:

1. curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py
2. python3.3 virtualenv.py my_new_env

or alternatively use distribute to install anything with the easy_install command.

This issue can be easily reproduced on filesystems with nanosecond precision (e.g. ext4).

Ah, sorry, forgot to add that info. I also am running ext4.

`make install DESTDIR=/some/temporary/path` can be used to avoid affecting installed version of Python.

Thanks for the details. Moving to ext4, I was still not seeing the problem as Grammar.txt and the pickle file have the same nanosecond ctime but that may be due to running in a vm environment or other factors.  Manually touching Grammar.txt, not surprisingly, does cause the reported behavior so that's sufficient for testing purposes.

The problem here is in the "libinstall" Makefile target and that hidden dependency within lib2to3 that, as Tomi notes, causes lib2to3 to try to recreate the pickles anytime it runs if it thinks they are out of date.  The fixes for Issue15645 cause the recipes for that target to create the two grammar pickles in the source Lib directory before files are copied and installed into the destination lib directories. But there is no guarantee that the source files, Grammar.txt and PatternGrammar.txt, are installed into the destination lib before the corresponding pickle files are.  lib2to3/pgen2/driver compares the value of os.path.getmtime() for the txt and pickle files (assuming both exist).  On many (most) file systems, the copies happen fast enough and the resolution of getmtime is coarse enough that the files are created with the same time value so post-installation use of lib2to3 works OK. But on file systems like ext4 with high-resolution time values, it may happen that one or both of the pickle files is copied first and ends up with a slightly older time value than its corresponding txt file.  In that case the first use of lib2to3 will try to recreate the pickle which will most likely fail with a file permission error since the install directories are typically read-only to non-root users.

The attached patch tries to work around around the problem by touching the two pickle files after they and all the other relevant lib files have been installed.

Since any user could run into this problem on the right configuration, I think the fix (assuming reviewers concur with it) should be cherry-picked for 3.3.0rc2.

> -	done
> +	done; \

Not needed.

> +	touch $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle || true

"-" prefix can be used to ignore failure of command:

+	-touch $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle

However maybe failure of touch should not be ignored here.

I did it that way to ensure that the touch is executed after the copies.  And the touch needs to be optional because the creation of the pickles can optionally fail.

> And the touch needs to be optional because the creation of the pickles can optionally fail.

If creation of pickles failed, then touch will create 1 file literally named "*Grammar*.pickle".

	...
	done; \
	for i in $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle; do \
		if test -f "$$i"; then \
			touch "$$i"; \
		fi; \
	done

Right you are, thanks!  I had that in an earlier version.  Are you able to verify the fix?

I think the patch for issue 15645 was incorrect. Instead of generating the pickles in the source tree and copying them, it should have arranged lib2to3 to generate them in the target directory instead (just as all the compileall invocations also generate files in LIBDEST).

There are multiple ways to achieve this. The least intrusive (perhaps) is to insert "$(DESTDIR)$(LIBDEST)" into sys.path before importing lib2to3 in the Makefile.

The approach that I suggest is to make lib2to3.pgen2.driver a script which expects a .txt path, and generates a pickle next to it, so the installation would do

$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/Grammar.txt
$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/PatternGrammar.txt

See als Issue15838.

The other alternative to fix this is to revert the patch in issue 15645  and then remove the -E flag from PYTHON_FOR_BUILD.

PYTHON_FOR_BUILD was introduced by a, possibly incomplete, attempt to introduce support for cross compiling. In particular, the variable was introduced in issue 14330. Neither the patch, nor the tracker discussion explains why the -E is necessary. It might be because all uses of $(BUILDPYTHON) that were replaced by PYTHON_FOR_BUILD had the -E flag at the time.

BTW. There also doesn't seem to documentation for cross building.

I think PYTHON_FOR_BUILD includes -E to make sure that if PYTHONPATH (or PYTHONHOME) is set that it then won't affect the build.

New changeset 2def9ef683da by Ned Deily in branch '2.7':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/2def9ef683da

New changeset adc5e6aca251 by Ned Deily in branch '3.2':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/adc5e6aca251

New changeset a829889f96b0 by Ned Deily in branch 'default':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/a829889f96b0

I like Martin's suggested approach.  It is explicit and much cleaner than the changes we introduced for Issue15645 as well as the original.  The applied changes revert the previous changes and instead implement an explicit build of the two pickles directly in the installed lib2to3 during "make install" regardless of the presence of pickles in the build directory.  This also resolves Issue15838.

Georg, I think a829889f96b0 should be considered for inclusion in 3.3.0.

Ronald, if you want to pursue the $PYTHON_FOR_BUILD -E issue, it would be best to use another specific issue for that.

Picked to release clone as 411d500e56a1.

New changeset 411d500e56a1 by Ned Deily in branch 'default':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/411d500e56a1

I have similar problem with python 3.7.0 or 3.7.2,
how to fix my installation to get it work?

please help

Thanks