classification
Title: installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles
Type: behavior Stage: committed/rejected
Components: 2to3 (2.x to 3.x conversion tool), Installation Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, benjamin.peterson, georg.brandl, loewis, ned.deily, python-dev, ronaldoussoren, tpievila
Priority: high Keywords: patch

Created on 2012-08-30 10:05 by tpievila, last changed 2012-09-09 09:18 by python-dev. This issue is now closed.

Messages (20)
msg169445 - (view) Author: Tomi Pieviläinen (tpievila) Date: 2012-08-30 10:05
Python 3.3rc1 source install modifies/installs lib2to3/Grammar.txt after modifying Grammar3.3.0.candidate.1.pickle:

-rw-r--r--. 1 root root  6589 2012-08-27 13:25:33.190414570 +0300 Grammar.txt
-rw-r--r--. 1 root root 20622 2012-08-27 13:25:33.150477192 +0300 Grammar3.3.0.candidate.1.pickle

This causes lib2to3/pgen2/driver:load_grammar to try to recreate the pickle (the _newer call on line 119 returns false), which triggers SandboxViolation in when using easy_install/distribute (https://bitbucket.org/tarek/distribute/issue/317/easy_install-fails-with-sandboxviolation) see .
msg169495 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-31 01:03
Can you give the steps to reproduce this issue?  Also, what platform and file system are in use?  I can guess what an underlying problem might be but I haven't been able to reproduce the failure myself.
msg169502 - (view) Author: Tomi Pieviläinen (tpievila) Date: 2012-08-31 06:41
Platform: RHEL 6.3 x86_64.

0. (remove old python 3 install, though this doesn't change anything)
1. Unpack 3.3rc1 source, cd Python-3.3.0rc1/
2. ./configure; make
3. sudo make install
4. ls -ltc --full-time /usr/local/lib/python3.3/lib2to3/

This shows that Grammar.txt is just a tiny bit newer than the pickle.

To produce the SandboxViolation:

1. curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py
2. python3.3 virtualenv.py my_new_env

or alternatively use distribute to install anything with the easy_install command.
msg169503 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2012-08-31 06:43
This issue can be easily reproduced on filesystems with nanosecond precision (e.g. ext4).
msg169504 - (view) Author: Tomi Pieviläinen (tpievila) Date: 2012-08-31 06:46
Ah, sorry, forgot to add that info. I also am running ext4.
msg169505 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2012-08-31 06:46
`make install DESTDIR=/some/temporary/path` can be used to avoid affecting installed version of Python.
msg169509 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-31 08:41
Thanks for the details. Moving to ext4, I was still not seeing the problem as Grammar.txt and the pickle file have the same nanosecond ctime but that may be due to running in a vm environment or other factors.  Manually touching Grammar.txt, not surprisingly, does cause the reported behavior so that's sufficient for testing purposes.
msg169619 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-09-01 11:02
The problem here is in the "libinstall" Makefile target and that hidden dependency within lib2to3 that, as Tomi notes, causes lib2to3 to try to recreate the pickles anytime it runs if it thinks they are out of date.  The fixes for Issue15645 cause the recipes for that target to create the two grammar pickles in the source Lib directory before files are copied and installed into the destination lib directories. But there is no guarantee that the source files, Grammar.txt and PatternGrammar.txt, are installed into the destination lib before the corresponding pickle files are.  lib2to3/pgen2/driver compares the value of os.path.getmtime() for the txt and pickle files (assuming both exist).  On many (most) file systems, the copies happen fast enough and the resolution of getmtime is coarse enough that the files are created with the same time value so post-installation use of lib2to3 works OK. But on file systems like ext4 with high-resolution time values, it may happen that one or both of the pickle files is copied first and ends up with a slightly older time value than its corresponding txt file.  In that case the first use of lib2to3 will try to recreate the pickle which will most likely fail with a file permission error since the install directories are typically read-only to non-root users.

The attached patch tries to work around around the problem by touching the two pickle files after they and all the other relevant lib files have been installed.

Since any user could run into this problem on the right configuration, I think the fix (assuming reviewers concur with it) should be cherry-picked for 3.3.0rc2.
msg169672 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2012-09-01 22:31
> -	done
> +	done; \

Not needed.

> +	touch $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle || true

"-" prefix can be used to ignore failure of command:

+	-touch $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle

However maybe failure of touch should not be ignored here.
msg169674 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-09-01 22:34
I did it that way to ensure that the touch is executed after the copies.  And the touch needs to be optional because the creation of the pickles can optionally fail.
msg169676 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2012-09-01 22:51
> And the touch needs to be optional because the creation of the pickles can optionally fail.

If creation of pickles failed, then touch will create 1 file literally named "*Grammar*.pickle".

	...
	done; \
	for i in $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle; do \
		if test -f "$$i"; then \
			touch "$$i"; \
		fi; \
	done
msg169677 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-09-01 22:57
Right you are, thanks!  I had that in an earlier version.  Are you able to verify the fix?
msg169999 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-07 17:46
I think the patch for issue 15645 was incorrect. Instead of generating the pickles in the source tree and copying them, it should have arranged lib2to3 to generate them in the target directory instead (just as all the compileall invocations also generate files in LIBDEST).

There are multiple ways to achieve this. The least intrusive (perhaps) is to insert "$(DESTDIR)$(LIBDEST)" into sys.path before importing lib2to3 in the Makefile.

The approach that I suggest is to make lib2to3.pgen2.driver a script which expects a .txt path, and generates a pickle next to it, so the installation would do

$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/Grammar.txt
$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/PatternGrammar.txt
msg170024 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-09-08 05:33
See als Issue15838.
msg170026 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-09-08 05:42
The other alternative to fix this is to revert the patch in issue 15645  and then remove the -E flag from PYTHON_FOR_BUILD.

PYTHON_FOR_BUILD was introduced by a, possibly incomplete, attempt to introduce support for cross compiling. In particular, the variable was introduced in issue 14330. Neither the patch, nor the tracker discussion explains why the -E is necessary. It might be because all uses of $(BUILDPYTHON) that were replaced by PYTHON_FOR_BUILD had the -E flag at the time.

BTW. There also doesn't seem to documentation for cross building.
msg170047 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-08 12:36
I think PYTHON_FOR_BUILD includes -E to make sure that if PYTHONPATH (or PYTHONHOME) is set that it then won't affect the build.
msg170071 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-09 02:08
New changeset 2def9ef683da by Ned Deily in branch '2.7':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/2def9ef683da

New changeset adc5e6aca251 by Ned Deily in branch '3.2':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/adc5e6aca251

New changeset a829889f96b0 by Ned Deily in branch 'default':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/a829889f96b0
msg170072 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-09-09 02:32
I like Martin's suggested approach.  It is explicit and much cleaner than the changes we introduced for Issue15645 as well as the original.  The applied changes revert the previous changes and instead implement an explicit build of the two pickles directly in the installed lib2to3 during "make install" regardless of the presence of pickles in the build directory.  This also resolves Issue15838.

Georg, I think a829889f96b0 should be considered for inclusion in 3.3.0.

Ronald, if you want to pursue the $PYTHON_FOR_BUILD -E issue, it would be best to use another specific issue for that.
msg170080 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-09-09 06:57
Picked to release clone as 411d500e56a1.
msg170088 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-09 09:18
New changeset 411d500e56a1 by Ned Deily in branch 'default':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/411d500e56a1
History
Date User Action Args
2012-09-09 09:18:55python-devsetmessages: + msg170088
2012-09-09 06:59:54ned.deilysetstatus: open -> closed
2012-09-09 06:57:06georg.brandlsetpriority: release blocker -> high

messages: + msg170080
2012-09-09 02:38:25ned.deilylinkissue15838 superseder
2012-09-09 02:32:51ned.deilysetfiles: - issue15822_33.patch
status: pending -> open
2012-09-09 02:32:18ned.deilysetstatus: open -> pending
resolution: fixed
messages: + msg170072

stage: patch review -> committed/rejected
2012-09-09 02:08:54python-devsetnosy: + python-dev
messages: + msg170071
2012-09-08 12:36:34loewissetmessages: + msg170047
2012-09-08 05:42:05ronaldoussorensetmessages: + msg170026
2012-09-08 05:33:53ronaldoussorensetmessages: + msg170024
2012-09-07 17:46:18loewissetnosy: + loewis
messages: + msg169999
2012-09-01 22:57:14ned.deilysetmessages: + msg169677
2012-09-01 22:51:53Arfreversetmessages: + msg169676
2012-09-01 22:34:08ned.deilysetmessages: + msg169674
2012-09-01 22:31:46Arfreversetmessages: + msg169672
2012-09-01 11:02:46ned.deilysetfiles: + issue15822_33.patch


keywords: + patch
stage: patch review
title: Python 3.3 creates lib2to3 grammar in wrong order -> installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles
nosy: + ronaldoussoren
versions: + Python 2.7, Python 3.2
messages: + msg169619
priority: high -> release blocker
components: + Installation
2012-08-31 08:41:25ned.deilysetmessages: + msg169509
2012-08-31 06:46:37Arfreversetmessages: + msg169505
2012-08-31 06:46:30tpievilasetmessages: + msg169504
2012-08-31 06:43:48Arfreversetmessages: + msg169503
2012-08-31 06:41:45tpievilasetmessages: + msg169502
2012-08-31 01:03:53ned.deilysetnosy: + ned.deily
messages: + msg169495
2012-08-30 16:21:50Arfreversetnosy: + Arfrever
2012-08-30 13:35:27pitrousetpriority: normal -> high
nosy: + georg.brandl, benjamin.peterson
2012-08-30 10:05:26tpievilacreate