classification
Title: Decoding error due to byte-compiling venv scripts at install time
Type: behavior Stage: resolved
Components: Installation Versions: Python 3.3, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.araujo, python-dev, vinay.sajip, zart
Priority: normal Keywords:

Created on 2012-10-27 18:42 by zart, last changed 2012-10-28 12:42 by python-dev. This issue is now closed.

Messages (9)
msg173955 - (view) Author: Konstantin Zemlyak (zart) Date: 2012-10-27 18:42
When installing python 3.3 under windows and checking "Compile .py files to byte code after installation" Lib/venv/scripts/nt/pydoc.py gets precompiled as well. This causes venv module to abort with "Error: 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte" while copying scripts from Lib/venv/scripts/nt since only .exe files are copied verbatim, and rest are treated as utf-8 text: http://hg.python.org/cpython/file/b20e473157b8/Lib/venv/__init__.py#l314

The solution is to add another exception to compileargs at Tools/msi/msi.py: http://hg.python.org/cpython/file/b20e473157b8/Tools/msi/msi.py#l419

This issue doesn't affect non-windows systems, since Lib/venv/scripts/posix doesn't contain .py files.
msg173956 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-10-27 18:50
Which file exactly causes the error?  If it’s pydoc.py, it’s strange because the source is ASCII.  Are you installing in a path with non-ASCII characters?
msg173957 - (view) Author: Konstantin Zemlyak (zart) Date: 2012-10-27 18:56
If option to precompile python files is checked in installer, pydoc.py gets compiled into binary files Lib\venv\scripts\nt\__pycache__\pydoc.cpython33.pyc and Lib\venv\scripts\nt\__pycache__\pydoc.cpython33.pyo.
msg173960 - (view) Author: Konstantin Zemlyak (zart) Date: 2012-10-27 19:01
Also current title is a bit wrong, since decoding error happens in runtime each time venv is invoked, while the source of the problem happens while byte-compiling stdlib at install time once.
msg173961 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-10-27 19:06
Are you installing in a path with non-ASCII characters?
msg173963 - (view) Author: Konstantin Zemlyak (zart) Date: 2012-10-27 19:09
Not at all. I have installed Python into "C:\Program Files (x86)\Python33". Also made a copy into "D:\Python33" and got the same results.

The problem is in file contents (pyc/pyo files are binary, utf-8 fails to decode them), not in filenames.
msg174015 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-10-28 09:12
New changeset 06a5fe86d87c by Vinay Sajip in branch '3.3':
Issue #16340: exclude venv/scripts from byte-compilation at installation time on Windows.
http://hg.python.org/cpython/rev/06a5fe86d87c

New changeset e8619f015ab9 by Vinay Sajip in branch 'default':
Issue #16340: Merged fix from 3.3.
http://hg.python.org/cpython/rev/e8619f015ab9
msg174016 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-28 09:16
I've made the msi.py change. Perhaps I should also catch the exception in the venv script-copying code? Leaving issue open for now.
msg174040 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-10-28 12:42
New changeset adefc83c1128 by Vinay Sajip in branch '3.3':
Closes #16340: Handle exception while copying script to venv.
http://hg.python.org/cpython/rev/adefc83c1128

New changeset fb969187ecc2 by Vinay Sajip in branch 'default':
Closes #16340: Merged fix from 3.3.
http://hg.python.org/cpython/rev/fb969187ecc2
History
Date User Action Args
2012-10-28 12:42:43python-devsetstatus: open -> closed
resolution: fixed
messages: + msg174040

stage: resolved
2012-10-28 09:16:39vinay.sajipsetmessages: + msg174016
2012-10-28 09:12:02python-devsetnosy: + python-dev
messages: + msg174015
2012-10-28 07:39:08zartsettitle: Decoding error at install time when byte-compiling venv scripts -> Decoding error due to byte-compiling venv scripts at install time
2012-10-27 19:09:34zartsetmessages: + msg173963
2012-10-27 19:06:49eric.araujosetmessages: + msg173961
title: Decoding error in venv when byte-compiling stdlib -> Decoding error at install time when byte-compiling venv scripts
2012-10-27 19:01:06zartsetmessages: + msg173960
2012-10-27 18:56:10zartsetmessages: + msg173957
2012-10-27 18:50:13eric.araujosetnosy: + vinay.sajip, eric.araujo
title: Error: 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte -> Decoding error in venv when byte-compiling stdlib
messages: + msg173956

versions: + Python 3.4
2012-10-27 18:42:48zartcreate