classification
Title: Windows install error when choosing to compile .py files
Type: behavior Stage: needs patch
Components: Installation, Windows Versions: Python 3.1, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brian.curtin, ggenellina, loewis, pds
Priority: high Keywords:

Created on 2009-08-17 13:05 by pds, last changed 2010-03-27 00:29 by ggenellina. This issue is now closed.

Messages (13)
msg91662 - (view) Author: (pds) Date: 2009-08-17 13:05
There seem to be 3 problems in Python 3.1.1rc1 Windows installer package.

1. Command line argument of compileall.py seems wrong.
2. UnicodeEncodeError occurs depending on code page.
3. Syntax errors.

First, I tried to install Python 3.1.1rc1 just by double-clicking the
Windows msi installer package file, python-3.1.1rc1.msi, as an
administrator account, with the following environment and settings.

Operating System version: Windows XP Professional SP3 (Japanese version)
Install options: Install for all users
Destination directory: C:\Python31
Advanced options: Enable "Compile .py files to byte code after installation"

Then the following dialog message appeared during installation.

There is a problem with this Windows Installer package. A program run as
part of the setup did not finish as expected. Contact your support
personnel or package vendor.

Despite the message, installation of Python interpreter seemed completed
because the programs were registered in Windows start menu.
So I uninstalled Python 3.1.1rc1 to make sure the system to be clean,
and retried installation with the following command from command prompt
so I could see the log file later.

msiexec /i python-3.1.1rc1.msi /L*v python-3.1.1rc1.log

Installation failed again, and the following is the part of the log file
(python-3.1.1rc1.log).

MSI (s) (18:50) [15:08:25:096]: Note: 1: 1722 2: CompilePyc 3:
C:\Python31\python.exe 4: -Wi "C:\Python31\Lib\compileall.py" -f -x
bad_coding|badsyntax|site-packages|py2_ "C:\Python31\Lib" 
MSI (s) (18:50) [15:08:25:096]: Note: 1: 2262 2: Error 3: -2147287038 
Error 1722. There is a problem with this Windows Installer package. A
program run as part of the setup did not finish as expected. Contact
your support personnel or package vendor.  Action CompilePyc, location:
C:\Python31\python.exe, command: -Wi "C:\Python31\Lib\compileall.py" -f
-x bad_coding|badsyntax|site-packages|py2_ "C:\Python31\Lib" 
MSI (s) (18:50) [15:10:58:677]: Note: 1: 2262 2: Error 3: -2147287038 
MSI (s) (18:50) [15:10:58:677]: Product: Python 3.1.1rc1 -- Error 1722.
There is a problem with this Windows Installer package. A program run as
part of the setup did not finish as expected. Contact your support
personnel or package vendor.  Action CompilePyc, location:
C:\Python31\python.exe, command: -Wi "C:\Python31\Lib\compileall.py" -f
-x bad_coding|badsyntax|site-packages|py2_ "C:\Python31\Lib" 

Also, installation completes normally if I choose not to compile .py
files in advanced options setting during the installer's setup dialog.

------
Problem 1: Command line argument of compileall.py seems wrong.

Because installation fails if I choose to compile .py files during
installation, I tried to compile .py files manually after installation
(without compilation) completes.

After finishing installation without compiling .py files, I did the
following command from Windows command prompt.

C:\Python31\python.exe "C:\Python31\Lib\compileall.py" -f -x
bad_coding|badsyntax|site-packages|py2_ "C:\Python31\Lib"

And I got the following error message.
(My Windows is Japanese version.)
'badsyntax' は、内部コマンドまたは外部コマンド、操作可能なプログラムまた
はバッチ ファイルとして認識されていません。

This means, in English,
'badsyntax' is not recognized as an internal or external command,
operable program or batch file.

So I thought the command would be interpreted correctly if I embrace the
following part of the command with "".

bad_coding|badsyntax|site-packages|py2_

So I did the following from command prompt.

C:\Python31\python.exe "C:\Python31\Lib\compileall.py" -f -x
"bad_coding|badsyntax|site-packages|py2_" "C:\Python31\Lib"

And the compilation seemed to proceed.

However, the log file above (python-3.1.1rc1.log) says the argument is
not embraced with "" when compileall.py script is invoked during
installation process.
I suppose the argument not embraced with "" is one of the reasons why
installation process is interrupted.
Is this a bug of the installer package?

------
Problem 2: UnicodeEncodeError occurs depending on code page.

Compiling .py files seems to proceed by double-quoting the argument
discussed above when manually invoking compileall.py script, but the
following error occurs.

Listing C:\Python31\Lib\lib2to3\tests\data ...
Compiling C:\Python31\Lib\lib2to3\tests\data\crlf.py ...
***   File "C:\Python31\Lib\lib2to3\tests\data\crlf.py", line 1
    print "hi"
             ^
SyntaxError: invalid syntax

Compiling C:\Python31\Lib\lib2to3\tests\data\different_encoding.py ...
*** Traceback (most recent call last):
  File "C:\Python31\Lib\py_compile.py", line 142, in compile
    codeobject = builtins.compile(codestring, dfile or file,'exec')
  File "C:\Python31\Lib\lib2to3\tests\data\different_encoding.py", line 3
    print
u'\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd8\xd9\xda\xdb\xdc\xdd\xde'
          ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python31\Lib\compileall.py", line 72, in compile_dir
    ok = py_compile.compile(fullname, None, dfile, True)
  File "C:\Python31\Lib\py_compile.py", line 146, in compile
    raise py_exc
py_compile.PyCompileError:   File
"C:\Python31\Lib\lib2to3\tests\data\different_encoding.py", line 3
    print
u'\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd8\xd9\xda\xdb\xdc\xdd\xde'
          ^
SyntaxError: invalid syntax


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python31\Lib\compileall.py", line 170, in <module>
    exit_status = int(not main())
  File "C:\Python31\Lib\compileall.py", line 160, in main
    force, rx, quiet):
  File "C:\Python31\Lib\compileall.py", line 97, in compile_dir
    quiet):
  File "C:\Python31\Lib\compileall.py", line 97, in compile_dir
    quiet):
  File "C:\Python31\Lib\compileall.py", line 97, in compile_dir
    quiet):
  File "C:\Python31\Lib\compileall.py", line 80, in compile_dir
    print(err.msg)
UnicodeEncodeError: 'cp932' codec can't encode character '\xdf' in
position 86: illegal multibyte sequence

'cp932' is the default code page of Japanese version of Windows.
So I tried the following command to change the current code page of the
command prompt window to 850 (Latin-1) before running compileall.py
script, aiming to avoid the seemingly code-page specific problem.

chcp 850

By doing so, the UnicodeEncodeError seemed to be avoided as expected.

------
Problem 3: Syntax errors.

After changing the code page to 850, the UnicodeEncodeError doesn't seem
to occur.
However, the syntax errors still occur as follows.

Compiling C:\Python31\Lib\lib2to3\tests\data\crlf.py ...
***   File "C:\Python31\Lib\lib2to3\tests\data\crlf.py", line 1
    print "hi"
             ^
SyntaxError: invalid syntax

Compiling C:\Python31\Lib\lib2to3\tests\data\different_encoding.py ...
***   File "C:\Python31\Lib\lib2to3\tests\data\different_encoding.py",
line 3
    print u'ßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞ'
          ^
SyntaxError: invalid syntax

I don't know about how these script files (crlf.py and
different_encoding.py) are used, but isn't it necessary for those lines
of print function to be written like print("hi") because the interpreter
version is 3.x?
msg91722 - (view) Author: (pds) Date: 2009-08-19 04:25
The same problems I reported persist in python-3.1.1.msi, too.
msg98788 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-02-03 16:38
I just saw this today while installing the 3.1.1 final MSI on a Server 2003 box.
msg101100 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-03-15 13:02
Please don't report multiple issues in a single bug report; when people follow up claiming they saw "this", it's then not clear what exactly they saw.

I have now fixed problem 1 in r78976, and put an installer incorporating that change on

http://www.dcl.hpi.uni-potsdam.de/home/loewis/u/python-2.7.14683.msi

Please confirm that this actually fixes the problem.
msg101103 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-03-15 13:21
Your MSI works for me and fixes problem 1.
msg101153 - (view) Author: (pds) Date: 2010-03-16 03:54
To loewis:

I'm sorry for submitting multiple issues in a single bug report.
Also, currently, I'm afraid I don't have a testing environment for your MSI build.

To other members who might follow this issue:

Please specify problem numbers when reporting.

Thanks.
msg101161 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-03-16 11:04
Merged last commit to py3k, as r78991.
msg101168 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-03-16 13:19
Problem 2 is resolved as of r78992
msg101173 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-03-16 16:21
Problem 3 is now fixed in r78994.

Leaving this report open for backporting the changes to 2.6/3.1
msg101465 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-03-21 22:03
Merged as r79248 and r79250.
msg101744 - (view) Author: Gabriel Genellina (ggenellina) Date: 2010-03-26 07:59
r78994 (exclude 2to3 tests from compileall) should be backported to trunk and the 2.6 branch, but I don't see them (yet).
msg101766 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-03-26 18:36
Gabriel: can you explain why that should be backported?
msg101776 - (view) Author: Gabriel Genellina (ggenellina) Date: 2010-03-27 00:29
Sorry for being so terse and not filling in the gaps! In the end, I changed my mind on this bug - it's not an installer issue.

The 2.6.5 MSI installer, when asked to compile .pyc files, exits with an error as reported here:
http://mail.python.org/pipermail/python-list/2010-March/1240182.html

Both Problem#1 and Problem#3 (as reported in this issue) apply to the 2.6.5 installer. Properly quoting the -x argument fixed Problem#1 (already done). But Problem#3 still applies: compileall fails on one test case in lib2to3 because of a syntax error (this was not so clearly stated in the c.l.p. post). The error is:

SyntaxError: ('invalid syntax', ('d:\\apps\\python26\\Lib\\lib2to3\\tests\\data\\py2_test_grammar.py', 936, 23, '        with manager(), manager():\n'))

That syntax is not 2.6 compatible (but 2.7 does allow it).

My initial idea was to completely exclude lib2to3\tests from being precompiled, as done in r78994 for py3k, and both for 2.6 and 2.7. But after looking more closely to the error (and noticing that it is legal in 2.7) I'd leave the installer as it is, and report the issue in py2_test_grammar.py as a separate bug.
History
Date User Action Args
2010-03-27 00:29:16ggenellinasetmessages: + msg101776
2010-03-26 18:36:21loewissetmessages: + msg101766
2010-03-26 07:59:47ggenellinasetnosy: + ggenellina
messages: + msg101744
2010-03-21 22:03:16loewissetstatus: open -> closed
resolution: fixed
messages: + msg101465
2010-03-16 16:21:05loewissetmessages: + msg101173
versions: + Python 2.6
2010-03-16 13:19:48loewissetmessages: + msg101168
2010-03-16 11:04:21loewissetmessages: + msg101161
2010-03-16 03:54:05pdssetmessages: + msg101153
2010-03-15 13:21:05brian.curtinsetmessages: + msg101103
2010-03-15 13:02:25loewissetnosy: + loewis
messages: + msg101100
2010-02-03 16:38:44brian.curtinsetpriority: high

type: compile error -> behavior
components: + Windows

nosy: + brian.curtin
messages: + msg98788
stage: needs patch
2009-08-19 04:25:11pdssetmessages: + msg91722
2009-08-17 13:05:08pdscreate