classification
Title: Benchmarks: 2to3 failures on the py3 side
Type: Stage:
Components: 2to3 (2.x to 3.x conversion tool), Benchmarks Versions: Python 3.2
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: benjamin.peterson, brett.cannon, collinwinter, ezio.melotti, francismb, pitrou
Priority: normal Keywords:

Created on 2012-02-10 17:32 by francismb, last changed 2012-09-13 16:23 by francismb. This issue is now closed.

Files
File name Uploaded Description Edit
FailuresErrors.txt francismb, 2012-02-10 17:32 Failures and errors summary
testsResult.txt francismb, 2012-09-13 11:29
testsResult2.txt francismb, 2012-09-13 13:15
Messages (11)
msg153069 - (view) Author: Francis MB (francismb) * Date: 2012-02-10 17:32
Steps to reproduce:
after following the usage instructions from make_perf3.sh go to the the py3 benchmarks directory and run the lib/2to3 tests (python3.2 test.py). The result are 3 failures and 17 errors. The equivalent procedure in the py2 benchmarks directory succeeds (python2.7 test.py).

A summary is attached (the whole paste at http://pastebin.com/PAAhGsdR)

Cheers,
francis
msg153073 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-10 19:02
It looks like 2to3 isn't being converted properly. Benjamin, is lib2to3 supposed to work on itself and result in a sane output? If not then the benchmark should probably drop its internal copy of lib2to3.

Antoine (since you last looked at the benchmark suite for Python 3), do you know anything about this?
msg153135 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-11 19:58
> Antoine (since you last looked at the benchmark suite for Python 3), do 
> you know anything about this?

I don't know. It works here. It may depend on the version of 2to3 used for the conversion, try passing PYTHON=some_newer_python when running make_perf.sh.
msg170374 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-09-12 12:47
It's translating properly for me now as well (but on a different machine). I will give it another try when I get home to see if I can reproduce success on the machine that was failing for me previously.
msg170420 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-09-12 21:08
I can't reproduce the problem anymore.
msg170431 - (view) Author: Francis MB (francismb) * Date: 2012-09-13 11:29
Just info:
I still can reproduce that. (It's maybe just something wrong with my setup?)
msg170432 - (view) Author: Francis MB (francismb) * Date: 2012-09-13 11:33
The actual tip for me is:

ci@random:~/Prog/cpython/benchmarks/py2$ hg tip
changeset:   164:61768f86170c
tag:         tip
user:        Brett Cannon <brett@python.org>
date:        Fri Aug 31 18:58:24 2012 -0400
summary:     Allow for the specification of a base directory for either interpreter
msg170434 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-09-13 12:40
Can you copy-and-paste the exact commands you are using to build and run the benchmarks (along with version info for the interpreter used to do the building) as I'm on the same revision and can't reproduce anymore?

$ mkdir benchmarks_py3k
$ cd benchmarks_py3k/
$ $ /usr/bin/python2.6 -V
Python 2.6.5
$ PYTHON=/usr/bin/python2.6 ../benchmarks/make_perf3.sh ../benchmarks
# ... 2to3 output
$ python3 -V
Python 3.2.3
$ python3 perf.py -f -b 2to3 `which python3` `which python3`
Running 2to3...
INFO:root:Running /usr/bin/python3 lib/2to3/2to3 -f all lib/2to3/lib2to3/refactor.py
INFO:root:Running `['/usr/bin/python3', 'lib/2to3/2to3', '-f', 'all', 'lib/2to3/lib2to3/refactor.py']` 1 time
INFO:root:Running /usr/bin/python3 lib/2to3/2to3 -f all lib/2to3/lib2to3/refactor.py
INFO:root:Running `['/usr/bin/python3', 'lib/2to3/2to3', '-f', 'all', 'lib/2to3/lib2to3/refactor.py']` 1 time

Report on Linux importthis.wat.corp.google.com 3.2.5-gg944 #1 SMP Thu Aug 2 02:35:13 PDT 2012 x86_64 x86_64
Total CPU cores: 12

### 2to3 ###
0.540000 -> 0.560000: 1.04x slower
msg170440 - (view) Author: Francis MB (francismb) * Date: 2012-09-13 13:15
The box is a debian (wheezy):
ci@random:~/Prog/cpython/benchmarks/py2$ uname -a
Linux random 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 GNU/Linux

The source repository dir:
ci@random:~/Prog/cpython/benchmarks/py2$ dir
lib  LICENSE.txt  make_perf3.sh  performance  perf.py  perf.pyc  README.txt  test_perf.py

Then:

ci@random:~/Prog/cpython/benchmarks/py2$ cd ..
ci@random:~/Prog/cpython/benchmarks$ mkdir benchmarks_py3k
ci@random:~/Prog/cpython/benchmarks$ dir
benchmarks_py3k  py2
ci@random:~/Prog/cpython/benchmarks$ cd benchmarks_py3k/
ci@random:~/Prog/cpython/benchmarks/benchmarks_py3k$
ci@random:~/Prog/cpython/benchmarks/benchmarks_py3k$ /usr/bin/python2.6 -V
Python 2.6.8
ci@random:~/Prog/cpython/benchmarks/benchmarks_py3k$ PYTHON=/usr/bin/python2.6 ../py2/make_perf3.sh ../py2
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
...
RefactoringTool: Warnings/messages while refactoring:
RefactoringTool: ### In file lib/2to3/example.py ###
RefactoringTool: Line 371: You should use a for loop here
RefactoringTool: Line 372: You should use a for loop here
RefactoringTool: Line 373: You should use a for loop here
RefactoringTool: Line 374: You should use a for loop here
RefactoringTool: Line 375: You should use a for loop here
RefactoringTool: ### In file lib/2to3/example.py ###
RefactoringTool: Line 393: You should use 'hasattr(foo, '__call__')' here.
RefactoringTool: Line 394: You should use 'operator.contains(foo, bar)' here.
RefactoringTool: Skipping implicit fixer: buffer
...
RefactoringTool: lib/mako/test/templates/subdir/foo/modtest.html.py
ci@random:~/Prog/cpython/benchmarks/benchmarks_py3k$ python3 -V
Python 3.2.3
python3 perf.py -f -b 2to3 `which python3` `which python3`
Running 2to3...
INFO:root:Running /usr/bin/python3 lib/2to3/2to3 -f all lib/2to3/lib2to3/refactor.py
INFO:root:Running `['/usr/bin/python3', 'lib/2to3/2to3', '-f', 'all', 'lib/2to3/lib2to3/refactor.py']` 1 time
INFO:root:Running /usr/bin/python3 lib/2to3/2to3 -f all lib/2to3/lib2to3/refactor.py
INFO:root:Running `['/usr/bin/python3', 'lib/2to3/2to3', '-f', 'all', 'lib/2to3/lib2to3/refactor.py']` 1 time

Report on Linux random 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 
Total CPU cores: 4

### 2to3 ###
0.616038 -> 0.624039: 1.01x slower


And further:
ci@random:~/Prog/cpython/benchmarks/benchmarks_py3k$ dir
lib  performance  perf.py
cd lib/2to3
ci@random:~/Prog/cpython/benchmarks/benchmarks_py3k/lib/2to3$ dir
2to3  example.py  HACKING  lib2to3  README  scripts  test.py
ci@random:~/Prog/cpython/benchmarks/benchmarks_py3k/lib/2to3$ python3 test.py 2> testsResult2.txt
msg170443 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-09-13 13:53
So the tests failed but the benchmarks finished without issue? I'm not too worried about the tests since they are almost all str/bytes interaction issues (although I don't get the .decode() failure). If the benchmarks run without errors then I'm satisfied, else we figure out what this fails ever or we have a already translated copy kept in a lib3 directory to prevent problems.
msg170445 - (view) Author: Francis MB (francismb) * Date: 2012-09-13 16:23
> So the tests failed but the benchmarks finished without issue?
Yes. But the apparent diff is that I've used 2.6.8

If the benchmarks run without errors then I'm satisfied, else we figure 
out what this fails ever
or we have a already translated copy kept in a lib3 directory to prevent 
problems.

The actual trunk revision for 2to3 from:
https://svn.python.org/projects/sandbox/trunk/2to3/ is
Revision 88981: /sandbox/trunk/2to3

the README.txt says:
2to3: r72994 from http://svn.python.org/projects/sandbox/trunk/2to3

I've just updated to that version but nothing changed: same errors (and 
a new test test_formfeed (lib2to3.tests.test_parser.TestDriver) that was ok)

cheers,
francis
History
Date User Action Args
2012-09-13 16:23:40francismbsetmessages: + msg170445
2012-09-13 13:53:01brett.cannonsetmessages: + msg170443
2012-09-13 13:15:39francismbsetfiles: + testsResult2.txt

messages: + msg170440
2012-09-13 12:40:08brett.cannonsetmessages: + msg170434
2012-09-13 11:33:28francismbsetmessages: + msg170432
2012-09-13 11:29:56francismbsetfiles: + testsResult.txt

messages: + msg170431
2012-09-12 21:08:52brett.cannonsetstatus: open -> closed
resolution: out of date
messages: + msg170420
2012-09-12 12:48:25brett.cannonsetassignee: benjamin.peterson -> brett.cannon
2012-09-12 12:47:56brett.cannonsetstatus: pending -> open

messages: + msg170374
2012-09-11 22:59:49ezio.melottisetstatus: open -> pending
2012-02-11 19:58:25pitrousetmessages: + msg153135
2012-02-10 19:05:46ezio.melottisetnosy: + ezio.melotti
2012-02-10 19:02:49brett.cannonsetassignee: collinwinter -> benjamin.peterson

messages: + msg153073
nosy: + brett.cannon, pitrou, benjamin.peterson
2012-02-10 17:32:33francismbcreate