classification
Title: dev guide appears not to cover the benchmarking suite
Type: enhancement Stage: resolved
Components: Devguide Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, dmalcolm, edmond.burnett, eric.snow, ezio.melotti, jramnani, kavya, meador.inge, pitrou, python-dev, ronaldoussoren
Priority: normal Keywords: easy, patch

Created on 2013-03-18 04:08 by dmalcolm, last changed 2014-06-22 23:49 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue-17449-readme.txt ronaldoussoren, 2013-07-29 15:10
issue17449.patch kavya, 2014-06-07 19:35
Messages (17)
msg184421 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2013-03-18 04:08
Does the devguide document the benchmarking suite anywhere?  I can't see it anywhere in the index on http://docs.python.org/devguide/ and google doesn't seem to show anything.

suggested content:
  * how to run the benchmarks for a Python 2 implementation
  * how to run the benchmarks for a Python 3 implementation

Sorry if it exists and I missed it
msg184670 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-19 20:28
No it does not, but it should.
msg185083 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-03-23 20:20
Mmmh... I'm not very warm on this one. If we start documenting the benchmark suite in the devguide, then people will start to be all gung ho about it, be obsessed with benchmark numbers, etc.

I guess what I'm saying is that *if* it's covered, then the very beginning of the section should explain that benchmarking is not important in most cases, and time spent on Python is generally better spent fixing bugs and adding features.
msg185103 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-23 22:19
Benchmarking is also useful to test that existing patches (that fix bug or add features) don't have a negative effect on the performance.  Some kind of warning that advises against using benchmarks just to try and optimize everything sounds like a good idea though.
msg188251 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2013-05-02 00:49
I think this will be useful information.  I tend to doing benchmarks
rather infrequently.  Whenever I do them I forgot how I setup the suite
the previous time around.

In fact, I found this issue just now while googling for how to setup and
run the official benchmark suite :-)
msg193863 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-29 13:27
Documenting this would be useful because it appears to be non-trivial to actually run the benchmark suite with Python 3. 

* Using "python perf.py -b default" fails due to some code in the
  benchmarks that is not python3 compatible

* Using "make_perf3.sh" and then running "... -b default" still fails,
  because not all libraries are translated (the test run stopped because
  splitfire couldn't be found)

* Using "-b 2n3" instead instead of "-b default" or "-b all" works, but
  runs only a subset of the benchmark.
msg193864 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-07-29 13:34
> * Using "-b 2n3" instead instead of "-b default" or "-b all" works,
> but
>   runs only a subset of the benchmark.

I think we should include a copy of the 2to3'ed third-party libs in
the benchmarks repo, this would running avoid the length and tedious
conversion on every user's computer.
msg193865 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-29 13:40
The overhead of 2to3 isn't too bad, especially because only a subset of the benchmarks is translated in the first place. 

It is more important that there is better documentation than there is now, it seems that you're supposed to use either '-b 2n3' or make_perf3.sh with '-b py3k'.  Neither of which are mentioned in README.txt.
msg193866 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-07-29 13:45
> The overhead of 2to3 isn't too bad, especially because only a subset
> of the benchmarks is translated in the first place.

Well, the biggest time is spent translating the third-party libraries,
not benchmark files.
Also, it's simply tedious manual work (you have to create a separate
directory etc.).

> It is more important that there is better documentation than there is
> now, it seems that you're supposed to use either '-b 2n3' or
> make_perf3.sh with '-b py3k'.  Neither of which are mentioned in
> README.txt.

That's because the benchmark suite was initially 2.x-only, and the
"make_perf3.sh" was a quick hack to make some of the benchmarks work
on 3.x.

Nowadays the goal is for benchmark files to be 2.x and 3.x compatible
in the same codebase, so that only third-party libs remain to
translate. Adding 3.x-compatible versions of the third-party libs
would eliminate that last step.
msg193868 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-29 14:01
The attached patch for the benchmark README.txt should document the status quo.
msg193871 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2013-07-29 15:04
On Mon, 2013-07-29 at 14:01 +0000, Ronald Oussoren wrote:
> Ronald Oussoren added the comment:
> 
> The attached patch for the benchmark README.txt should document the status quo.
> 
> ----------
> Added file: http://bugs.python.org/file31073/issue-18181-full-v3.txt
It looks like you attached the wrong patch: the patch you attached seems
to be unrelated to benchmarking (and presumably relates to issue 18181)
msg193872 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-29 15:10
Sorry about that. I stored the patch file in the wrong directory and attached the most recent file in the directory where I had expected the patch to be.
msg193876 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-07-29 15:38
I agree with Antoine that the time has come to drop make_perf3.sh and make it so that Python 3 doesn't require a separate build of the benchmark directory. Created http://bugs.python.org/issue18586 to track this idea.
msg216406 - (view) Author: Jeff Ramnani (jramnani) * Date: 2014-04-15 22:23
Now that bug #18586 is closed, could the Dev Guide point benchmarkers to the benchmarks repo and its README?

http://hg.python.org/benchmarks/file/9a1136898539/README.txt
msg216516 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-16 17:48
Jeff: yes it could :) Do you want to provide a patch?
msg219964 - (view) Author: Kavya Joshi (kavya) Date: 2014-06-07 19:35
I added a `Benchmarking` section in the dev guide with the relevant links and the intended use (testing rather than optimizing).
I also added an entry for the section in the Index.

Testing/verification:
I built the docs and visually inspected them.
msg221326 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-06-22 23:49
New changeset 428b6350307e by Benjamin Peterson in branch 'default':
add a pointer to the benchmarks repo (closes #17449)
http://hg.python.org/devguide/rev/428b6350307e
History
Date User Action Args
2014-06-22 23:49:18python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg221326

resolution: fixed
stage: needs patch -> resolved
2014-06-07 19:35:31kavyasetfiles: + issue17449.patch

nosy: + kavya
messages: + msg219964

keywords: + patch
2014-04-16 17:48:13pitrousetmessages: + msg216516
2014-04-15 22:23:22jramnanisetnosy: + jramnani
messages: + msg216406
2013-07-29 15:38:50brett.cannonsetmessages: + msg193876
2013-07-29 15:10:46ronaldoussorensetfiles: + issue-17449-readme.txt

messages: + msg193872
2013-07-29 15:09:13ronaldoussorensetfiles: - issue-18181-full-v3.txt
2013-07-29 15:04:11dmalcolmsetmessages: + msg193871
2013-07-29 14:01:31ronaldoussorensetfiles: + issue-18181-full-v3.txt

messages: + msg193868
2013-07-29 13:45:43pitrousetmessages: + msg193866
2013-07-29 13:40:19ronaldoussorensetmessages: + msg193865
2013-07-29 13:34:05pitrousetmessages: + msg193864
2013-07-29 13:27:19ronaldoussorensetnosy: + ronaldoussoren
messages: + msg193863
2013-05-29 19:34:08eric.snowsetnosy: + eric.snow
2013-05-02 00:49:34meador.ingesetnosy: + meador.inge
messages: + msg188251
2013-04-06 02:16:48edmond.burnettsetnosy: + edmond.burnett
2013-03-23 22:19:44ezio.melottisetmessages: + msg185103
2013-03-23 20:20:17pitrousetnosy: + pitrou
messages: + msg185083
2013-03-23 14:14:43ezio.melottisettype: enhancement
stage: needs patch
2013-03-19 20:28:03brett.cannonsetkeywords: + easy
assignee: brett.cannon ->
messages: + msg184670
2013-03-18 04:08:17dmalcolmcreate