classification
Title: Merge profile/cProfile in 3.n+1
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: alexandre.vassalotti Nosy List: BreamoreBoy, MrJean1, alexandre.vassalotti, asvetlov, barry, belopolsky, benjamin.peterson, brett.cannon, eric.araujo, eric.snow, flox, georg.brandl, hdiogenes
Priority: critical Keywords: patch

Created on 2008-05-19 20:02 by brett.cannon, last changed 2013-10-15 04:39 by alexandre.vassalotti. This issue is now closed.

Files
File name Uploaded Description Edit
profile_module2.tgz MrJean1, 2009-06-02 18:40 merged profile/cProfile module, rev'd
pyprofile.py alexandre.vassalotti, 2009-06-28 17:01 Profiler interface for lsprof.
lsprof.py alexandre.vassalotti, 2009-07-07 00:37 Python version of the _lsprof module (updated).
Messages (22)
msg67073 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-05-19 20:02
cProfile should go away in 3.0 (as a public module) and just be what 
profile uses when available.
msg67896 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2008-06-10 04:56
I will try to fix issue. I cannot promise that I will get it done before
the beta though.
msg68043 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-06-12 02:09
This didn't get done for the betas, but we're still going to release. 
Knocking it down to critical.
msg71312 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-08-18 02:59
Can you get this done before beta 3?
msg71392 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2008-08-18 23:26
Sorry, I can't. I would love to get this done, but I am just too busy
this week (and next week doesn't look any better). Furthermore, the
amount of work that the profile/cProfile merge will require is quite
considerable. So, I don't expect someone else could get this done before
the beta.
msg71393 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-08-18 23:27
Let's retarget 3.1 then.
msg88553 - (view) Author: Humberto Diogenes (hdiogenes) Date: 2009-05-29 23:42
How is this going?
msg88613 - (view) Author: Jean Brouwers (MrJean1) Date: 2009-05-31 21:56
Attached is *an* attempt to combine the old profile/cProfile modules into a single 
one called profile.  Only the C and Python source and test files are included, not 
any documentation files.  More details are in the README file, copied below.

Hopefully, this is useful.

/Jean Brouwers


These are the modified files to replace the old profile module with the fast cProfile 
one and rename the latter.  A complete list and description follow.


./README

./profileNEW/setup.py
./profileNEW/Lib/profile.py
./profileNEW/Lib/test/profilee.py
./profileNEW/Lib/test/test_profile.py
./profileNEW/Modules/_profile.c

./profilePy3k/setup.py
./profilePy3k/Lib/profile.py
./profilePy3k/Lib/test/profilee.py
./profilePy3k/Lib/test/test_cprofile.py
./profilePy3k/Lib/test/test_profile.py
./profilePy3k/Modules/_lsprof.c

./diff_-u_profilePy3k_Modules_lsprof.c_profileNEW_Modules_profile.c.txt
./diff_-ur_profilePy3k_profileNEW.txt

./_profile_hires.c



Directory profileNEW contains all the source and test files in their corresponding 
directories.  Directory  profilePy3k contains the old, original files from the py3k 
trunk taken on May 31, 2009.

Basically, the old _lsprof.c file has been modified and renamed to _profile.c.  Both 
old test_profile.py and test_cprofile.py files have been combined into a single new 
file test_profile.py.

The file  diff_-ur_profilePy3k_profileNEW.txt  contains the forward diffences between 
all the new and orginal files.

File diff_-u_profilePy3k_Modules_lsprof.c_profileNEW_Modules_profile.c.txt contains 
just the differences between the old  _lsprof.c and new  _profile.c files.

Lastly, for your consideration, file _profile_hires.c is the new _profile.c file 
enhanced to use a high resolution timer, see <http://bugs.python.org/issue2281>.  
This file would replace the _profile.c file in the profileNEW/Modules directory.



The new files and tests were installed in a Python 3.0.1 build and compiled and 
tested on MacOS X 10.4.11 Tiger (Intel).  The patches from issue 5330 were also 
installed in this build, see <http://bugs.python.org/issue5330>.  All tests passed 
after regenerating the expected results

% python.exe Lib/test/test_profile.py -r
Regenerating Lib/test/test_profile.py...

% python.exe Lib/test/test_profile.py 
test_bad_counter_during_dealloc (__main__.ProfileTest) ... ok
test_calling_conventions (__main__.ProfileTest) ... ok
test_profile (__main__.ProfileTest) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.017s

OK
msg88614 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2009-05-31 22:05
Thanks for the code, Jean. With Python 3.1 about to go out the door this
will have to be a 3.2 thing. But I plan to start looking at this module
merge some time in July so I should get to looking at what you did then
(unless someone beats me to it).
msg88624 - (view) Author: Jean Brouwers (MrJean1) Date: 2009-05-31 23:50
I just added another comment about the high-resolution timer in <http://bugs.python.org/issue2281>.
msg88762 - (view) Author: Jean Brouwers (MrJean1) Date: 2009-06-02 18:40
Fixed a couple of typos in the README and _profile_hires.c files of the 
attached profile_module2.tgz tar ball.
msg89171 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-06-09 22:08
Jean Brouwers wrote:
> All tests passed after regenerating the expected results

The tests for profile are output test. If you regenerate them, they pass
for sure.

Merging cProfile/profile will require a lot more work than just renaming
the module. The designs of cProfile and profile are significantly
different. So a successful merge will require settling one profiler
design (cProfile is the better designed one, IMHO) and rewrite the other
one to match the chosen design.
msg89778 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-06-28 16:58
Yesterday, I ported the _lsprof module to Python as an experiment. I
still have a few issues to iron out, but the core of the functionality
is there.  

I am not sure yet how this experiment will fit in the profile/cProfile
merge. I wrote the port mainly to get a better idea of what is needed
for the merge and how the profiler works.
msg90205 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-07-07 00:37
Here's an updated version of my lsprof.py module. I fixed a few bugs,
includes the awful reference leak that was present in the previous
version (i.e., the profiler code was keeping a reference to every frame
executed).

I consider the lsprof.py code functionally equivalent to _lsprof. Now, I
am wondering if I should add the features from profile.py missing in
_lsprof, e.g., the bias argument and the calibration code.
msg97612 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2010-01-11 22:50
Any comment about the code I posted?
msg97647 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-01-12 18:35
I'll see if I can have a look sometime soon, but my profile experience is practically non-existent so I am not the best person for a code review.
msg110190 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-13 13:36
Is there anyone who could review this, it seems a shame that someone's work has been sitting in limbo for over a year?
msg110216 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-13 18:54
Is this a case of pure python implementation with a C speedup?  Can you upload the patch to Rietveld?  I'll take a look once it is there.
msg110217 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-07-13 18:55
Alexandre has commit privs so he could just go ahead and commit if he wants.

I do hope to look at this code at some point in my life. I will see if I can't find time at EuroPython next week, especially if Alexander has fixes to it and still feels the need for a review.
msg165958 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-07-20 17:52
So I didn't find time at EuroPython. =) Is this dead, Alexandre, or do you want to see this happen for Python 3.4?
msg199897 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-10-14 14:28
Last desperate ping for 3.4?
msg199974 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-10-15 04:39
Let's close this. There is no point keeping this languishing any further.

The profile and cProfile modules are different enough that we can't merge them without breaking compatibility. They should treated as two separate profilers.

Maybe in the future we should include a drop-in module written in Python for cProfile to be used for another Python implementation. But for now, I don't see the benefit, since the profile module is good enough and alternate implementations should be able to re-do the _lsprof accelerator in their target language without too much trouble if desired.
History
Date User Action Args
2013-10-15 04:39:36alexandre.vassalottisetstatus: open -> closed
assignee: alexandre.vassalotti
resolution: wont fix
messages: + msg199974
2013-10-14 14:28:35georg.brandlsetnosy: + georg.brandl
messages: + msg199897
2012-11-17 15:05:49brett.cannonunlinkissue2775 dependencies
2012-07-21 10:10:34floxsetnosy: + flox
2012-07-20 17:52:13brett.cannonsetmessages: + msg165958
2012-06-24 20:11:53pitrousettitle: Merge profile/cProfile in 3.0 -> Merge profile/cProfile in 3.n+1
2011-12-09 19:32:16eric.snowsetnosy: + eric.snow
2010-10-14 23:12:31eric.araujosetnosy: + eric.araujo
2010-07-13 18:55:41brett.cannonsetmessages: + msg110217
2010-07-13 18:54:10belopolskysetnosy: + belopolsky
messages: + msg110216
2010-07-13 13:36:47BreamoreBoysetnosy: + BreamoreBoy
messages: + msg110190
2010-03-23 05:57:51asvetlovsetnosy: + asvetlov
2010-01-12 18:35:48brett.cannonsetmessages: + msg97647
2010-01-11 22:50:47alexandre.vassalottisetmessages: + msg97612
2009-07-07 00:37:16alexandre.vassalottisetfiles: - lsprof.py
2009-07-07 00:37:06alexandre.vassalottisetfiles: + lsprof.py

messages: + msg90205
2009-06-28 17:01:52alexandre.vassalottisetfiles: + pyprofile.py
2009-06-28 16:58:55alexandre.vassalottisetfiles: + lsprof.py

messages: + msg89778
2009-06-09 22:08:10alexandre.vassalottisetmessages: + msg89171
2009-06-02 18:40:27MrJean1setfiles: - profile_module.tgz
2009-06-02 18:40:15MrJean1setfiles: + profile_module2.tgz

messages: + msg88762
2009-05-31 23:50:44MrJean1setmessages: + msg88624
2009-05-31 22:13:15rhettingersetversions: + Python 3.2, - Python 3.1
2009-05-31 22:05:15brett.cannonsetmessages: + msg88614
stage: patch review
2009-05-31 21:56:31MrJean1setfiles: + profile_module.tgz
nosy: + MrJean1
messages: + msg88613

2009-05-29 23:42:42hdiogenessetnosy: + hdiogenes
messages: + msg88553
2008-08-18 23:27:40benjamin.petersonsetpriority: release blocker -> critical
nosy: + benjamin.peterson
messages: + msg71393
versions: + Python 3.1, - Python 3.0
2008-08-18 23:26:07alexandre.vassalottisetkeywords: + patch
messages: + msg71392
2008-08-18 02:59:48barrysetmessages: + msg71312
2008-07-31 02:16:40benjamin.petersonsetpriority: critical -> release blocker
2008-06-12 02:09:22barrysetpriority: release blocker -> critical
nosy: + barry
messages: + msg68043
2008-06-10 04:56:46alexandre.vassalottisetnosy: + alexandre.vassalotti
messages: + msg67896
2008-05-19 20:02:29brett.cannonlinkissue2775 dependencies
2008-05-19 20:02:13brett.cannoncreate