classification
Title: Enhanced hotshot profiler with high-resolution timer
Type: enhancement Stage:
Components: Extension Modules Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: MrJean1, fdrake, georg.brandl, gregory.p.smith
Priority: normal Keywords: patch

Created on 2008-03-02 23:45 by MrJean1, last changed 2010-08-21 23:57 by georg.brandl. This issue is now closed.

Files
File name Uploaded Description Edit
hires_hotshot.tgz MrJean1, 2008-03-02 23:45
hires_hotshot2.tgz MrJean1, 2008-03-04 23:52
hires_hotshot3.tgz MrJean1, 2008-03-08 00:30
hires_hotshot4.tgz MrJean1, 2008-03-09 22:45
hires_hotshot5.tgz MrJean1, 2008-03-12 21:46
hires_hotshot_2.6.diffs.tgz MrJean1, 2008-03-22 19:31
Messages (10)
msg63194 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-02 23:45
The _hotshot module uses the gettimeofday function to profile the run 
time.

I enhanced the hotshot module in Python 2.5.2 to use a high resolution 
timer where available (RDTSC on x86/_64, MFTB/U on PowerPC and gethrtime 
on Solaris).

The improved hotshot module has been tested on 32-bit MacOS X 10.4.11 
Tiger (Intel) and 10.3.9 Panther (PPC), on 64-bit RHEL 3u7 (Opteron) and 
on 64-bit Solaris 10.

The 3 modified files are Modules/_hotshot.c, Lib/hotshot/log.py and 
Lib/hotshot/stats.py are attached.
msg63224 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-03 18:49
Do *not* use the attached files.  A fix is forthcoming.  My apologies.
msg63268 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-04 23:52
Attached are the improved hotspot files, rev 2.  The changes are rather 
drastic, in particular since hires time delta's may exceed 32-bit int.

These 3 files have been tested with Python 2.5.2 only and on 32-bit 
Linux and MacOS X and on 64-bit Linux and Solaris 10.  They do pass the 
hotspot and other tests.

However, no other platforms or compilers have been tested.  Except for 
64-bit PPC, the code should fall back to the previous behavior using 
gettimeofday.
msg63373 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-08 00:30
Here is another version of the profiler files with 7 minor changes in the 
_hotshot.c file.  Use this hires_version3 set, if any.
msg63407 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-08 21:05
There are 7 other, potentially serious issues in the original _hotshot.c 
file.  All those are being fixed for the next version of the enhanced 
hotshot files.

The 7 issues are:

1) functions flush_data and do_stop may create an infinite recursion on 
line 567.

2) The pack_string and pack_add_info functions do not check for string 
lengths exceeding BUFFERSIZE and may corrupt memory in that case.

3) In line 1182, linetimings should be frametimngs:
   {"frametimings", T_LONG, offsetof(ProfilerObject, linetimings), 
READONLY},

4) Coverage members frametimings, linetimings and lineevents are set on 
line 1562 ff. after different values may have been written into the log 
file header by function write_header on lines 1440 ff.

5) The coverage__doc__ on line 1546 is incomplete.

6) At several places, errors are detected but not reported and not sets.

7) An int value is passed where a long is specified or expected.  T_LONG 
should be T_INT on lines 1182-1184 and there are several calls to PyInt_FromLong with an int argument on line 487 ff. and other places.
msg63429 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-09 22:45
Attached is the latest, hopefully final version of the enhanced and 
improved hotshot profiler files.  In addition to fixes for the 7 issues 
mentioned previously, type Py_ssize_t is used instead of size_t.  Use 
this verion 4 and discard all previous ones.

Please review this code in file _hotshot.c and let me know of any rule 
or guideline violations.  I am more than happy to correct any mistakes.
msg63485 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-12 21:46
Attached is yet another, final update of the enhancements to the hotshot 
profiler.  It includes modifications of all 3 files, Modules/_hotshot.c, 
Lib/hotshot/log.py and Lib/hotshot/stats.py.
msg64335 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-22 19:31
Here are the forward diffs of the 3 files against the version in the 
trunk.
msg74466 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2008-10-07 17:55
May I ask what the reasoning is for further developing hotshot as part
of the core?  My understanding, based on discussions on python-dev, is
that hotshot is being deprecated in favor of cProfile.

If hotshot still provides functionality that's interesting (and it may),
perhaps it makes sense to maintain and improve it in a separate
distribution.
msg114614 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-08-21 23:57
hotshot won't get any more updates.
History
Date User Action Args
2010-08-21 23:57:05georg.brandlsetstatus: open -> closed

nosy: + georg.brandl
messages: + msg114614

resolution: out of date
2008-10-07 17:55:39fdrakesetmessages: + msg74466
2008-10-07 17:47:41MrJean1setversions: + Python 2.7, - Python 2.5.3
2008-10-07 17:41:47MrJean1setversions: + Python 2.5.3, - Python 2.6
2008-05-25 09:33:08gregory.p.smithsetpriority: normal
nosy: + gregory.p.smith
keywords: + patch
2008-03-22 19:31:30MrJean1setfiles: + hires_hotshot_2.6.diffs.tgz
messages: + msg64335
versions: + Python 2.6, - Python 2.5
2008-03-12 21:46:50MrJean1setfiles: + hires_hotshot5.tgz
messages: + msg63485
2008-03-09 22:45:16MrJean1setfiles: + hires_hotshot4.tgz
messages: + msg63429
2008-03-08 21:05:30MrJean1setmessages: + msg63407
2008-03-08 00:30:40MrJean1setfiles: + hires_hotshot3.tgz
messages: + msg63373
2008-03-04 23:52:34MrJean1setfiles: + hires_hotshot2.tgz
messages: + msg63268
2008-03-03 22:08:24benjamin.petersonsetnosy: + fdrake
2008-03-03 18:49:54MrJean1setmessages: + msg63224
2008-03-02 23:45:10MrJean1create