classification
Title: hotshot.stats.load fails with AssertionError
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, ajaksu2, amaury.forgeotdarc, barry, brett.cannon, dstanek, glchapman, gsakkis, jlgijsbers, pitrou, sdahlbac
Priority: normal Keywords: patch

Created on 2004-02-19 08:05 by sdahlbac, last changed 2014-09-05 23:09 by brian.curtin.

Files
File name Uploaded Description Edit
900092-patch.txt barry, 2005-07-07 23:26 Patch candidate for Python 2.4.x
900092-patch-2.txt barry, 2005-07-08 15:02 Test suite patch
bug.prof brett.cannon, 2007-02-10 01:48 crasher from gsakkis
Messages (26)
msg20042 - (view) Author: Simon Dahlbacka (sdahlbac) Date: 2004-02-19 08:05
trying to do a 

hotshot.stats.load("myprofiling_file.prof")

fails with assertionerror

assert not self._stack


python 2.3.2 on WinXP

msg20043 - (view) Author: Johannes Gijsbers (jlgijsbers) * Date: 2004-09-24 21:58
Logged In: YES 
user_id=469548

While the original report isn't very useful, I've ran into
this problem multiple times as well. I can reproduce it
using the attached profile file (compressed because of the
large size) and the following console session:

Python 2.3.4 (#2, Jul  5 2004, 09:15:05)
[GCC 3.3.4 (Debian 1:3.3.4-2)] on linux2
Type "help", "copyright", "credits" or "license" for more
information.
>>> import hotshot.stats
>>> stats = hotshot.stats.load('roundup.prof')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.3/hotshot/stats.py", line 12, in load
    return StatsLoader(filename).load()
  File "/usr/lib/python2.3/hotshot/stats.py", line 51, in load
    assert not self._stack
AssertionError
>>>

I'm not sure who's baby hotshot really is, so I'm leaving
this unassigned.
msg20044 - (view) Author: Johannes Gijsbers (jlgijsbers) * Date: 2004-09-24 22:00
Logged In: YES 
user_id=469548

Hmm, the file was too big, even though it was compressed.
I've uploaded it to
http://home.student.uva.nl/johannes.gijsbers/roundup.prof.bz2
now.
msg20045 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2004-09-27 14:41
Logged In: YES 
user_id=12800

Could this be related to 1019882?
msg20046 - (view) Author: Greg Chapman (glchapman) Date: 2004-11-08 23:32
Logged In: YES 
user_id=86307

I ran into this today, so I decided to look into it.  It
looks to me like the problem only happens if you profile
with lineevents enabled.  In that case, hotshot uses the
tracer_callback function (in _hotshot.c) to dispatch trace
events.  This function explicitly ignores exception returns
(PyTrace_EXCEPTION), which can lead to an unbalanced stack
of calls/returns when the log is loaded (if an profiled
function exits with an exception).

It seems on the surface that tracer_callback ought to handle
exceptions the same way as normal returns.  This would be
consistent with what happens when profiler_callback is used,
since PyEval_EvalFrame dispatches sends a Py_RETURN to
c_profilefunc when exiting because of an exception.
msg20047 - (view) Author: Greg Chapman (glchapman) Date: 2004-11-08 23:54
Logged In: YES 
user_id=86307

Well, the superficial fix doesn't work (sorry for posting
too soon).  It turns out that PyTrace_EXCEPTION is sent for
any exception, not just one causing the function to exit. 
So I guess the best fix may be to have hotshot always
install its profiler_callback to handle CALLS and RETURNS,
and then optionally install the tracer_callback to handle
only LINEs.  Anyway, that works for the one test case I've
been using (a runcall of a function which simply does
"import pickle").

By the way, I'm testing with 2.4b1.
msg20048 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2005-07-07 23:26
Logged In: YES 
user_id=12800

See 900092-patch.txt for a candidate patch against Python
2.4.1.  Props to Justin Campbell for most of the heavily
lifting (but you can blame me for any problems ;).

This fix restore the tracing of a 'return' event for
exceptions that cause a function to exit.
msg20049 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2005-07-08 15:02
Logged In: YES 
user_id=12800

900092-patch-2.txt fixes the test suite for the extra return
event.
msg20050 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2005-08-15 18:14
Logged In: YES 
user_id=12800

Patches applied for Python 2.4.2 and 2.5a1
msg20051 - (view) Author: George Sakkis (gsakkis) Date: 2007-02-08 23:37
Has this ever been reported again since it was closed ? I just got it today (python 2.5). The prof file is 11MB, I'll see if I can reproduce the bug with a smaller one.
msg20052 - (view) Author: George Sakkis (gsakkis) Date: 2007-02-09 00:08
Ok, I reduced the prof file to 38K. Can I upload an attachment here ? I don't see how.
msg20053 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2007-02-09 18:15
I think only OPs and project members can upload.  Email it to me, George (brett at python.org), mention the bug # and I will upload it.
msg20054 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2007-02-10 01:48
Attaching the file George made.
File Added: bug.prof
msg82011 - (view) Author: Daniel Diniz (ajaksu2) Date: 2009-02-14 11:32
Has test, patch and sample data for reproducing.
msg89975 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-07-01 13:45
I added some prints in the hotshot.stats.load function, this lead to
interesting things:

- first, the program seems to spawn subprocesses which trace themselves
in the same file.

- however, there is real issue when fork() is called on a traced
program: calls to _execvpe should not appear in the trace!
IMO PyOS_AfterFork should disable any trace or profile function.
msg110391 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-15 22:08
Is it worth leaving this open, given this from section 26.5 of the 2.7 docs.  "For common usage it is recommended to use cProfile instead. hotshot is not maintained and might be removed from the standard library in the future."
msg110823 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-19 22:38
Closing as hotshot is not maintained and is not documented in py3k.
msg110826 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-07-19 22:44
I don't think this should have been closed just yet. If the issue still exists in 2.x, it could still be fixed in the remaining 2.6 release, or any of the future 2.7 releases.

You are right that it won't apply to 3.x since hotshot is gone there.
msg110827 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-19 22:50
Who of our overworked volunteers is going to do the work, given that hotshot doesn't even feature on the maintainers list?
msg114323 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-19 05:55
Closing as no reply to msg110827.
msg214139 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2014-03-19 21:29
It was brought to my attention that this is still an issue and that it was closed. Reopening.
msg214145 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-03-19 23:20
IMHO reopening this was a complete and utter waste of core devs time, its already passed its tenth birthday!!!
msg214146 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2014-03-19 23:22
It remains a valid issue. Time is not a factor.
msg214147 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-03-19 23:29
You clearly haven't bothered to read that there is *NO* maintainer, that the docs for 2.x recommend that you *DON'T* use it, that its been removed from 3.x  and that nobody has yet volunteered to do anything with it.  If its so important, why don't you do the work instead of trying to palm it off onto people who have far more important things to consider?  I'll therefore repeat so that you very clearly get my message, this is a complete and utter waste of core devs time.
msg214148 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-19 23:35
Brian Curtin is a core developer, perhaps you should trust his judgement.
msg214150 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-03-19 23:44
Then God help Python is all I can say.
History
Date User Action Args
2014-09-05 23:09:30brian.curtinsetnosy: - brian.curtin
2014-09-05 18:11:26BreamoreBoysetversions: + Python 2.7, - Python 2.6
2014-03-19 23:44:14BreamoreBoysetmessages: + msg214150
2014-03-19 23:37:55dstaneksetnosy: + dstanek
2014-03-19 23:35:02pitrousetnosy: + pitrou
messages: + msg214148
2014-03-19 23:29:51BreamoreBoysetmessages: + msg214147
2014-03-19 23:22:15brian.curtinsetmessages: + msg214146
2014-03-19 23:20:16BreamoreBoysetmessages: + msg214145
2014-03-19 21:29:34brian.curtinsetstatus: closed -> open
resolution: wont fix ->
messages: + msg214139
2010-08-19 05:55:04BreamoreBoysetstatus: open -> closed
resolution: wont fix
messages: + msg114323
2010-07-19 22:50:19BreamoreBoysetstatus: pending -> open

messages: + msg110827
2010-07-19 22:44:40brian.curtinsetstatus: closed -> pending

nosy: + brian.curtin
messages: + msg110826

resolution: wont fix -> (no value)
2010-07-19 22:38:05BreamoreBoysetstatus: open -> closed
resolution: wont fix
messages: + msg110823
2010-07-15 22:08:47BreamoreBoysetnosy: + BreamoreBoy
messages: + msg110391
2009-07-01 14:00:42amaury.forgeotdarclinkissue1303673 superseder
2009-07-01 14:00:42amaury.forgeotdarcunlinkissue1303673 dependencies
2009-07-01 13:45:24amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg89975
2009-03-20 22:35:11ajaksu2linkissue1303673 dependencies
2009-02-14 11:32:44ajaksu2setversions: + Python 2.6, - Python 2.5
nosy: + ajaksu2
title: hotshot.stats.load -> hotshot.stats.load fails with AssertionError
messages: + msg82011
keywords: + patch
type: behavior
stage: patch review
2004-02-19 08:05:42sdahlbaccreate