classification
Title: hotshot.stats.load fails with AssertionError
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ajaksu2, amaury.forgeotdarc, barry, brett.cannon, glchapman, gsakkis, jlgijsbers, sdahlbac (8)
Priority: normal Keywords patch

Created on 2004-02-19 08:05 by sdahlbac, last changed 2009-07-01 13:45 by amaury.forgeotdarc.

Files
File name Uploaded Description Edit Remove
900092-patch.txt barry, 2005-07-07 23:26 Patch candidate for Python 2.4.x
900092-patch-2.txt barry, 2005-07-08 15:02 Test suite patch
bug.prof brett.cannon, 2007-02-10 01:48 crasher from gsakkis
Messages (15)
msg20042 - (view) Author: Simon Dahlbacka (sdahlbac) Date: 2004-02-19 08:05
trying to do a 

hotshot.stats.load("myprofiling_file.prof")

fails with assertionerror

assert not self._stack


python 2.3.2 on WinXP

msg20043 - (view) Author: Johannes Gijsbers (jlgijsbers) Date: 2004-09-24 21:58
Logged In: YES 
user_id=469548

While the original report isn't very useful, I've ran into
this problem multiple times as well. I can reproduce it
using the attached profile file (compressed because of the
large size) and the following console session:

Python 2.3.4 (#2, Jul  5 2004, 09:15:05)
[GCC 3.3.4 (Debian 1:3.3.4-2)] on linux2
Type "help", "copyright", "credits" or "license" for more
information.
>>> import hotshot.stats
>>> stats = hotshot.stats.load('roundup.prof')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.3/hotshot/stats.py", line 12, in load
    return StatsLoader(filename).load()
  File "/usr/lib/python2.3/hotshot/stats.py", line 51, in load
    assert not self._stack
AssertionError
>>>

I'm not sure who's baby hotshot really is, so I'm leaving
this unassigned.
msg20044 - (view) Author: Johannes Gijsbers (jlgijsbers) Date: 2004-09-24 22:00
Logged In: YES 
user_id=469548

Hmm, the file was too big, even though it was compressed.
I've uploaded it to
http://home.student.uva.nl/johannes.gijsbers/roundup.prof.bz2
now.
msg20045 - (view) Author: Barry A. Warsaw (barry) * Date: 2004-09-27 14:41
Logged In: YES 
user_id=12800

Could this be related to 1019882?
msg20046 - (view) Author: Greg Chapman (glchapman) Date: 2004-11-08 23:32
Logged In: YES 
user_id=86307

I ran into this today, so I decided to look into it.  It
looks to me like the problem only happens if you profile
with lineevents enabled.  In that case, hotshot uses the
tracer_callback function (in _hotshot.c) to dispatch trace
events.  This function explicitly ignores exception returns
(PyTrace_EXCEPTION), which can lead to an unbalanced stack
of calls/returns when the log is loaded (if an profiled
function exits with an exception).

It seems on the surface that tracer_callback ought to handle
exceptions the same way as normal returns.  This would be
consistent with what happens when profiler_callback is used,
since PyEval_EvalFrame dispatches sends a Py_RETURN to
c_profilefunc when exiting because of an exception.
msg20047 - (view) Author: Greg Chapman (glchapman) Date: 2004-11-08 23:54
Logged In: YES 
user_id=86307

Well, the superficial fix doesn't work (sorry for posting
too soon).  It turns out that PyTrace_EXCEPTION is sent for
any exception, not just one causing the function to exit. 
So I guess the best fix may be to have hotshot always
install its profiler_callback to handle CALLS and RETURNS,
and then optionally install the tracer_callback to handle
only LINEs.  Anyway, that works for the one test case I've
been using (a runcall of a function which simply does
"import pickle").

By the way, I'm testing with 2.4b1.
msg20048 - (view) Author: Barry A. Warsaw (barry) * Date: 2005-07-07 23:26
Logged In: YES 
user_id=12800

See 900092-patch.txt for a candidate patch against Python
2.4.1.  Props to Justin Campbell for most of the heavily
lifting (but you can blame me for any problems ;).

This fix restore the tracing of a 'return' event for
exceptions that cause a function to exit.
msg20049 - (view) Author: Barry A. Warsaw (barry) * Date: 2005-07-08 15:02
Logged In: YES 
user_id=12800

900092-patch-2.txt fixes the test suite for the extra return
event.
msg20050 - (view) Author: Barry A. Warsaw (barry) * Date: 2005-08-15 18:14
Logged In: YES 
user_id=12800

Patches applied for Python 2.4.2 and 2.5a1
msg20051 - (view) Author: George Sakkis (gsakkis) Date: 2007-02-08 23:37
Has this ever been reported again since it was closed ? I just got it today (python 2.5). The prof file is 11MB, I'll see if I can reproduce the bug with a smaller one.
msg20052 - (view) Author: George Sakkis (gsakkis) Date: 2007-02-09 00:08
Ok, I reduced the prof file to 38K. Can I upload an attachment here ? I don't see how.
msg20053 - (view) Author: Brett Cannon (brett.cannon) * Date: 2007-02-09 18:15
I think only OPs and project members can upload.  Email it to me, George (brett at python.org), mention the bug # and I will upload it.
msg20054 - (view) Author: Brett Cannon (brett.cannon) * Date: 2007-02-10 01:48
Attaching the file George made.
File Added: bug.prof
msg82011 - (view) Author: Daniel Diniz (ajaksu2) Date: 2009-02-14 11:32
Has test, patch and sample data for reproducing.
msg89975 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) Date: 2009-07-01 13:45
I added some prints in the hotshot.stats.load function, this lead to
interesting things:

- first, the program seems to spawn subprocesses which trace themselves
in the same file.

- however, there is real issue when fork() is called on a traced
program: calls to _execvpe should not appear in the trace!
IMO PyOS_AfterFork should disable any trace or profile function.
History
Date User Action Args
2009-07-01 14:00:42amaury.forgeotdarcunlinkissue1303673 dependencies
2009-07-01 14:00:42amaury.forgeotdarclinkissue1303673 superseder
2009-07-01 13:45:24amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg89975
2009-03-20 22:35:11ajaksu2linkissue1303673 dependencies
2009-02-14 11:32:44ajaksu2setversions: + Python 2.6, - Python 2.5
nosy: + ajaksu2
title: hotshot.stats.load -> hotshot.stats.load fails with AssertionError
messages: + msg82011
keywords: + patch
type: behavior
stage: patch review
2004-02-19 08:05:42sdahlbaccreate