classification
Title: Regression in pstats
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, belopolsky, doughellmann, ezio.melotti, fdrake, georg.brandl, j1m, matthew.fremont, stromnov, therve
Priority: normal Keywords: patch

Created on 2009-11-21 01:03 by belopolsky, last changed 2011-01-09 07:48 by georg.brandl. This issue is now closed.

Files
File name Uploaded Description Edit
bug.zip belopolsky, 2009-11-21 01:03
issue7372.diff belopolsky, 2009-11-21 02:20 Patch against revision 76430
Messages (10)
msg95574 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2009-11-21 01:03
Attached zip archive can be run directly to reproduce the bug:

$ ./python.exe bug.zip 
Traceback (most recent call last):
  File "/Users/sasha/Work/python-svn/trunk/Lib/runpy.py", line 162, in 
_run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/sasha/Work/python-svn/trunk/Lib/runpy.py", line 72, in 
_run_code
    exec code in run_globals
  File "bug.zip/__main__.py", line 9, in <module>
  File "/Users/sasha/Work/python-svn/trunk/Lib/pstats.py", line 268, in 
strip_dirs
    (cc, nc, tt, ct, newcallers))
  File "/Users/sasha/Work/python-svn/trunk/Lib/pstats.py", line 516, in 
add_func_stats
    add_callers(t_callers, callers))
  File "/Users/sasha/Work/python-svn/trunk/Lib/pstats.py", line 526, in 
add_callers
    zip(caller, new_callers[func])])
TypeError: zip argument #1 must support iteration


This is a regression introduced by r60149 which fixed processing of 
cProfile data, but broke processing of profile.

It is fairly easy to fix this problem in pstats, but a better solution 
may be to make profile output compatible with cProfile.

See also issue1269 and issue1315 .
msg95575 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2009-11-21 02:20
I am attaching a simple fix.  The test for data format could be taken out 
of the loop, but it is probably not worth complicating the logic.  In any 
case the real issue is whether profile and cProfile should produce 
compatible output or not.
msg95657 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-11-23 20:31
Alexander, how hard would it be to change the profile output?
msg95659 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2009-11-23 21:06
It would be fairly easy to change format before dumping stats to disk,
but unfortunately Profile does not encapsulate it's run-time data and
changing its format would be a mess.

Of course, one can hack around that by turning Profile.stats into a
computed property, but that's hardly an acceptable solution.

I think it is worthwhile fixing the on-disk format, but as a long-tem
solution one may consider adding common (possibly iterator-based) API
to both profile and cProfile that pstats can use instead of accessing
internal data structures.

Creating an ABC for a standard Python profiler, seems to be a
worthwhile project that in the future may simplify plugging in 3rd
party profilers, but I don't have expertise to undertake such a
project.  That would require experience with multiple Python profilers
that I don't have.

On Mon, Nov 23, 2009 at 3:31 PM, Georg Brandl <report@bugs.python.org> wrote:
>
> Georg Brandl <georg@python.org> added the comment:
>
> Alexander, how hard would it be to change the profile output?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue7372>
> _______________________________________
>
msg98306 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-26 00:35
See also #7372.

AFAIU add_callers can receive either a tuple or an int (and this is what is not working now). When I looked at #7372 I wasn't able to find out why it might receive two different objects and not always the same (i.e. always a tuple, possibly with only one int). From the first message it seems that cProfile returns the tuple of 4 ints and profile returns a single int.
Can profile be fixed to return a tuple (of 4 or 1 ints) too?
msg98307 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-26 00:43
I meant "See also #6710".
msg98331 - (view) Author: Jim Fulton (j1m) * (Python committer) Date: 2010-01-26 12:04
On Mon, Jan 25, 2010 at 7:35 PM, Ezio Melotti <report@bugs.python.org> wrote:
>
> Ezio Melotti <ezio.melotti@gmail.com> added the comment:
>
> See also #7372.
>
> AFAIU add_callers can receive either a tuple or an int (and this is what is not working now). When I looked at #7372 I wasn't able to find out why it might receive two different objects and not always the same (i.e. always a tuple, possibly with only one int). From the first message it seems that cProfile returns the tuple of 4 ints and profile returns a single int.
> Can profile be fixed to return a tuple (of 4 or 1 ints) too?

Ah interesting.  I ran into this recently when I noticed that some
analysis tools I'd written a while back no longer worked and had to be
updated because data that were ints were now tuples.

Aside from these particular issues, it would be really great if the
profiler stats data structure was specified. :)

Jim
msg112502 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-08-02 17:24
Fixed with your patch in r83531.
msg125796 - (view) Author: Doug Hellmann (doughellmann) * (Python committer) Date: 2011-01-08 19:36
I just ran into this problem with pstats under Python 2.7.  The ticket is marked as "fixed", but it looks like the change was only checked in under the py3k branch.

What's the policy on fixing stuff like this for 2.7 patch releases?

Thanks,
Doug
msg125831 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-01-09 07:48
This change was backported to the 2.7 branch in r83533; it should therefore be in 2.7.1, but not 2.7.
History
Date User Action Args
2011-01-09 07:48:01georg.brandlsetnosy: fdrake, georg.brandl, belopolsky, therve, alexandre.vassalotti, stromnov, matthew.fremont, ezio.melotti, doughellmann, j1m
messages: + msg125831
2011-01-08 19:36:58doughellmannsetnosy: + doughellmann
messages: + msg125796
2010-08-02 17:24:59georg.brandlsetstatus: open -> closed
resolution: fixed
messages: + msg112502
2010-01-26 12:04:27j1msetmessages: + msg98331
2010-01-26 00:43:56ezio.melottisetmessages: + msg98307
2010-01-26 00:35:53ezio.melottisetpriority: normal

nosy: + fdrake, alexandre.vassalotti, ezio.melotti, j1m
messages: + msg98306

type: crash -> behavior
stage: needs patch
2010-01-26 00:25:36ezio.melottilinkissue6710 superseder
2009-11-23 21:06:19belopolskysetmessages: + msg95659
2009-11-23 20:31:34georg.brandlsetmessages: + msg95657
2009-11-21 16:06:02belopolskysetversions: + Python 3.1, Python 3.2
2009-11-21 02:20:42belopolskysetfiles: + issue7372.diff
keywords: + patch
messages: + msg95575
2009-11-21 01:03:10belopolskycreate