classification
Title: Add Systemtap/DTrace probes
Type: enhancement Stage: patch review
Components: Build Versions: Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder: Add DTrace probes
View: 13405
Assigned To: dmalcolm Nosy List: BreamoreBoy, Garen, belopolsky, chrismiles, danchr, dhduvall, dmalcolm, fche, glyph, hazmat, jbaker, jcea, laca, mjw, movement, pitrou, rhettinger, robert.kern, ronaldoussoren, serverhorror, sirg3, techtonik, twleung, wsanchez
Priority: normal Keywords: needs review, patch

Created on 2008-10-12 16:08 by bretthoerner, last changed 2012-05-11 20:59 by dmalcolm. This issue is now closed.

Files
File name Uploaded Description Edit
python-2.6-dtrace.diff bretthoerner, 2008-10-12 16:08 dtrace patch against python 2.6 (with rude/broken Makefile)
unnamed twleung, 2009-01-22 21:41
dtrace.diff skip.montanaro, 2009-01-25 22:00
unnamed twleung, 2009-04-22 21:21
stap-python.patch mjw, 2009-12-08 21:27 Cleaned up systemtap/dtrace patch
add-dtrace-or-systemtap-markers-to-trunk.patch dmalcolm, 2010-02-26 20:45 The patch I'm applying in Fedora 13, regenerated against SVN trunk
python-3.1.1-systemtap.patch dmalcolm, 2010-02-26 20:51 The patch against 3.1.1 I'm applying in Fedora 13
smime.p7s ronaldoussoren, 2010-07-19 07:45
py3k-add-systemtap-2010-09-27.patch dmalcolm, 2010-09-27 23:50 Updated version of patch, against py3k (r85046)
py3k-add-systemtap-2010-10-25.patch dmalcolm, 2010-10-25 16:27
py3k-add-systemtap-2010-10-25-002.patch dmalcolm, 2010-10-25 21:20
py3k-add-systemtap-2010-10-25-003.patch dmalcolm, 2010-10-25 22:52
Messages (85)
msg74670 - (view) Author: Brett Hoerner (bretthoerner) Date: 2008-10-12 16:08
It would be great if the main Python distribution supported DTrace on
(Open)Solaris / FreeBSD / OS X.

I've attached a patch against 2.6 that instruments function calls.  It's
from the ed scripts included in Apple's Python distribution:
http://developer.apple.com/opensource/index.html (PSF License)

It is imperfect because I wasn't sure how to do the equivalent of IFDEF
in the Makefile.pre.in, as you can see this patch will make ceval.o
depend on pydtrace.h, which depends on having a local `dtrace` command.
 The rest of the patch properly uses IFDEFs.
msg74676 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-10-12 18:48
At one point we were told Apple would try to backport their dtrace
instrumentation. I don't know what the status of that is, but it
obviously has not happened.
msg75809 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2008-11-13 02:45
It appears that Apple has dtracified their Python exeutable in Leopard.
Any chance that they can be persuaded to release a patch?  I'm working
on a patch (based on some work from a friend at work), but it seems like
Apple already has a horse in the race.  It would be nice if we could
see what they have done.
msg75810 - (view) Author: Brett Hoerner (bretthoerner) Date: 2008-11-13 02:57
They have released the changes, that's what my patch (attached to the
issue) is based on.

It's working, it just needs to be cleaned up (it will fail, I believe,
for people on systems without DTrace - as I said I'm not very familiar
with Makefiles).

You can use it now against 2.6 and probably with very few changes
against 2.5.
msg75811 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2008-11-13 03:31
Brett> They have released the changes, that's what my patch (attached to
    Brett> the issue) is based on.

I see the reference to Apple in your original post, but can't find anything
related to dtrace & python starting from the URL you gave.  Do you have
something more specific?

At this point Jeff's code does a fair bit more than simply tracing the
CALL_FUNCTION opcode but needs some work with the other CALL_FUNCTION_*
opodes.  It also has some obmalloc tracing which I've not yet tested.  I
would have thought Apple would more heavily instrument the interpreter than
it appears they have.

    Brett> It's working, it just needs to be cleaned up (it will fail, I
    Brett> believe, for people on systems without DTrace - as I said I'm not
    Brett> very familiar with Makefiles).

I took care of the configure.in/Makefile.pre.in stuff today.  The intent is
that you would configure using --enable-dtrace then not have to do anything
else to build a dtrace-aware interpreter.

    Brett> You can use it now against 2.6 and probably with very few changes
    Brett> against 2.5.  

We are using 2.4 at work and it works there.  I'm fairly certain we should
be able to get it working for the entire 2.x and 3.x series with only a
little effort.

(I still need to get approval to release it, but I don't think that will be
a big deal.  I've already alerted my boss and he's generally receptive to
such things.)

Skip
msg75825 - (view) Author: Brett Hoerner (bretthoerner) Date: 2008-11-13 15:09
On Wed, Nov 12, 2008 at 9:31 PM, Skip Montanaro <report@bugs.python.org> wrote:
> I see the reference to Apple in your original post, but can't find anything
> related to dtrace & python starting from the URL you gave.  Do you have
> something more specific?

http://www.opensource.apple.com/darwinsource/10.5.5/python-30.1.2/

That isn't Python 3.0, that's what I guess they call Python "Apple
Version 30"?  Anyways, it's their Python 2.5.1 and their "patches"
against it are actually ed scripts, contained in fix/

That's where my patch comes from, it's just changing the .ed scripts
to a patch and applying it against 2.6

> At this point Jeff's code does a fair bit more than simply tracing the
> CALL_FUNCTION opcode but needs some work with the other CALL_FUNCTION_*
> opodes.  It also has some obmalloc tracing which I've not yet tested.  I
> would have thought Apple would more heavily instrument the interpreter than
> it appears they have.

Sun has released a patch against DTrace that probes more than just
function calls also.  At least, it provides line number and some other
information, http://cvs.opensolaris.org/source/xref//jds/spec-files/trunk/patches/Python-07-dtrace.diff

I couldn't figure out why it broke the compile on OS X though, and
could only get it working on Solaris.

There's another problem (I think) with DTrace probes ... if the Apple
guys release Apple-Probe-Python.d and the Sun guys release
Sun-Probe-Python.d (just two scripts) and we setup our probes
differently, one or both will fail (because one may expect a probe for
foo while another expects a probe for bar).  Kind of sucks,
considering Sun was first and Apple chose to probe Python differently.
 Now we have to pick one or make a 3rd... I could be very mistaken
here, though.
msg75828 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2008-11-13 16:05
Brett> http://www.opensource.apple.com/darwinsource/10.5.5/python-30.1.2/
    ...
    Brett> http://cvs.opensolaris.org/source/xref//jds/spec-files/trunk/patches/Python-07-dtrace.diff

Thanks for the pointers.  I'll work on getting a uniform patch which
incorporates both.  (Though some of the comments in the OpenSolaris patch
are a bit scary.)

    Brett> There's another problem (I think) with DTrace probes ... if the
    Brett> Apple guys release Apple-Probe-Python.d and the Sun guys release
    Brett> Sun-Probe-Python.d (just two scripts) and we setup our probes
    Brett> differently, one or both will fail (because one may expect a
    Brett> probe for foo while another expects a probe for bar).  Kind of
    Brett> sucks, considering Sun was first and Apple chose to probe Python
    Brett> differently.

Yeah, that would suck.  It would be nice if they could agree on a common set
of probes.  I don't know that we have any contacts within the two
development communities but if we can scare some up maybe we can get them to
talk to each other. :-/

Skip
msg75832 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-11-13 18:55
On Thu, Nov 13, 2008 at 08:05, Skip Montanaro <report@bugs.python.org> wrote:
>
> Skip Montanaro <skip@pobox.com> added the comment:
>
> Brett> http://www.opensource.apple.com/darwinsource/10.5.5/python-30.1.2/
>    ...
>    Brett> http://cvs.opensolaris.org/source/xref//jds/spec-files/trunk/patches/Python-07-dtrace.diff
>
> Thanks for the pointers.  I'll work on getting a uniform patch which
> incorporates both.  (Though some of the comments in the OpenSolaris patch
> are a bit scary.)
>
>    Brett> There's another problem (I think) with DTrace probes ... if the
>    Brett> Apple guys release Apple-Probe-Python.d and the Sun guys release
>    Brett> Sun-Probe-Python.d (just two scripts) and we setup our probes
>    Brett> differently, one or both will fail (because one may expect a
>    Brett> probe for foo while another expects a probe for bar).  Kind of
>    Brett> sucks, considering Sun was first and Apple chose to probe Python
>    Brett> differently.
>
> Yeah, that would suck.  It would be nice if they could agree on a common set
> of probes.  I don't know that we have any contacts within the two
> development communities but if we can scare some up maybe we can get them to
> talk to each other. :-/
>

Obviously Ronald is the Apple insider to talk to. If you want Sun, you
might want to talk to Ted Leung and see who he can put you in touch
with.
msg75834 - (view) Author: Ted Leung (twleung) Date: 2008-11-13 19:59
And courtesy of Philip Jenvey, here I am.

I would *really* like to work to help make this happen.   Laszlo Peter
at Sun has been doing the ports of Python on Solaris, but we are not up
to 2.6 just yet.  I'm attaching a pointer to his patches against 2.5:

http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/Python25-07-dtrace.diff

These patches include John Levon's ustack provider, which if you care
about DTrace and Python, you want to have.

I've also had some conversations with people at Apple, and they have
agreed that they will pull DTrace probes from python.org if they got in
there.  That would solve the diverging probe problem.

I'd love to have a discussion about DTrace probe futures for CPython --
probably on python-dev.
msg75849 - (view) Author: Laszlo (Laca) Peter (laca) Date: 2008-11-14 06:02
I'm the python package maintainer at Sun.
We would really like to get the dtrace probes upstream, let me know how
I can help.
msg80381 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-01-22 21:35
So I completely dropped the ball on this.  It appears we have some
folks from Sun and Brett surmised that Ronald Oussoren would be the
likely person to do the heavy lifting on the Apple side of things.
Ronald, I've made you nosy.  I will try to get the Solaris patch Ted
referenced running against the trunk (2.7) and py3k (3.1) code bases,
but it would be helpful if you could take a quick stab at it as well.
msg80382 - (view) Author: Ted Leung (twleung) Date: 2009-01-22 21:41
On Jan 22, 2009, at 1:35 PM, Skip Montanaro wrote:

>
> Skip Montanaro <skip@pobox.com> added the comment:
>
> So I completely dropped the ball on this.  It appears we have some
> folks from Sun and Brett surmised that Ronald Oussoren would be the
> likely person to do the heavy lifting on the Apple side of things.
> Ronald, I've made you nosy.  I will try to get the Solaris patch Ted
> referenced running against the trunk (2.7) and py3k (3.1) code bases,
> but it would be helpful if you could take a quick stab at it as well.

Great.  Actually the latest version of the patch is this one: <http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/Python26-07-dtrace.diff 
 >.   I tried to quickly compile it on MacOS 10.5, Python 2.6.1, but  
it didn't work right out of the box.

Ted
msg80388 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-01-22 22:53
After applying the patch and reconfiguring I get compilation errors in
Python/ceval.c.  I suspect it's because there is a new header file,
Python/python.h.  That's probably found by the #include directive in
favor of Include/Python.h because of the Mac's case-insensitive file
system.  Changing the Makefile targets and actions so it's named
python-dtrace.h seems to work a bit better.  I get an error later
running dtrace which I have yet to investigate.
msg80395 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-01-23 02:00
me> I get an error later running dtrace which I have yet to investigate.

Apple's dtrace program doesn't support the -G flag.  When I remove it from
Makefile.pre.in and rebuild I get an error about privileges:

    dtrace -o Python/dtrace.o  -C -s ./Python/python.d Python/ceval.o
    dtrace: failed to initialize dtrace: DTrace requires additional privileges

I tried adding "sudo" to the dtrace command.  Then it prompts for my
password and emits this error:

    sudo dtrace -o Python/dtrace.o  -C -s ./Python/python.d Python/ceval.o
    Password:
    dtrace: failed to compile script ./Python/python.d: line 11: extraneous argument 'Python/ceval.o' ($1 is not referenced)

I'm not sure what to do at this point.  I'm not really a dtrace person.
Perhaps the Apple and Sun dtrace experts can offer a way out of this little
corner.

Skip
msg80436 - (view) Author: Laszlo (Laca) Peter (laca) Date: 2009-01-24 05:17
Please see here for discussion about the -G flag on OS X:
http://markmail.org/message/4nheqnexjr2o6mcx

If I read it correctly, on OS X, you will need to use -h
instead of -G and it won't emit an object file (dtrace.o)
so you will not need to link it.

Unfortunately, this means that the makefile will have to
be different on Solaris and OS X :(
msg80447 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-01-24 13:46
Laca> Please see here for discussion about the -G flag on OS X:
    Laca> http://markmail.org/message/4nheqnexjr2o6mcx

    Laca> If I read it correctly, on OS X, you will need to use -h instead
    Laca> of -G and it won't emit an object file (dtrace.o) so you will not
    Laca> need to link it.

    Laca> Unfortunately, this means that the makefile will have to be
    Laca> different on Solaris and OS X :(

We can worm around that using configure.  Thanks for the reference.  That
will go a long way toward getting the Sun dtrace probes working on Mac OSX.
For reference by others, here's Lee Packham's dtrace patch for PostgreSQL on
Mac OSX: http://blog.leenux.org.uk/wp-content/uploads/2008/10/postgresql-825-fixosxdtracediff.gz
msg80542 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-01-25 22:00
Here's a patch against the current trunk (2.7) which compiles on my Mac.  It
adds a --with-dtrace configure option.  The code checks to see if the -G
option is understood by the dtrace command.  If so, dtrace support is added
Sun-style.  If not we do things Apple's way.

The existing test cases pass on Apple except for one case in test_sys which
tries to confirm the size of a frame object.  I added an #ifdef WITH_DTRACE
around the extra slot but there is no way of telling from Python code that
this extra slot is there.  That would have to somehow be exposed to the
Python programmer so the test can be adjusted.  All tests pass when
--with-dtrace is omitted.  I have not yet tried this on Solaris.  I will try
to get to it this week if someone doesn't beat me to it.

There are as yet no new dtrace test cases so I can't confirm that the added
dtrace support actually works.  There are also no documentation updates.
Unlike most compile options this adds a significant new feature to the
runtime environment so some documentation changes are probably called for.
msg80671 - (view) Author: Ted Leung (twleung) Date: 2009-01-27 22:05
I tried building this on my Mac and got this;

ar cr libpython2.7.a Python/_warnings.o Python/Python-ast.o Python/ 
asdl.o Python/ast.o Python/bltinmodule.o Python/ceval.o Python/ 
compile.o Python/codecs.o Python/errors.o Python/frozen.o Python/ 
frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o  
Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/ 
graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/ 
modsupport.o Python/mystrtoul.o Python/mysnprintf.o Python/peephole.o  
Python/pyarena.o Python/pyfpe.o Python/pymath.o Python/pystate.o  
Python/pythonrun.o Python/structmember.o Python/symtable.o Python/ 
sysmodule.o Python/traceback.o Python/getopt.o Python/pystrcmp.o  
Python/pystrtod.o Python/formatter_unicode.o Python/formatter_string.o  
Python/dynload_shlib.o @DTRACEOBJS@  Python/mactoolboxglue.o Python/ 
thread.o
ar: @DTRACEOBJS@: No such file or directory
make: *** [libpython2.7.a] Error 1

my configure line was:

./configure --enable-framework --enable-toolbox-glue --with-threads -- 
enable-dtrace

On Jan 25, 2009, at 2:00 PM, Skip Montanaro wrote:

>
> Skip Montanaro <skip@pobox.com> added the comment:
>
> Here's a patch against the current trunk (2.7) which compiles on my  
> Mac.  It
> adds a --with-dtrace configure option.  The code checks to see if  
> the -G
> option is understood by the dtrace command.  If so, dtrace support  
> is added
> Sun-style.  If not we do things Apple's way.
>
> The existing test cases pass on Apple except for one case in  
> test_sys which
> tries to confirm the size of a frame object.  I added an #ifdef  
> WITH_DTRACE
> around the extra slot but there is no way of telling from Python  
> code that
> this extra slot is there.  That would have to somehow be exposed to  
> the
> Python programmer so the test can be adjusted.  All tests pass when
> --with-dtrace is omitted.  I have not yet tried this on Solaris.  I  
> will try
> to get to it this week if someone doesn't beat me to it.
>
> There are as yet no new dtrace test cases so I can't confirm that  
> the added
> dtrace support actually works.  There are also no documentation  
> updates.
> Unlike most compile options this adds a significant new feature to the
> runtime environment so some documentation changes are probably  
> called for.
>
> Added file: http://bugs.python.org/file12861/dtrace.diff
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue4111>
> _______________________________________
msg80679 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-01-27 23:39
Ted> I tried building this on my Mac and got this;

Forgive me if I'm preaching to the choir here.

Did you run autoconf or autoreconf after applying the patch?  If not,
@DTRACEOBJS@ would not be a substitutable string.  It's fairly common (at
least in the Python community) to omit modified configure scripts from these
sorts of patches because the changes to generated configure scripts between
different versions of autoconf are so massive that they dwarf the actual
functional changes in the patch, often by a couple orders of magnitude.

Skip
msg80691 - (view) Author: Ted Leung (twleung) Date: 2009-01-28 03:06
I didn't run auto(re)conf.  After I did that, all was well.  However,  
the ustack provider doesn't appear to be working correctly.   I tried  
running the py_profile.d from the DTrace toolkit, and it doesn't show  
any stack traces, and when the script starts up it says

cc1: warning: /dev/fd/5 is shorter than expected

The basic function entry/exit probes appear to be working.

John +nosy'ed himself, so perhaps he'll have some insight?

On Jan 27, 2009, at 3:39 PM, Skip Montanaro wrote:

>
> Skip Montanaro <skip@pobox.com> added the comment:
>
> Ted> I tried building this on my Mac and got this;
>
> Forgive me if I'm preaching to the choir here.
>
> Did you run autoconf or autoreconf after applying the patch?  If not,
> @DTRACEOBJS@ would not be a substitutable string.  It's fairly  
> common (at
> least in the Python community) to omit modified configure scripts  
> from these
> sorts of patches because the changes to generated configure scripts  
> between
> different versions of autoconf are so massive that they dwarf the  
> actual
> functional changes in the patch, often by a couple orders of  
> magnitude.
>
> Skip
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue4111>
> _______________________________________
msg81229 - (view) Author: John Levon (movement) Date: 2009-02-05 21:12
I haven't seen "shorter than expected" message before, sounds like a Mac
OS X specific thing.

As for never getting any probes out, this is where it gets fun.
Debugging this is very tricky indeed: it involves you dropping into the
kernel debugger and looking at the interpreter's state. Most likely, the
compiler is doing something different with the relevant Python
functions, causing the logic in the DTrace ustack probe to fail.

I don't even know how the Apple guys debug this sort of thing, but
they'd definitely need to be involved.
msg86273 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-22 06:12
Skip, it doesn't appear that the ustack helper is getting incorporated
into the OS X build anywhere. This rule is obviously wrong (compiling
the wrong input file with the wrong flags):

Include/phelper.h: $(srcdir)/Include/phelper.d
	dtrace -o $@ $(DFLAGS) -C -h -s $(srcdir)/Python/python.d

I *think* it should be something along these lines:

Include/phelper.h: $(srcdir)/Include/phelper.d
	dtrace -o $@ -DPHELPER $(DFLAGS) $(CPPFLAGS) -C -h -s
$(srcdir)/Include/phelper.d

There are some static functions in the standard system headers that get
pulled in that need to be avoided. The #define _SYS_STAT_H up at the top
avoids similar problems on Solaris. I'm working through the OS X headers
now, but I'm stuck on trying to avoid sys/_structs.h. Unfortunately, it
does not have the #ifndef _SYS__STRUCTS_H magic up at the top that would
allow me to do so, nor any convenient #ifdefs around the offending
declaration. <sigh>

Even if that would work, phelper.h is not #included anywhere. I'm not
really sure where it needs to go. On Solaris, the object file gets
linked in, but on OS X, you don't generate an object file; you just
#include the .h. Somewhere. I'm guessing where the pydtrace.h is
currently, but I don't really know. John, do you have any insights?

Also, s/DTRADEHDRS/DTRACEHDRS/
msg86277 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-22 06:41
Got a bit farther. Adding this stanza to the top of phelper.d gets past
the issues in the headers:

#ifdef __APPLE__
#define _SYS_TIME_H_
#define _SYS_SELECT_H_
#define __MATH_H__
#define _OS__OSBYTEORDER_H
#define _FD_SET
#define __GNUC_VA_LIST
#endif /* __APPLE__ */


However, I now get a more legitimate dtrace compilation error that John
might be able to help us interpret:

$ dtrace -o /Users/rkern/hg/Python-2.5.4/Include/phelper.h -DPHELPER 
-I. -IInclude -I/Users/rkern/hg/Python-2.5.4/Include  -C -h -s
/Users/rkern/hg/Python-2.5.4/Include/phelper.d
dtrace: failed to compile script
/Users/rkern/hg/Python-2.5.4/Include/phelper.d: line 110: relocation
remains against user symbol D``PyEval_EvalFrameEx (offset 0x5)


I also tried running this without -DPHELPER as a regular DTrace script
rather than a ustack helper and ran into a problem that I've noticed
with any OS X Python build I've tried. I cannot seem to probe any of the
C functions in the Python interpreter. There are no simply
pid$target:a.out:: probes available. I'm wondering if that is an effect
of their being in a .framework, but I think I've been able to probe
other symbols other .frameworks.
msg86278 - (view) Author: Laszlo (Laca) Peter (laca) Date: 2009-04-22 07:34
Look at these lines at the beginning of phelper.d:

#if defined(__i386)
#define        startframe PyEval_EvalFrameEx
#define        endframe PyEval_EvalCodeEx
#elif defined(__amd64)
#define        PyEval_EvalFrameEx PyEval_EvalFrameExReal
#define        startframe PyEval_EvalFrameExReal
#define        endframe PyEval_EvalCodeEx
#elif defined(__sparc)
#define        PyEval_EvalFrameEx PyEval_EvalFrameExReal
#define        startframe PyEval_EvalFrameEx
#define        endframe PyEval_EvalFrameExReal
#endif

You may need to adjust these if your compiler defines
something different from the __i386 or __amd64.
I guess an

#else
#error your architecture was not recognized
#endif

would be useful here.
msg86279 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-22 07:45
This is on an Intel Core 2 Duo running OS X 10.5.6. __i386 is defined.
msg86290 - (view) Author: John Levon (movement) Date: 2009-04-22 12:10
Robert, I have no idea how Mac OS does pstack helpers without generating
object files, sorry.

> no simply pid$target:a.out:: probes available.

Hmm. Try adding -Z to see if that helps.

> /Users/rkern/hg/Python-2.5.4/Include/phelper.d: line 110: relocation
> remains against user symbol D``PyEval_EvalFrameEx (offset 0x5)

This is trying to tell you that there's no such function. Indeed, this
isn't mentioned in my original patches, have you been editing it?
msg86322 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-22 20:00
John, -Z does not appear to help:

$ sudo dtrace -Z -n 'pid$target::PyEval_EvalFrameEx:entry' -c python   
dtrace: description 'pid$target::PyEval_EvalFrameEx:entry' matched 0 probes

I'm not sure how that would help. If I'm reading the man page correctly,
that just let's dtrace continue on even when the probe specifiers do not
match anything. It still doesn't let me probe these functions.

The only modification I made to phelper.d was to add the stanza of
#defines given a few messages up. The failing line expands in the C
preprocessor to this line (expanding at_evalframe(), then startframe and
endframe):

dtrace:helper:ustack: /((uintptr_t)arg0 >=
((uintptr_t)&``PyEval_EvalFrameEx) &&      (uintptr_t)arg0 < (
(uintptr_t)&``PyEval_EvalCodeEx))/ 

This is based on the phelper.d in dtrace.diff attached to this ticket by
Skip. It is the same (modulo my #defines) as the phelper.d in
Python26-07-dtrace.diff from OpenSolaris as linked by Ted.

Hmm, I am applying this to Python 2.5.4, though. Perhaps I need to go
back to the Python25-07-dtrace.diff, which does appear to use
PyEval_EvalFrame. PyEval_EvalFrameEx is in Python 2.5, so I thought I
would try the latest and greatest.

If I were to process it with dtrace -h, do you think that I could name
it as a .c and compile it into an object file with gcc? Or does dtrace
-G do significantly more stuff behind the scenes to generate its object
file?
msg86326 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-04-22 20:24
Sorry, I've been away from this issue.  I was sort of hoping the Sun and
Apple folks would just work things out amongst themselves and present us
with a fait accompli. ;-) I'll try to mess around with this a little.

    Robert> $ sudo dtrace ...

Perhaps not quite on-topic for this tracker item, but it bugs me that the
mere compilation of a D script requires root privileges.  I guess there's
not much we can do about this other than complain to Sun and Apple though.

Skip
msg86330 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-22 20:42
Skip> Perhaps not quite on-topic for this tracker item, but it bugs me
that the
Skip> mere compilation of a D script requires root privileges.

It doesn't. "dtrace -G" and "dtrace -h" (the only "mere compilation"
that dtrace does) run without root privileges. That line you quoted was
actually probing a process and thus needed root privileges.
msg86335 - (view) Author: John Levon (movement) Date: 2009-04-22 21:17
Yes, my mistake. I noticed this typo in the original patch:

    207 +PyObject *
    208 +PyEval_EvalFrameexEx(PyFrameObject *f, int throwflag))

Can you:

 - verify that HAVE_DTRACE is indeed defined for ceval.c
 - do an nm on ceval.o and look which Eval* functions you do have

Somehow, PyEval_EvalFrameEx seems not to be ending up in the .o file. It
should be easy for you to work out why...
msg86337 - (view) Author: Ted Leung (twleung) Date: 2009-04-22 21:21
On Apr 22, 2009, at 1:24 PM, Skip Montanaro wrote:

>
> Skip Montanaro <skip@pobox.com> added the comment:
>
> Sorry, I've been away from this issue.  I was sort of hoping the Sun  
> and
> Apple folks would just work things out amongst themselves and  
> present us
> with a fait accompli. ;-) I'll try to mess around with this a little.

I've been in touch with some folks at Apple about this, but it's not  
at the very top of their priority list....
msg86338 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-22 21:26
Ah, duh, of course. The problem here with PyEval_EvalFrameEx is that I
don't have ceval.o on the command line *at all* since OS X's dtrace
doesn't support -G. It doesn't appear to accept ceval.o with -h, either,
so I suppose that adding the ustack helper may simply be impossible
unless Apple adds the -G flag.

Thanks for you help John.
msg86343 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-22 22:12
Hmm, wait a second. Never mind. The Solaris patches don't have ceval.o
on the line for compiling phelper.o, either.

If dtrace needs to resolve the symbol PyEval_EvalFrameEx in an object
file, how does it know to look in ceval.o for the phelper.d? WITH_DTRACE
is defined in ceval.c and PyEval_EvalFrameEx is in ceval.o, but I don't
understand how that relates to compiling phelper.d.
msg86384 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-23 23:43
Is there any interest in my expanding the list of probes? Ruby has quite
a few more than function-entry and function-return, to give some
examples of what is possible:

http://dev.joyent.com/projects/ruby-dtrace/wiki/Ruby+DTrace+probes+and+arguments

I think that adding probes that correspond to PyTrace_LINE and
PyTrace_EXCEPTION would be straightforward and worthwhile. PyTrace_C_*
may also be worthwhile, but you can probably accomplish similar things
with the normal pid probes if you know the C function names (although
something like printing the name of a raised exception will probably
require a dedicated probe).

Adding probes to replicate what the LLTRACE configuration option did,
but dynamically, might be interesting.
msg86385 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2009-04-23 23:46
> Is there any interest in my expanding the list of probes? 

Definitively!!!.
msg86389 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-04-24 02:03
Robert> Is there any interest in my expanding the list of probes? 

Yes.  Jeff Garrett (a guy I work with) added some more DTrace probes to a
2.4 source tree at work.  I mentioned them in an earlier message.  I'll
check with him at work tomorrow and see about making sure they can be
released.

I think it's important to get Sun and Apple in sync as well.  Maybe more
important than adding new probes.  (Though if the Python community drives
the creation of new probes perhaps both will pick them up.)

Skip
msg86392 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-24 03:24
We could probably merge Apple's and Sun's probes without too much
trouble. Apple simply extended function-entry to include the argcount in
addition to Sun's (filename, funcname, lineno) arguments. We could use
Apple's probe while retaining compatibility with Sun-oriented DTrace
scripts already in existence.

The two different function-returns are a bit of a problem. I recommend
starting with Sun's argument list because there are already a number of
quite useful scripts that use it in the DTraceToolkit. We could extend
Sun's argument list to add the *object pointer, but I don't actually
know of a use case. I haven't seen a script in the wild that uses it. It
seems like it would be tricky to write something in a DTrace script that
could make much use of it besides printing out the address. Maybe you
could navigate your way down to the type name, but that might be
unreliable. I suggest adding the *object pointer argument only if we can
devise a use case or if one of the Apple folks pop up with one. I don't
feel too bad making Apple devs modify a single character from their
internal scripts if they don't make them public.
msg86396 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-24 05:02
James McIlree from Apple has informed me on dtrace-discuss that ustack
helpers cannot currently be built on OS X. Bummer.
msg86399 - (view) Author: Robert Kern (robert.kern) Date: 2009-04-24 06:45
Ah, I misread the Apple function-return probe code. Its extra argument
is the type name of the return object or "error" if an exception was
raised, not the returned object itself. Could be useful.
msg96154 - (view) Author: Mark Wielaard (mjw) Date: 2009-12-08 21:27
I took the patch and tweaked it a little so that it works with systemtap
(at least for the function entry/exit probes). It is against 2.6.2 since
that was what was in the Fedora 12 src package I was using. Cleaned up
the configure checks a little, but otherwise it is almost exactly as is.

Also tracked in Fedora as 
https://bugzilla.redhat.com/show_bug.cgi?id=545179
msg99986 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-02-24 01:23
can I ask if this (very useful) feature is on time for Python 2.7?
msg100041 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2010-02-24 15:56
Jesus> can I ask if this (very useful) feature is on time for Python 2.7?

You can ask, but I suspect you'd be disappointed in the answer.  Do you have
time to look at the issue?  The biggest sticking point in my mind is coming
up with a uniform set of probes which would be supported by both Sun and
Apple.

Skip
msg100169 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-02-26 19:00
I have no experience integrating DTrace probes, so I guess would be a bit risky to trying to deal with this myself, since 2.7 deadline is pretty close.

If the only remaining issue is deciding about supporting SUN or Apple "style", and we have the risk of delaying this so much that we lose the 2.7 train, would be better to decide in one direction (I don't know about the difference, so I can't talk) and ship "something".

That is, if there is no reason to decide for A or B, we should use a coin to decide, instead of losing the opportunity of 2.7. DTrace is an amazing technology!

What do you think?
msg100173 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-02-26 20:41
At Red Hat we've done some work on this feature.  I'm sorry for not updating this issue, I was swamped with tasks both pre-PyCon and during the event; I did show the following to various folks at PyCon; I attempted to find Skip at PyCon but was unsuccessful)

Unfortunately I have two further versions of this patch for you :(

High level summary
------------------
SystemTap is a similar tool for DTrace, but for Linux.

At Red Hat we've been looking at implementing this using SystemTap.  SystemTap makes some attempts to be at least somewhat compatible with DTrace, and we are able to get the DTrace support to build against SystemTap.

I measured the performance and found that some versions of the patch impose a significant performance penalty.

We have a version of the patch that works with SystemTap on Linux, and appears not to impose a performance penalty for the cases I think are important (see notes below)

I have also ported the patch to Python 3 (I pass the strings back to the probe support as UTF-8 encoded)

I hope that there's enough commonality between the SystemTap and DTrace patches to permit common parts to be committed to Python, perhaps eventually supporting both.  Having said that, I don't have any systems that can run DTrace.

I've chosen to build the upcoming Fedora 13's Python 2 and Python 3 rpms with this support enabled (I'm a Python guy, not a SystemTap guy, FWIW, but we have a strong in-house SystemTap team).  We've been tracking this downstream for Fedora, our detailed discussions are within this bug in our downstream tracker:
https://bugzilla.redhat.com/show_bug.cgi?id=545179
along with status:
  https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python
and documentation:
  https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python_2


Details
-------
SystemTap supplies a "dtrace" executable implemented in Python, which mimics the "dtrace" binary.
$ which dtrace
/usr/bin/dtrace
$ file $(which dtrace)
/usr/bin/dtrace: python script text executable
$ dtrace --help
Usage /usr/bin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]
Where -h builds a systemtap header file from the .d file
      -C when used with -h, also run cpp preprocessor
      -o specifies an explicit output file name,
         the default for -G is file.o and -h is file.h
      -I when running cpp pass through this -I include Path
      -s specifies the name of the .d input file
      -G builds a stub file.o from file.d,
         which is required by some packages that use dtrace.

I'm attaching a version of the patch against svn trunk.

This adds a "--with-dtrace" configuration option (I filed issue 7997 to better document how to do this).

Tested on a Fedora 12 box and works, with:
systemtap-1.1-2.fc12.i686
systemtap-sdt-devel-1.1-1.fc12.i686
systemtap-runtime-1.1-2.fc12.i686
kernel-PAE-2.6.31.12-174.2.3.fc12.i686
kernel-PAE-devel-2.6.31.12-174.2.3.fc12.i686
kernel-PAE-debuginfo-2.6.31.12-174.2.3.fc12.i686
kernel-headers-2.6.31.12-174.2.3.fc12.i686
kernel-debuginfo-common-i686-2.6.31.12-174.2.3.fc12.i686
kernel-firmware-2.6.31.12-174.2.3.fc12.noarch

Upon building --with-dtrace (but actually with systemtap), the linked binary containing ceval.o should contain a "probes" section (the following is a statically-linked build, hence the probes are in "./python"):

$ eu-readelf -S ./python | grep probes
[25] .probes              PROGBITS     081f9d28 1b0d28 000054  0 WA     0   0  8

$ eu-readelf -x .probes python

Hex dump of section [25] '.probes', 84 bytes at offset 0x1b0d28:
  0x00000000 66756e63 74696f6e 5f5f656e 74727900 function__entry.
  0x00000010 50524231 00000000 289d1f08 00000000 PRB1....(.......
  0x00000020 a50e0f08 00000000 66756e63 74696f6e ........function
  0x00000030 5f5f7265 7475726e 00000000 50524231 __return....PRB1
  0x00000040 509d1f08 00000000 1a0f0f08 00000000 P...............
  0x00000050 00000000                            ....

The above contains locations of "nop" instructions within the binary.  Systemtap generates kernel modules, and it uses this information to generate modules that monkeypatch the "nop" instruction at runtime to jump to code that gathers the pertinent information (filename, lineno etc) and returns that information to the kernel, and thus back to the user-space probe process.

Example of use
--------------
Here's an example of running regrtest whilst probing python to see function call and return during a run of the regression test suite (all in one shell command; obviously it would be saner to split the systemtap probe into a separate function):

[david@brick trunk-stap]$ stap -v -e'probe python.function.entry = process("./python").mark("function__entry")
{
    filename = user_string($arg1);
    funcname = user_string($arg2);
    lineno = $arg3;
}
probe python.function.return = process("./python").mark("function__return")
{
    filename = user_string($arg1);
    funcname = user_string($arg2);
    lineno = $arg3;
}
probe python.function.entry { printf("%s => %s in %s:%d\n", thread_indent(1), funcname, filename, lineno); }

probe python.function.return {printf("%s <= %s in %s:%d\n", thread_indent(-1), funcname, filename, lineno);}' \
-c "./python Lib/test/regrtest.py"

Output is as follow (note that I used -v to get verbose info on systemtap's passes):
Pass 1: parsed user script and 65 library script(s) using 20132virt/12268res/2052shr kb, in 220usr/10sys/242real ms.
Pass 2: analyzed script: 2 probe(s), 12 function(s), 2 embed(s), 2 global(s) using 24868virt/14788res/3392shr kb, in 20usr/10sys/37real ms.
Pass 3: using cached /home/david/.systemtap/cache/26/stap_2689ea1aadf52992794511e0d6996aca_5352.c
Pass 4: using cached /home/david/.systemtap/cache/26/stap_2689ea1aadf52992794511e0d6996aca_5352.ko
Pass 5: starting run.
     0 python(6769): => <module> in /home/david/coding/python-svn/trunk-stap/Lib/site.py:59
  1063 python(6769):  => <module> in /home/david/coding/python-svn/trunk-stap/Lib/os.py:22
  2510 python(6769):   => <module> in /home/david/coding/python-svn/trunk-stap/Lib/posixpath.py:11
  2756 python(6769):    => <module> in /home/david/coding/python-svn/trunk-stap/Lib/stat.py:4
  2792 python(6769):    <= <module> in /home/david/coding/python-svn/trunk-stap/Lib/stat.py:94
  3015 python(6769):    => <module> in /home/david/coding/python-svn/trunk-stap/Lib/genericpath.py:5
  3051 python(6769):    <= <module> in /home/david/coding/python-svn/trunk-stap/Lib/genericpath.py:85
  3713 python(6769):    => <module> in /home/david/coding/python-svn/trunk-stap/Lib/warnings.py:1
  3948 python(6769):     => <module> in /home/david/coding/python-svn/trunk-stap/Lib/linecache.py:6
  3984 python(6769):     <= <module> in /home/david/coding/python-svn/trunk-stap/Lib/linecache.py:68
(etc)

Performance
-----------
There are four cases here from a performance perspective:
(a) the source code without the patch
(b) the source code with the patch, configured --without-dtrace
(c) the source code with the patch, configured --with-dtrace, but without any "live" probes using the static marker
(d) the source code with the patch, configured --with-dtrace, with one or more "live" probes using the static marker

The performance goals I've been working towards are:
  - that (a) and (b) should have identical behavior, and
  - that (b) and (c) should have no significant performance differences i.e. having the markers compiled in shouldn't cost anything (as a packager of Python I want to be able to supply a common build, and the vast majority of my users won't be using the markers)
  - that it's acceptable for (d) to have a moderate performance cost compared to (c)

I used the unladen-swallow benchmark suite to examine performance.  

Our initial version of the patch (based on http://bugs.python.org/file12861/dtrace.diff ) showed a significant performance penalty for case (c) relative to case (b): 5-14% additional time being taken on most benchmarks, in rough (but, I believe, credible) tests.  Some notes on this can be seen at https://bugzilla.redhat.com/show_bug.cgi?id=545179#c12

I believe this regression is due to the generation and storing of f_calllineno at every frame entry.

We removed this and further benchmarking showed an insignificant performance difference between (b) and (c), though I wasn't as rigorous as I could have been with these benchmarks.  Some notes on this can be seen at https://bugzilla.redhat.com/show_bug.cgi?id=545179#c14

Case (d) probably has a significant penalty relative to (c), but I don't regard that as a blocker.  I haven't yet attempted to measure the performance impact of live probes.

Other issues
------------
If I'm reading things correctly, the marker is actually instrumenting frame entry and exit, not function entry and exit, so it probably needs renamimg.

There are many other possible static markers that could be implemented, but it's probably easiest to do this one first, then add more later (We brainstormed some ideas here: https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python )

In my Fedora rpm packages I also supply a "tap set" (a kind of SystemTap module, for code reuse), that modularizes parts of the tracing above to minimize boilerplate when writing a .stp script that uses the markers.
This can be seen here: http://cvs.fedoraproject.org/viewvc/rpms/python/devel/libpython.stp?view=markup

I also include an example script, which can be seen here:
http://cvs.fedoraproject.org/viewvc/rpms/python/devel/systemtap-example.stp?view=markup


Hope this is helpful; sorry if this is muddying the waters further :(
Dave
msg100174 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-02-26 20:51
Attached is the patch I'm currently applying in Fedora 13 to Python 3.

The patch is actually against the 3.1.1 tarball, rather than SVN; sorry (swamped with post-pycon tasks here), as that's what I've been testing this work against.

The patch is somewhat different to the Python 2 case, in that the filename and module name are stored internally by Python as unicode.  The markers convert these to UTF8 for the case when the markers are in use by an active probe.  Unfortunately it's now possible for this to fail (e.g. low memory).

An example of using this for a non-Latin 1 script can be seen towards the end of:
https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python_2
msg106208 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-05-21 00:01
Ping...

We already missed the 2.7 boat. Any hope for 3.2?
msg110693 - (view) Author: Mark Lawrence (BreamoreBoy) Date: 2010-07-18 21:50
Comments suggest this is wanted, can our build experts get this into 3.2?
msg110699 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-07-18 23:41
+1!
msg110700 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-19 00:07
If any RedHat/Fedora people are tuned in, can you give us an update on Systemtap/DTrace support in RedHat/Fedora python?  I understand that one of the main problems with bringing this upstream was the lack of uniformity between Apple and Sun approaches.  These days Apple and Sun(Oracle) are increasingly unfriendly to open source, so with a third solution available from RedHat, the choice seems obvious.
msg110718 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-07-19 07:45
On 19 Jul, 2010, at 1:07, Alexander Belopolsky wrote:

> 
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
> 
> If any RedHat/Fedora people are tuned in, can you give us an update on Systemtap/DTrace support in RedHat/Fedora python?  I understand that one of the main problems with bringing this upstream was the lack of uniformity between Apple and Sun approaches.  These days Apple and Sun(Oracle) are increasingly unfriendly to open source, so with a third solution available from RedHat, the choice seems obvious.

What do you mean by increasing unfriendly in Apple's case? I haven't noted any change from them in the last few years. Not that their (perceived) unfriendlyness is relevant.

I'm +1 on merging this feature having dtrace support is pretty convenient.

Ronald
msg117490 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-09-27 23:50
Updated py3k version of patch.

Changes:
  - renamed the probepoints:
      "function__entry" -> "frame__entry"
      "function__return" -> "frame__exit"
    as I believe this better describes what these do    
  - added a test suite: Lib/test/test_systemtap.py
  - reformatted to use 4-space indentation
  - slightly reworked the configure.in code

Some error-handling issue remain (see the FIXMEs in the code)

Tested and working on Fedora 13 x86_64, using:
  systemtap-1.3-1.fc13.x86_64
  kernel-2.6.33.3-84.fc13.x86_64
  kernel-devel-2.6.33.3-84.fc13.x86_64
  kernel-headers-2.6.33.3-84.fc13.x86_64
  kernel-debuginfo-2.6.33.3-84.fc13.x86_64
  kernel-debuginfo-common-x86_64-2.6.33.3-84.fc13.x86_64
msg117510 - (view) Author: Mark Wielaard (mjw) Date: 2010-09-28 10:36
>  - renamed the probepoints:
>      "function__entry" -> "frame__entry"
>      "function__return" -> "frame__exit"
>    as I believe this better describes what these do    

Are you sure you want to do this? You are right that it better describes the action that these probe points signify. But since previous versions of this patch have been used and applied people might be depending on the old names. It is not a very big issue, we can always write a tapset to translate the old to new names. But it is an interface break for anybody who might already have scripts around.
msg117514 - (view) Author: anatoly techtonik (techtonik) Date: 2010-09-28 12:33
I don't understand the last request. Is there already a practice to depend on patches that were applied by people, but were not released, reviewed or documented in some PEP?
msg117515 - (view) Author: Mark Wielaard (mjw) Date: 2010-09-28 13:14
The original patch was created to be as close as possible to the support that Sun and Apple added to their python implementation for Solaris and MacOS. Changing the probe point names would make the current patch slightly different for scripts written against those. That isn't a major problem, and can be worked around in user scripts or a tapset that translates the old names to new names. But it is something to consider before changing the probe names.
msg117519 - (view) Author: anatoly techtonik (techtonik) Date: 2010-09-28 13:57
So, it was released. But why they didn't send their patches here? We could review them, or we want Sun and Apple create Python specifications unilaterally? It would be best to contact them and notify about the change. Is it something a PSF should do?
msg117521 - (view) Author: John Levon (movement) Date: 2010-09-28 14:01
Anatoly, see above. We have done exactly that. (Honestly, when I first wrote the changes, I saw them as too hacky to ever be accepted upstream).

This name change seems frivolous and will indeed break existing scripts.
msg118574 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-10-13 20:08
Updated version of patch attached (against py3k; r85426)

I've changed the names back to "function__entry" and "function__return".
msg118575 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-10-13 20:11
What do we need to have this ready for Python 3.2?. The oportunity window is closing fast, and 3.3 is two years away...

So now or never...
msg119263 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-10-21 05:58
I've the same question as Jesús Cea Avión: what is needed to get this in 3.2?  This would IMHO be a useful feature.
msg119270 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-21 09:15
> I've the same question as Jesús Cea Avión: what is needed to get this
> in 3.2?

Same as usual: someone to review, apply, commit. Why do you ask?
msg119556 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-10-25 16:03
I am trying to review this for 3.2, but I am having some issues. For instance, "include/pydtrace.d" is not present in the last patch.

Please, post a patch with all the required changes in the same (patch) file. Hurry, we are still on track for 3.2. :-).

Another question: I am not able to decide between Sun/Apple style, or breaking dtrace scripts compatibility completely. Anybody has an opinion about this?. Is this actually important?. Are there so many legacy dtrace scripts out there?.
msg119557 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-25 16:15
2010/10/25 Jesús Cea Avión <report@bugs.python.org>:
..
> Another question: I am not able to decide between Sun/Apple style, or breaking dtrace scripts
> compatibility completely. Anybody has an opinion about this?. Is this actually important?. Are
> there so many legacy dtrace scripts out there?.
>

I would say compatibility with Sun/Apple probes should not stand in
the way of implementing this in cpython.  Of course familiarity to
existing users is a consideration, but in cases where self-consistency
can be improved, I don't think we should be overly concerned about
legacy scripts.  I would estimate that 90% of future users will never
have used either Sun or Apple probes, 9% will have used Apple and 1%
Sun. (This is completely unscientific guess, of course.)
msg119559 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-10-25 16:27
Updated version of the patch; this ought to contain Include/pydtrace.d:

$ diffstat /tmp/py3k-add-systemtap-2010-10-25.patch
 Include/pydtrace.d         |   10 
 Lib/test/test_systemtap.py |   89 ++++++
 Makefile.pre.in            |   10 
 Python/ceval.c             |   75 +++++
 configure                  |  576 +++++++++++++++++++++++----------------------
 configure.in               |   34 ++
 pyconfig.h.in              |    3 
 7 files changed, 522 insertions(+), 275 deletions(-)

Patch contains FIXMEs (sorry), which clearly need addressing.

As for the objectives, do folks here agree with the "Performance" notes in http://bugs.python.org/issue4111#msg100173 ?
msg119561 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-10-25 16:54
Malcolm, does your last patch address the performance issue?. Ideally, dtrace support should be compiled in by default, so performance issues are important.

Idealy, performance difference between compiling dtrace or not should be negligible. Until you actually use it, of course.
msg119562 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-10-25 17:00
We need some documentation, too.

Maybe a new chapter, or a new section in the debug chapter. Better the first, since this is not only for debugging, but for performance study too.
msg119563 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-10-25 17:02
Compiling the code WITHOUT dtrace support gives an error:

"""
Undefined                       first referenced
 symbol                             in file
__dtrace_python___function__entry   ./libpython3.2m.so
__dtraceenabled_python___function__return ./libpython3.2m.so
__dtrace_python___function__return  ./libpython3.2m.so
__dtraceenabled_python___function__entry ./libpython3.2m.so
ld: fatal: Symbol referencing errors. No output written to python
collect2: ld returned 1 exit status
"""
msg119564 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-10-25 17:12
Compiling WITH dtrace... shows the same error :-????
msg119566 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-10-25 17:26
I am using Solaris 10, but the configure script detects "apple" implementation.
msg119578 - (view) Author: Frank Ch. Eigler (fche) Date: 2010-10-25 20:39
I believe the problem jcea is experiencing is with the
solaris (/linux?) branch of the configure.in:

    if dtrace -G -o /dev/null -s $srcdir/Include/pydtrace.d 2>/dev/null

It seems solaris doesn't like the -o /dev/null part.  Try
specifying some real temporary file name instead.
msg119579 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-10-25 21:20
Updated patch, removing the FIXMEs, and slightly reworking the test code.

I've wrapped the whole of get_frame_marker_info with a PyErr_Fetch/PyErr_Restore pair: the PyUnicode_AsUTF8String calls could fail with a MemoryError, and we don't want to confuse the regular exception handling within ceval.

I'm not sure how to write a unit test to test for this: perhaps we could corrupt the __code__ instance so that co_filename is not a PyUnicodeObject, leading to a TypeError within the function, but that's a readonly attribute.  Any ideas?

I've also added a unit test for a non-ASCII script (Lib/test/systemtap_sample_☠.py), containing a non-ASCII identifier (文字化け).

The non-ASCII script name (Lib/test/systemtap_sample_☠.py) may be controversial: do we have anything like that in the source tree yet?  Is there any risk of messing up the build across platforms, or of impacting the Hg migration?

Still to-do:
  - address the "Undefined symbol" issues seen by jcea.
  - documentation
  - doublecheck performance
msg119580 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-25 21:42
> I've wrapped the whole of get_frame_marker_info with a
> PyErr_Fetch/PyErr_Restore pair: the PyUnicode_AsUTF8String calls could
> fail with a MemoryError, and we don't want to confuse the regular
> exception handling within ceval.

If PyUnicode_AsUTF8String() is meant to encode a filename, you should
use PyUnicode_EncodeFSDefault() instead.

> The non-ASCII script name (Lib/test/systemtap_sample_☠.py) may be
> controversial: do we have anything like that in the source tree yet?
> Is there any risk of messing up the build across platforms, or of
> impacting the Hg migration?

It would be better to generate the sample dynamically, so that users
with an incompatible locale or filesystem aren't prevented from checking
out the source.
msg119583 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-10-25 22:52
> It would be better to generate the sample dynamically, so that users
> with an incompatible locale or filesystem aren't prevented from checking
> out the source.
Thanks: am attaching updated patch: I've removed Lib/test/systemtap_sample_☠.py, and now generate a similarly-named file during the test, using test.support.TEST_FN and unlink

Still TODO:
  - address pitrou's concerns about PyUnicode_AsUTF8String from msg119580
  - address the "Undefined symbol" issues seen by jcea (msg119563 onwards)
  - documentation
  - doublecheck performance
  - perhaps add a systemtap "tapset", and demo code using it (like I did in Fedora's python3 RPMs)
  - anything else I've missed :)
msg119584 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-10-25 23:05
I should note that I've only ever been testing this with SystemTap, on Linux.  I don't have a box with DTrace, and I've never directly used it.  I wouldn't today be able to diagnose a buildbot failure related to DTrace (I believe I would with systemtap, fwiw).

Are there any DTrace experts around who would be willing to handle the DTrace side of this?  If not, would it be reasonable to make this issue be only explicitly about "systemtap"?  (e.g. perhaps change the "configure" argument accordingly?)  Alternatively, given that this bug originally started as an RFE about DTrace, should we split out systemtap as a separate RFE?

I'm sorry if I've "muddied the waters" by doing this.

For example, the only test coverage I've written (test_systemtap.py) checks for the presence of a "stap" executable, and skips the tests if it's not found.  I'm not familiar enough with DTrace to create the same for DTrace.

FWIW (in response to IRC question): "thread_indent" is documented here:
http://sourceware.org/systemtap/SystemTap_Beginners_Guide/systemtapscript-handler.html#thread_indent

It looks like it may be systemtap-specific; however the only usage is within test_systemtap.py, guarded by the presence of a "stap" binary, skipping the tests if it is not found.
msg119621 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-26 17:29
For the record, replacing /dev/null with conftest.out in the configure test solves the detection problem (and allows Python to build cleanly). However, there is then a problem in test_systemtap (even when replacing "stap" with "dtrace") since the syntax for scripts doesn't seem compatible.

In any case, the trivial fix for configure:

diff -r 777b171a63ae -r 1784ac25b52e configure
--- a/configure Tue Oct 26 18:50:46 2010 +0200
+++ b/configure Tue Oct 26 18:54:09 2010 +0200
@@ -9178,7 +9178,7 @@
 $as_echo "$with_dtrace" >&6; }
 if test ! -z "$with_dtrace"
 then
-    if dtrace -G -o /dev/null -s $srcdir/Include/pydtrace.d 2>/dev/null
+    if dtrace -G -o conftest.out -s $srcdir/Include/pydtrace.d 2>/dev/null
     then
 
 $as_echo "#define WITH_DTRACE 1" >>confdefs.h
diff -r 777b171a63ae -r 1784ac25b52e configure.in
--- a/configure.in      Tue Oct 26 18:50:46 2010 +0200
+++ b/configure.in      Tue Oct 26 18:54:09 2010 +0200
@@ -2466,7 +2466,7 @@
 AC_MSG_RESULT([$with_dtrace])
 if test ! -z "$with_dtrace"
 then
-    if dtrace -G -o /dev/null -s $srcdir/Include/pydtrace.d 2>/dev/null
+    if dtrace -G -o conftest.out -s $srcdir/Include/pydtrace.d 2>/dev/null
     then
        AC_DEFINE(WITH_DTRACE, 1, 
         [Define if you want to compile in Dtrace support])
msg119622 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-26 17:29
(my last message was about building on OpenSolaris)
msg121374 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-11-17 18:37
Dave, we need some kind of documentation, if we expect to ship this in Python 3.2. The deadline is only 10-15 days away.

Could you write something able to be in the standard documentation?.
msg122392 - (view) Author: John Levon (movement) Date: 2010-11-25 17:55
configure.in has:

AC_MSG_RESULT([$with_dtrace])
...
AC_MSG_RESULT($with_dtrace)

Why twice? It looks confusing.
msg122534 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2010-11-27 16:19
Some references to keep around:

http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/
http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/Python26-07-dtrace.diff
http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/Python-07-dtrace.diff
msg133102 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2011-04-06 00:42
Some more references:

Read the notes under the slides:
https://dgl.cx/2011/01/dtrace-and-perl

https://dgl.cx/dtrace

http://dtrace.org/blogs/

What do we need to unblock this?
msg133114 - (view) Author: anatoly techtonik (techtonik) Date: 2011-04-06 07:07
2011/4/6 Jesús Cea Avión <report@bugs.python.org>:
>
> Jesús Cea Avión <jcea@jcea.es> added the comment:
>
> Some more references:
>
> Read the notes under the slides:
> https://dgl.cx/2011/01/dtrace-and-perl
>
> https://dgl.cx/dtrace
>
> http://dtrace.org/blogs/
>
> What do we need to unblock this?

Summarize 30+ page discussion in a new issue.
Blog about it on http://blog.python.org/
--
anatoly t.
msg133147 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2011-04-06 17:02
jcea: I notice that on 2011-02-22 you made these changes:
  assignee: dmalcolm -> dino.viehland
  nosy: +dino.viehland
  versions: +Python 3.3 -Python 3.2

Did you mean to change the assignee, or was this an accident?
msg133151 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2011-04-06 18:04
Malcolm, it was a mistake. I only wanted to change the TARGET.

I assign the issue to you again.

Dino, I delete you from the nosy list now. Sorry for the inconvenience.

My excuses to both.
msg147388 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2011-11-10 01:01
Anybody still working on this?.

We missed the 2.7 boat. DO NOT MISS THE 3.3 ONE!!! :-)
msg148209 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2011-11-23 22:16
This project continues in issue #13405.
msg160449 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2012-05-11 20:59
Issue #13405 covers DTrace; I've taken the liberty of also opening issue #14776 to cover SystemTap.  I hope that once one of these is in the tree it will be easier to get the other one in.
History
Date User Action Args
2012-05-11 20:59:48dmalcolmsetmessages: + msg160449
2011-11-23 22:16:56jceasetmessages: + msg148209
2011-11-14 19:56:55jceasetstatus: open -> closed
resolution: out of date
superseder: Add DTrace probes
2011-11-14 19:43:12jcealinkissue13405 dependencies
2011-11-10 01:03:38bretthoernersetnosy: - bretthoerner
2011-11-10 01:01:50jceasetmessages: + msg147388
2011-06-19 01:22:24serverhorrorsetnosy: + serverhorror
2011-04-06 18:04:08jceasetassignee: dino.viehland -> dmalcolm

messages: + msg133151
nosy: - dino.viehland
2011-04-06 17:02:47dmalcolmsetmessages: + msg133147
2011-04-06 07:07:11techtoniksetmessages: + msg133114
2011-04-06 00:42:17jceasetmessages: + msg133102
2011-02-23 01:19:30jceasetassignee: dmalcolm -> dino.viehland

nosy: + dino.viehland
versions: + Python 3.3, - Python 3.2
2010-11-27 16:19:16jceasetmessages: + msg122534
2010-11-25 17:55:18movementsetmessages: + msg122392
2010-11-17 18:37:10jceasetmessages: + msg121374
2010-11-17 18:20:17hazmatsetnosy: + hazmat
2010-10-26 17:29:54pitrousetmessages: + msg119622
2010-10-26 17:29:11pitrousetmessages: + msg119621
2010-10-25 23:05:45dmalcolmsetmessages: + msg119584
2010-10-25 22:52:35dmalcolmsetfiles: + py3k-add-systemtap-2010-10-25-003.patch

messages: + msg119583
2010-10-25 21:42:33pitrousetmessages: + msg119580
2010-10-25 21:20:51dmalcolmsetfiles: + py3k-add-systemtap-2010-10-25-002.patch

messages: + msg119579
2010-10-25 20:39:39fchesetnosy: + fche
messages: + msg119578
2010-10-25 20:23:39dhduvallsetnosy: + dhduvall
2010-10-25 17:26:49jceasetmessages: + msg119566
2010-10-25 17:12:04jceasetmessages: + msg119564
2010-10-25 17:02:01jceasetmessages: + msg119563
2010-10-25 17:00:02jceasetmessages: + msg119562
2010-10-25 16:54:57giampaolo.rodolasetnosy: - giampaolo.rodola
2010-10-25 16:54:10jceasetmessages: + msg119561
2010-10-25 16:27:56dmalcolmsetfiles: + py3k-add-systemtap-2010-10-25.patch

messages: + msg119559
2010-10-25 16:15:50belopolskysetmessages: + msg119557
2010-10-25 16:03:03jceasetassignee: dmalcolm
messages: + msg119556
2010-10-21 09:15:30pitrousetnosy: + pitrou
messages: + msg119270
2010-10-21 05:58:37ronaldoussorensetmessages: + msg119263
2010-10-13 20:11:23jceasetmessages: + msg118575
2010-10-13 20:08:47dmalcolmsetmessages: + msg118574
2010-09-28 14:01:10movementsetmessages: + msg117521
2010-09-28 13:57:30techtoniksetmessages: + msg117519
2010-09-28 13:14:45mjwsetmessages: + msg117515
2010-09-28 12:33:14techtoniksetmessages: + msg117514
2010-09-28 10:36:15mjwsetmessages: + msg117510
2010-09-27 23:50:17dmalcolmsetfiles: + py3k-add-systemtap-2010-09-27.patch

messages: + msg117490
2010-09-03 08:17:42techtoniksetnosy: + techtonik
2010-09-01 19:59:59giampaolo.rodolasetnosy: + giampaolo.rodola
2010-07-19 07:45:03ronaldoussorensetfiles: + smime.p7s

messages: + msg110718
2010-07-19 00:07:31belopolskysetmessages: + msg110700
2010-07-18 23:41:18jceasetmessages: + msg110699
2010-07-18 21:50:39BreamoreBoysetnosy: + BreamoreBoy
messages: + msg110693
2010-05-21 02:21:50r.david.murraysetnosy: + Garen, - skip.montanaro, prescod
priority: normal
assignee: prescod -> (no value)
keywords: + patch
stage: patch review
2010-05-21 00:01:21jceasetpriority: normal -> (no value)

assignee: prescod
versions: + Python 3.2, - Python 3.1, Python 2.7
keywords: + needs review, - patch
nosy: + prescod, skip.montanaro, - Garen

messages: + msg106208
2010-05-20 20:37:46skip.montanarosetassignee: skip.montanaro -> (no value)

nosy: - skip.montanaro
2010-04-15 21:54:28Garensetnosy: + Garen
2010-02-26 20:51:31dmalcolmsetfiles: + python-3.1.1-systemtap.patch

messages: + msg100174
2010-02-26 20:45:14dmalcolmsetfiles: + add-dtrace-or-systemtap-markers-to-trunk.patch
2010-02-26 20:41:47dmalcolmsetmessages: + msg100173
2010-02-26 19:00:26jceasetmessages: + msg100169
2010-02-24 15:56:00skip.montanarosetmessages: + msg100041
2010-02-24 01:23:25jceasetmessages: + msg99986
2010-01-15 18:18:33glyphsetnosy: + glyph
2010-01-15 17:45:48wsanchezsetnosy: + wsanchez
2009-12-08 21:27:26mjwsetfiles: + stap-python.patch
title: Add DTrace probes -> Add Systemtap/DTrace probes
nosy: skip.montanaro, rhettinger, jcea, ronaldoussoren, belopolsky, movement, bretthoerner, laca, twleung, jbaker, robert.kern, sirg3, chrismiles, danchr, dmalcolm, mjw
messages: + msg96154

components: - Installation
2009-12-07 05:35:12mjwsetnosy: + mjw
2009-09-18 13:41:44dmalcolmsetnosy: + dmalcolm
2009-09-18 00:24:35jbakersetnosy: + jbaker
2009-04-24 06:45:16robert.kernsetmessages: + msg86399
2009-04-24 05:03:00robert.kernsetmessages: + msg86396
2009-04-24 03:24:08robert.kernsetmessages: + msg86392
2009-04-24 02:03:18skip.montanarosetmessages: + msg86389
2009-04-23 23:46:10jceasetmessages: + msg86385
2009-04-23 23:43:33robert.kernsetmessages: + msg86384
2009-04-22 22:12:12robert.kernsetmessages: + msg86343
2009-04-22 21:26:48robert.kernsetmessages: + msg86338
2009-04-22 21:21:49twleungsetfiles: + unnamed

messages: + msg86337
2009-04-22 21:17:35movementsetmessages: + msg86335
2009-04-22 20:42:07robert.kernsetmessages: + msg86330
2009-04-22 20:24:06skip.montanarosetmessages: + msg86326
2009-04-22 20:00:01robert.kernsetmessages: + msg86322
2009-04-22 12:10:41movementsetmessages: + msg86290
2009-04-22 07:45:21robert.kernsetmessages: + msg86279
2009-04-22 07:34:47lacasetmessages: + msg86278
2009-04-22 06:41:45robert.kernsetmessages: + msg86277
2009-04-22 06:12:58robert.kernsetmessages: + msg86273
2009-03-09 11:38:20danchrsetnosy: + danchr
2009-02-23 06:02:59chrismilessetnosy: + chrismiles
2009-02-19 15:38:46sirg3setnosy: + sirg3
2009-02-05 21:12:21movementsetmessages: + msg81229
2009-02-02 13:11:09jceasetnosy: + jcea
2009-01-28 03:06:38twleungsetmessages: + msg80691
2009-01-27 23:39:51skip.montanarosetmessages: + msg80679
2009-01-27 22:05:26twleungsetmessages: + msg80671
2009-01-27 16:50:41movementsetnosy: + movement
2009-01-25 22:00:13skip.montanarosetfiles: + dtrace.diff
messages: + msg80542
2009-01-24 21:59:06brett.cannonsetnosy: - brett.cannon
2009-01-24 13:46:43skip.montanarosetmessages: + msg80447
2009-01-24 05:17:54lacasetmessages: + msg80436
2009-01-23 02:00:46skip.montanarosetmessages: + msg80395
2009-01-22 22:53:58skip.montanarosetmessages: + msg80388
2009-01-22 21:41:29twleungsetfiles: + unnamed
messages: + msg80382
2009-01-22 21:35:05skip.montanarosetassignee: skip.montanaro
versions: + Python 3.1, Python 2.7, - Python 2.6
messages: + msg80381
nosy: + ronaldoussoren
2009-01-22 21:15:26rhettingersetnosy: + rhettinger
2009-01-22 20:56:07robert.kernsetnosy: + robert.kern
2008-11-14 06:03:00lacasetnosy: + laca
messages: + msg75849
2008-11-13 19:59:01twleungsetnosy: + twleung
messages: + msg75834
2008-11-13 18:55:33brett.cannonsetmessages: + msg75832
2008-11-13 16:05:23skip.montanarosetmessages: + msg75828
2008-11-13 15:09:43bretthoernersetmessages: + msg75825
2008-11-13 03:31:44skip.montanarosetmessages: + msg75811
2008-11-13 02:57:59bretthoernersetmessages: + msg75810
2008-11-13 02:45:05skip.montanarosetmessages: + msg75809
2008-11-13 02:26:51belopolskysetnosy: + belopolsky
2008-11-12 18:36:15skip.montanarosetnosy: + skip.montanaro
2008-10-12 18:48:20brett.cannonsetnosy: + brett.cannon
messages: + msg74676
2008-10-12 16:08:14bretthoernercreate