Issue4111
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2008-10-12 16:08 by bretthoerner, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
python-2.6-dtrace.diff | bretthoerner, 2008-10-12 16:08 | dtrace patch against python 2.6 (with rude/broken Makefile) | ||
unnamed | twleung, 2009-01-22 21:41 | |||
dtrace.diff | skip.montanaro, 2009-01-25 22:00 | |||
unnamed | twleung, 2009-04-22 21:21 | |||
stap-python.patch | mjw, 2009-12-08 21:27 | Cleaned up systemtap/dtrace patch | ||
add-dtrace-or-systemtap-markers-to-trunk.patch | dmalcolm, 2010-02-26 20:45 | The patch I'm applying in Fedora 13, regenerated against SVN trunk | ||
python-3.1.1-systemtap.patch | dmalcolm, 2010-02-26 20:51 | The patch against 3.1.1 I'm applying in Fedora 13 | ||
smime.p7s | ronaldoussoren, 2010-07-19 07:45 | |||
py3k-add-systemtap-2010-09-27.patch | dmalcolm, 2010-09-27 23:50 | Updated version of patch, against py3k (r85046) | ||
py3k-add-systemtap-2010-10-25.patch | dmalcolm, 2010-10-25 16:27 | |||
py3k-add-systemtap-2010-10-25-002.patch | dmalcolm, 2010-10-25 21:20 | |||
py3k-add-systemtap-2010-10-25-003.patch | dmalcolm, 2010-10-25 22:52 |
Pull Requests | |||
---|---|---|---|
URL | Status | Linked | Edit |
PR 25115 | open | shihai1991, 2021-03-31 17:30 |
Messages (85) | |||
---|---|---|---|
msg74670 - (view) | Author: Brett Hoerner (bretthoerner) | Date: 2008-10-12 16:08 | |
It would be great if the main Python distribution supported DTrace on (Open)Solaris / FreeBSD / OS X. I've attached a patch against 2.6 that instruments function calls. It's from the ed scripts included in Apple's Python distribution: http://developer.apple.com/opensource/index.html (PSF License) It is imperfect because I wasn't sure how to do the equivalent of IFDEF in the Makefile.pre.in, as you can see this patch will make ceval.o depend on pydtrace.h, which depends on having a local `dtrace` command. The rest of the patch properly uses IFDEFs. |
|||
msg74676 - (view) | Author: Brett Cannon (brett.cannon) * | Date: 2008-10-12 18:48 | |
At one point we were told Apple would try to backport their dtrace instrumentation. I don't know what the status of that is, but it obviously has not happened. |
|||
msg75809 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2008-11-13 02:45 | |
It appears that Apple has dtracified their Python exeutable in Leopard. Any chance that they can be persuaded to release a patch? I'm working on a patch (based on some work from a friend at work), but it seems like Apple already has a horse in the race. It would be nice if we could see what they have done. |
|||
msg75810 - (view) | Author: Brett Hoerner (bretthoerner) | Date: 2008-11-13 02:57 | |
They have released the changes, that's what my patch (attached to the issue) is based on. It's working, it just needs to be cleaned up (it will fail, I believe, for people on systems without DTrace - as I said I'm not very familiar with Makefiles). You can use it now against 2.6 and probably with very few changes against 2.5. |
|||
msg75811 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2008-11-13 03:31 | |
Brett> They have released the changes, that's what my patch (attached to Brett> the issue) is based on. I see the reference to Apple in your original post, but can't find anything related to dtrace & python starting from the URL you gave. Do you have something more specific? At this point Jeff's code does a fair bit more than simply tracing the CALL_FUNCTION opcode but needs some work with the other CALL_FUNCTION_* opodes. It also has some obmalloc tracing which I've not yet tested. I would have thought Apple would more heavily instrument the interpreter than it appears they have. Brett> It's working, it just needs to be cleaned up (it will fail, I Brett> believe, for people on systems without DTrace - as I said I'm not Brett> very familiar with Makefiles). I took care of the configure.in/Makefile.pre.in stuff today. The intent is that you would configure using --enable-dtrace then not have to do anything else to build a dtrace-aware interpreter. Brett> You can use it now against 2.6 and probably with very few changes Brett> against 2.5. We are using 2.4 at work and it works there. I'm fairly certain we should be able to get it working for the entire 2.x and 3.x series with only a little effort. (I still need to get approval to release it, but I don't think that will be a big deal. I've already alerted my boss and he's generally receptive to such things.) Skip |
|||
msg75825 - (view) | Author: Brett Hoerner (bretthoerner) | Date: 2008-11-13 15:09 | |
On Wed, Nov 12, 2008 at 9:31 PM, Skip Montanaro <report@bugs.python.org> wrote: > I see the reference to Apple in your original post, but can't find anything > related to dtrace & python starting from the URL you gave. Do you have > something more specific? http://www.opensource.apple.com/darwinsource/10.5.5/python-30.1.2/ That isn't Python 3.0, that's what I guess they call Python "Apple Version 30"? Anyways, it's their Python 2.5.1 and their "patches" against it are actually ed scripts, contained in fix/ That's where my patch comes from, it's just changing the .ed scripts to a patch and applying it against 2.6 > At this point Jeff's code does a fair bit more than simply tracing the > CALL_FUNCTION opcode but needs some work with the other CALL_FUNCTION_* > opodes. It also has some obmalloc tracing which I've not yet tested. I > would have thought Apple would more heavily instrument the interpreter than > it appears they have. Sun has released a patch against DTrace that probes more than just function calls also. At least, it provides line number and some other information, http://cvs.opensolaris.org/source/xref//jds/spec-files/trunk/patches/Python-07-dtrace.diff I couldn't figure out why it broke the compile on OS X though, and could only get it working on Solaris. There's another problem (I think) with DTrace probes ... if the Apple guys release Apple-Probe-Python.d and the Sun guys release Sun-Probe-Python.d (just two scripts) and we setup our probes differently, one or both will fail (because one may expect a probe for foo while another expects a probe for bar). Kind of sucks, considering Sun was first and Apple chose to probe Python differently. Now we have to pick one or make a 3rd... I could be very mistaken here, though. |
|||
msg75828 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2008-11-13 16:05 | |
Brett> http://www.opensource.apple.com/darwinsource/10.5.5/python-30.1.2/ ... Brett> http://cvs.opensolaris.org/source/xref//jds/spec-files/trunk/patches/Python-07-dtrace.diff Thanks for the pointers. I'll work on getting a uniform patch which incorporates both. (Though some of the comments in the OpenSolaris patch are a bit scary.) Brett> There's another problem (I think) with DTrace probes ... if the Brett> Apple guys release Apple-Probe-Python.d and the Sun guys release Brett> Sun-Probe-Python.d (just two scripts) and we setup our probes Brett> differently, one or both will fail (because one may expect a Brett> probe for foo while another expects a probe for bar). Kind of Brett> sucks, considering Sun was first and Apple chose to probe Python Brett> differently. Yeah, that would suck. It would be nice if they could agree on a common set of probes. I don't know that we have any contacts within the two development communities but if we can scare some up maybe we can get them to talk to each other. :-/ Skip |
|||
msg75832 - (view) | Author: Brett Cannon (brett.cannon) * | Date: 2008-11-13 18:55 | |
On Thu, Nov 13, 2008 at 08:05, Skip Montanaro <report@bugs.python.org> wrote: > > Skip Montanaro <skip@pobox.com> added the comment: > > Brett> http://www.opensource.apple.com/darwinsource/10.5.5/python-30.1.2/ > ... > Brett> http://cvs.opensolaris.org/source/xref//jds/spec-files/trunk/patches/Python-07-dtrace.diff > > Thanks for the pointers. I'll work on getting a uniform patch which > incorporates both. (Though some of the comments in the OpenSolaris patch > are a bit scary.) > > Brett> There's another problem (I think) with DTrace probes ... if the > Brett> Apple guys release Apple-Probe-Python.d and the Sun guys release > Brett> Sun-Probe-Python.d (just two scripts) and we setup our probes > Brett> differently, one or both will fail (because one may expect a > Brett> probe for foo while another expects a probe for bar). Kind of > Brett> sucks, considering Sun was first and Apple chose to probe Python > Brett> differently. > > Yeah, that would suck. It would be nice if they could agree on a common set > of probes. I don't know that we have any contacts within the two > development communities but if we can scare some up maybe we can get them to > talk to each other. :-/ > Obviously Ronald is the Apple insider to talk to. If you want Sun, you might want to talk to Ted Leung and see who he can put you in touch with. |
|||
msg75834 - (view) | Author: Ted Leung (twleung) | Date: 2008-11-13 19:59 | |
And courtesy of Philip Jenvey, here I am. I would *really* like to work to help make this happen. Laszlo Peter at Sun has been doing the ports of Python on Solaris, but we are not up to 2.6 just yet. I'm attaching a pointer to his patches against 2.5: http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/Python25-07-dtrace.diff These patches include John Levon's ustack provider, which if you care about DTrace and Python, you want to have. I've also had some conversations with people at Apple, and they have agreed that they will pull DTrace probes from python.org if they got in there. That would solve the diverging probe problem. I'd love to have a discussion about DTrace probe futures for CPython -- probably on python-dev. |
|||
msg75849 - (view) | Author: Laszlo (Laca) Peter (laca) | Date: 2008-11-14 06:02 | |
I'm the python package maintainer at Sun. We would really like to get the dtrace probes upstream, let me know how I can help. |
|||
msg80381 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-01-22 21:35 | |
So I completely dropped the ball on this. It appears we have some folks from Sun and Brett surmised that Ronald Oussoren would be the likely person to do the heavy lifting on the Apple side of things. Ronald, I've made you nosy. I will try to get the Solaris patch Ted referenced running against the trunk (2.7) and py3k (3.1) code bases, but it would be helpful if you could take a quick stab at it as well. |
|||
msg80382 - (view) | Author: Ted Leung (twleung) | Date: 2009-01-22 21:41 | |
On Jan 22, 2009, at 1:35 PM, Skip Montanaro wrote: > > Skip Montanaro <skip@pobox.com> added the comment: > > So I completely dropped the ball on this. It appears we have some > folks from Sun and Brett surmised that Ronald Oussoren would be the > likely person to do the heavy lifting on the Apple side of things. > Ronald, I've made you nosy. I will try to get the Solaris patch Ted > referenced running against the trunk (2.7) and py3k (3.1) code bases, > but it would be helpful if you could take a quick stab at it as well. Great. Actually the latest version of the patch is this one: <http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/Python26-07-dtrace.diff >. I tried to quickly compile it on MacOS 10.5, Python 2.6.1, but it didn't work right out of the box. Ted |
|||
msg80388 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-01-22 22:53 | |
After applying the patch and reconfiguring I get compilation errors in Python/ceval.c. I suspect it's because there is a new header file, Python/python.h. That's probably found by the #include directive in favor of Include/Python.h because of the Mac's case-insensitive file system. Changing the Makefile targets and actions so it's named python-dtrace.h seems to work a bit better. I get an error later running dtrace which I have yet to investigate. |
|||
msg80395 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-01-23 02:00 | |
me> I get an error later running dtrace which I have yet to investigate. Apple's dtrace program doesn't support the -G flag. When I remove it from Makefile.pre.in and rebuild I get an error about privileges: dtrace -o Python/dtrace.o -C -s ./Python/python.d Python/ceval.o dtrace: failed to initialize dtrace: DTrace requires additional privileges I tried adding "sudo" to the dtrace command. Then it prompts for my password and emits this error: sudo dtrace -o Python/dtrace.o -C -s ./Python/python.d Python/ceval.o Password: dtrace: failed to compile script ./Python/python.d: line 11: extraneous argument 'Python/ceval.o' ($1 is not referenced) I'm not sure what to do at this point. I'm not really a dtrace person. Perhaps the Apple and Sun dtrace experts can offer a way out of this little corner. Skip |
|||
msg80436 - (view) | Author: Laszlo (Laca) Peter (laca) | Date: 2009-01-24 05:17 | |
Please see here for discussion about the -G flag on OS X: http://markmail.org/message/4nheqnexjr2o6mcx If I read it correctly, on OS X, you will need to use -h instead of -G and it won't emit an object file (dtrace.o) so you will not need to link it. Unfortunately, this means that the makefile will have to be different on Solaris and OS X :( |
|||
msg80447 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-01-24 13:46 | |
Laca> Please see here for discussion about the -G flag on OS X: Laca> http://markmail.org/message/4nheqnexjr2o6mcx Laca> If I read it correctly, on OS X, you will need to use -h instead Laca> of -G and it won't emit an object file (dtrace.o) so you will not Laca> need to link it. Laca> Unfortunately, this means that the makefile will have to be Laca> different on Solaris and OS X :( We can worm around that using configure. Thanks for the reference. That will go a long way toward getting the Sun dtrace probes working on Mac OSX. For reference by others, here's Lee Packham's dtrace patch for PostgreSQL on Mac OSX: http://blog.leenux.org.uk/wp-content/uploads/2008/10/postgresql-825-fixosxdtracediff.gz |
|||
msg80542 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-01-25 22:00 | |
Here's a patch against the current trunk (2.7) which compiles on my Mac. It adds a --with-dtrace configure option. The code checks to see if the -G option is understood by the dtrace command. If so, dtrace support is added Sun-style. If not we do things Apple's way. The existing test cases pass on Apple except for one case in test_sys which tries to confirm the size of a frame object. I added an #ifdef WITH_DTRACE around the extra slot but there is no way of telling from Python code that this extra slot is there. That would have to somehow be exposed to the Python programmer so the test can be adjusted. All tests pass when --with-dtrace is omitted. I have not yet tried this on Solaris. I will try to get to it this week if someone doesn't beat me to it. There are as yet no new dtrace test cases so I can't confirm that the added dtrace support actually works. There are also no documentation updates. Unlike most compile options this adds a significant new feature to the runtime environment so some documentation changes are probably called for. |
|||
msg80671 - (view) | Author: Ted Leung (twleung) | Date: 2009-01-27 22:05 | |
I tried building this on my Mac and got this; ar cr libpython2.7.a Python/_warnings.o Python/Python-ast.o Python/ asdl.o Python/ast.o Python/bltinmodule.o Python/ceval.o Python/ compile.o Python/codecs.o Python/errors.o Python/frozen.o Python/ frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/ graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/ modsupport.o Python/mystrtoul.o Python/mysnprintf.o Python/peephole.o Python/pyarena.o Python/pyfpe.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/structmember.o Python/symtable.o Python/ sysmodule.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/formatter_unicode.o Python/formatter_string.o Python/dynload_shlib.o @DTRACEOBJS@ Python/mactoolboxglue.o Python/ thread.o ar: @DTRACEOBJS@: No such file or directory make: *** [libpython2.7.a] Error 1 my configure line was: ./configure --enable-framework --enable-toolbox-glue --with-threads -- enable-dtrace On Jan 25, 2009, at 2:00 PM, Skip Montanaro wrote: > > Skip Montanaro <skip@pobox.com> added the comment: > > Here's a patch against the current trunk (2.7) which compiles on my > Mac. It > adds a --with-dtrace configure option. The code checks to see if > the -G > option is understood by the dtrace command. If so, dtrace support > is added > Sun-style. If not we do things Apple's way. > > The existing test cases pass on Apple except for one case in > test_sys which > tries to confirm the size of a frame object. I added an #ifdef > WITH_DTRACE > around the extra slot but there is no way of telling from Python > code that > this extra slot is there. That would have to somehow be exposed to > the > Python programmer so the test can be adjusted. All tests pass when > --with-dtrace is omitted. I have not yet tried this on Solaris. I > will try > to get to it this week if someone doesn't beat me to it. > > There are as yet no new dtrace test cases so I can't confirm that > the added > dtrace support actually works. There are also no documentation > updates. > Unlike most compile options this adds a significant new feature to the > runtime environment so some documentation changes are probably > called for. > > Added file: http://bugs.python.org/file12861/dtrace.diff > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue4111> > _______________________________________ |
|||
msg80679 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-01-27 23:39 | |
Ted> I tried building this on my Mac and got this; Forgive me if I'm preaching to the choir here. Did you run autoconf or autoreconf after applying the patch? If not, @DTRACEOBJS@ would not be a substitutable string. It's fairly common (at least in the Python community) to omit modified configure scripts from these sorts of patches because the changes to generated configure scripts between different versions of autoconf are so massive that they dwarf the actual functional changes in the patch, often by a couple orders of magnitude. Skip |
|||
msg80691 - (view) | Author: Ted Leung (twleung) | Date: 2009-01-28 03:06 | |
I didn't run auto(re)conf. After I did that, all was well. However, the ustack provider doesn't appear to be working correctly. I tried running the py_profile.d from the DTrace toolkit, and it doesn't show any stack traces, and when the script starts up it says cc1: warning: /dev/fd/5 is shorter than expected The basic function entry/exit probes appear to be working. John +nosy'ed himself, so perhaps he'll have some insight? On Jan 27, 2009, at 3:39 PM, Skip Montanaro wrote: > > Skip Montanaro <skip@pobox.com> added the comment: > > Ted> I tried building this on my Mac and got this; > > Forgive me if I'm preaching to the choir here. > > Did you run autoconf or autoreconf after applying the patch? If not, > @DTRACEOBJS@ would not be a substitutable string. It's fairly > common (at > least in the Python community) to omit modified configure scripts > from these > sorts of patches because the changes to generated configure scripts > between > different versions of autoconf are so massive that they dwarf the > actual > functional changes in the patch, often by a couple orders of > magnitude. > > Skip > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue4111> > _______________________________________ |
|||
msg81229 - (view) | Author: John Levon (movement) | Date: 2009-02-05 21:12 | |
I haven't seen "shorter than expected" message before, sounds like a Mac OS X specific thing. As for never getting any probes out, this is where it gets fun. Debugging this is very tricky indeed: it involves you dropping into the kernel debugger and looking at the interpreter's state. Most likely, the compiler is doing something different with the relevant Python functions, causing the logic in the DTrace ustack probe to fail. I don't even know how the Apple guys debug this sort of thing, but they'd definitely need to be involved. |
|||
msg86273 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-22 06:12 | |
Skip, it doesn't appear that the ustack helper is getting incorporated into the OS X build anywhere. This rule is obviously wrong (compiling the wrong input file with the wrong flags): Include/phelper.h: $(srcdir)/Include/phelper.d dtrace -o $@ $(DFLAGS) -C -h -s $(srcdir)/Python/python.d I *think* it should be something along these lines: Include/phelper.h: $(srcdir)/Include/phelper.d dtrace -o $@ -DPHELPER $(DFLAGS) $(CPPFLAGS) -C -h -s $(srcdir)/Include/phelper.d There are some static functions in the standard system headers that get pulled in that need to be avoided. The #define _SYS_STAT_H up at the top avoids similar problems on Solaris. I'm working through the OS X headers now, but I'm stuck on trying to avoid sys/_structs.h. Unfortunately, it does not have the #ifndef _SYS__STRUCTS_H magic up at the top that would allow me to do so, nor any convenient #ifdefs around the offending declaration. <sigh> Even if that would work, phelper.h is not #included anywhere. I'm not really sure where it needs to go. On Solaris, the object file gets linked in, but on OS X, you don't generate an object file; you just #include the .h. Somewhere. I'm guessing where the pydtrace.h is currently, but I don't really know. John, do you have any insights? Also, s/DTRADEHDRS/DTRACEHDRS/ |
|||
msg86277 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-22 06:41 | |
Got a bit farther. Adding this stanza to the top of phelper.d gets past the issues in the headers: #ifdef __APPLE__ #define _SYS_TIME_H_ #define _SYS_SELECT_H_ #define __MATH_H__ #define _OS__OSBYTEORDER_H #define _FD_SET #define __GNUC_VA_LIST #endif /* __APPLE__ */ However, I now get a more legitimate dtrace compilation error that John might be able to help us interpret: $ dtrace -o /Users/rkern/hg/Python-2.5.4/Include/phelper.h -DPHELPER -I. -IInclude -I/Users/rkern/hg/Python-2.5.4/Include -C -h -s /Users/rkern/hg/Python-2.5.4/Include/phelper.d dtrace: failed to compile script /Users/rkern/hg/Python-2.5.4/Include/phelper.d: line 110: relocation remains against user symbol D``PyEval_EvalFrameEx (offset 0x5) I also tried running this without -DPHELPER as a regular DTrace script rather than a ustack helper and ran into a problem that I've noticed with any OS X Python build I've tried. I cannot seem to probe any of the C functions in the Python interpreter. There are no simply pid$target:a.out:: probes available. I'm wondering if that is an effect of their being in a .framework, but I think I've been able to probe other symbols other .frameworks. |
|||
msg86278 - (view) | Author: Laszlo (Laca) Peter (laca) | Date: 2009-04-22 07:34 | |
Look at these lines at the beginning of phelper.d: #if defined(__i386) #define startframe PyEval_EvalFrameEx #define endframe PyEval_EvalCodeEx #elif defined(__amd64) #define PyEval_EvalFrameEx PyEval_EvalFrameExReal #define startframe PyEval_EvalFrameExReal #define endframe PyEval_EvalCodeEx #elif defined(__sparc) #define PyEval_EvalFrameEx PyEval_EvalFrameExReal #define startframe PyEval_EvalFrameEx #define endframe PyEval_EvalFrameExReal #endif You may need to adjust these if your compiler defines something different from the __i386 or __amd64. I guess an #else #error your architecture was not recognized #endif would be useful here. |
|||
msg86279 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-22 07:45 | |
This is on an Intel Core 2 Duo running OS X 10.5.6. __i386 is defined. |
|||
msg86290 - (view) | Author: John Levon (movement) | Date: 2009-04-22 12:10 | |
Robert, I have no idea how Mac OS does pstack helpers without generating object files, sorry. > no simply pid$target:a.out:: probes available. Hmm. Try adding -Z to see if that helps. > /Users/rkern/hg/Python-2.5.4/Include/phelper.d: line 110: relocation > remains against user symbol D``PyEval_EvalFrameEx (offset 0x5) This is trying to tell you that there's no such function. Indeed, this isn't mentioned in my original patches, have you been editing it? |
|||
msg86322 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-22 20:00 | |
John, -Z does not appear to help: $ sudo dtrace -Z -n 'pid$target::PyEval_EvalFrameEx:entry' -c python dtrace: description 'pid$target::PyEval_EvalFrameEx:entry' matched 0 probes I'm not sure how that would help. If I'm reading the man page correctly, that just let's dtrace continue on even when the probe specifiers do not match anything. It still doesn't let me probe these functions. The only modification I made to phelper.d was to add the stanza of #defines given a few messages up. The failing line expands in the C preprocessor to this line (expanding at_evalframe(), then startframe and endframe): dtrace:helper:ustack: /((uintptr_t)arg0 >= ((uintptr_t)&``PyEval_EvalFrameEx) && (uintptr_t)arg0 < ( (uintptr_t)&``PyEval_EvalCodeEx))/ This is based on the phelper.d in dtrace.diff attached to this ticket by Skip. It is the same (modulo my #defines) as the phelper.d in Python26-07-dtrace.diff from OpenSolaris as linked by Ted. Hmm, I am applying this to Python 2.5.4, though. Perhaps I need to go back to the Python25-07-dtrace.diff, which does appear to use PyEval_EvalFrame. PyEval_EvalFrameEx is in Python 2.5, so I thought I would try the latest and greatest. If I were to process it with dtrace -h, do you think that I could name it as a .c and compile it into an object file with gcc? Or does dtrace -G do significantly more stuff behind the scenes to generate its object file? |
|||
msg86326 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-04-22 20:24 | |
Sorry, I've been away from this issue. I was sort of hoping the Sun and Apple folks would just work things out amongst themselves and present us with a fait accompli. ;-) I'll try to mess around with this a little. Robert> $ sudo dtrace ... Perhaps not quite on-topic for this tracker item, but it bugs me that the mere compilation of a D script requires root privileges. I guess there's not much we can do about this other than complain to Sun and Apple though. Skip |
|||
msg86330 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-22 20:42 | |
Skip> Perhaps not quite on-topic for this tracker item, but it bugs me that the Skip> mere compilation of a D script requires root privileges. It doesn't. "dtrace -G" and "dtrace -h" (the only "mere compilation" that dtrace does) run without root privileges. That line you quoted was actually probing a process and thus needed root privileges. |
|||
msg86335 - (view) | Author: John Levon (movement) | Date: 2009-04-22 21:17 | |
Yes, my mistake. I noticed this typo in the original patch: 207 +PyObject * 208 +PyEval_EvalFrameexEx(PyFrameObject *f, int throwflag)) Can you: - verify that HAVE_DTRACE is indeed defined for ceval.c - do an nm on ceval.o and look which Eval* functions you do have Somehow, PyEval_EvalFrameEx seems not to be ending up in the .o file. It should be easy for you to work out why... |
|||
msg86337 - (view) | Author: Ted Leung (twleung) | Date: 2009-04-22 21:21 | |
On Apr 22, 2009, at 1:24 PM, Skip Montanaro wrote: > > Skip Montanaro <skip@pobox.com> added the comment: > > Sorry, I've been away from this issue. I was sort of hoping the Sun > and > Apple folks would just work things out amongst themselves and > present us > with a fait accompli. ;-) I'll try to mess around with this a little. I've been in touch with some folks at Apple about this, but it's not at the very top of their priority list.... |
|||
msg86338 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-22 21:26 | |
Ah, duh, of course. The problem here with PyEval_EvalFrameEx is that I don't have ceval.o on the command line *at all* since OS X's dtrace doesn't support -G. It doesn't appear to accept ceval.o with -h, either, so I suppose that adding the ustack helper may simply be impossible unless Apple adds the -G flag. Thanks for you help John. |
|||
msg86343 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-22 22:12 | |
Hmm, wait a second. Never mind. The Solaris patches don't have ceval.o on the line for compiling phelper.o, either. If dtrace needs to resolve the symbol PyEval_EvalFrameEx in an object file, how does it know to look in ceval.o for the phelper.d? WITH_DTRACE is defined in ceval.c and PyEval_EvalFrameEx is in ceval.o, but I don't understand how that relates to compiling phelper.d. |
|||
msg86384 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-23 23:43 | |
Is there any interest in my expanding the list of probes? Ruby has quite a few more than function-entry and function-return, to give some examples of what is possible: http://dev.joyent.com/projects/ruby-dtrace/wiki/Ruby+DTrace+probes+and+arguments I think that adding probes that correspond to PyTrace_LINE and PyTrace_EXCEPTION would be straightforward and worthwhile. PyTrace_C_* may also be worthwhile, but you can probably accomplish similar things with the normal pid probes if you know the C function names (although something like printing the name of a raised exception will probably require a dedicated probe). Adding probes to replicate what the LLTRACE configuration option did, but dynamically, might be interesting. |
|||
msg86385 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2009-04-23 23:46 | |
> Is there any interest in my expanding the list of probes? Definitively!!!. |
|||
msg86389 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-04-24 02:03 | |
Robert> Is there any interest in my expanding the list of probes? Yes. Jeff Garrett (a guy I work with) added some more DTrace probes to a 2.4 source tree at work. I mentioned them in an earlier message. I'll check with him at work tomorrow and see about making sure they can be released. I think it's important to get Sun and Apple in sync as well. Maybe more important than adding new probes. (Though if the Python community drives the creation of new probes perhaps both will pick them up.) Skip |
|||
msg86392 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-24 03:24 | |
We could probably merge Apple's and Sun's probes without too much trouble. Apple simply extended function-entry to include the argcount in addition to Sun's (filename, funcname, lineno) arguments. We could use Apple's probe while retaining compatibility with Sun-oriented DTrace scripts already in existence. The two different function-returns are a bit of a problem. I recommend starting with Sun's argument list because there are already a number of quite useful scripts that use it in the DTraceToolkit. We could extend Sun's argument list to add the *object pointer, but I don't actually know of a use case. I haven't seen a script in the wild that uses it. It seems like it would be tricky to write something in a DTrace script that could make much use of it besides printing out the address. Maybe you could navigate your way down to the type name, but that might be unreliable. I suggest adding the *object pointer argument only if we can devise a use case or if one of the Apple folks pop up with one. I don't feel too bad making Apple devs modify a single character from their internal scripts if they don't make them public. |
|||
msg86396 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-24 05:02 | |
James McIlree from Apple has informed me on dtrace-discuss that ustack helpers cannot currently be built on OS X. Bummer. |
|||
msg86399 - (view) | Author: Robert Kern (robert.kern) | Date: 2009-04-24 06:45 | |
Ah, I misread the Apple function-return probe code. Its extra argument is the type name of the return object or "error" if an exception was raised, not the returned object itself. Could be useful. |
|||
msg96154 - (view) | Author: Mark Wielaard (mjw) | Date: 2009-12-08 21:27 | |
I took the patch and tweaked it a little so that it works with systemtap (at least for the function entry/exit probes). It is against 2.6.2 since that was what was in the Fedora 12 src package I was using. Cleaned up the configure checks a little, but otherwise it is almost exactly as is. Also tracked in Fedora as https://bugzilla.redhat.com/show_bug.cgi?id=545179 |
|||
msg99986 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-02-24 01:23 | |
can I ask if this (very useful) feature is on time for Python 2.7? |
|||
msg100041 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2010-02-24 15:56 | |
Jesus> can I ask if this (very useful) feature is on time for Python 2.7? You can ask, but I suspect you'd be disappointed in the answer. Do you have time to look at the issue? The biggest sticking point in my mind is coming up with a uniform set of probes which would be supported by both Sun and Apple. Skip |
|||
msg100169 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-02-26 19:00 | |
I have no experience integrating DTrace probes, so I guess would be a bit risky to trying to deal with this myself, since 2.7 deadline is pretty close. If the only remaining issue is deciding about supporting SUN or Apple "style", and we have the risk of delaying this so much that we lose the 2.7 train, would be better to decide in one direction (I don't know about the difference, so I can't talk) and ship "something". That is, if there is no reason to decide for A or B, we should use a coin to decide, instead of losing the opportunity of 2.7. DTrace is an amazing technology! What do you think? |
|||
msg100173 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2010-02-26 20:41 | |
At Red Hat we've done some work on this feature. I'm sorry for not updating this issue, I was swamped with tasks both pre-PyCon and during the event; I did show the following to various folks at PyCon; I attempted to find Skip at PyCon but was unsuccessful) Unfortunately I have two further versions of this patch for you :( High level summary ------------------ SystemTap is a similar tool for DTrace, but for Linux. At Red Hat we've been looking at implementing this using SystemTap. SystemTap makes some attempts to be at least somewhat compatible with DTrace, and we are able to get the DTrace support to build against SystemTap. I measured the performance and found that some versions of the patch impose a significant performance penalty. We have a version of the patch that works with SystemTap on Linux, and appears not to impose a performance penalty for the cases I think are important (see notes below) I have also ported the patch to Python 3 (I pass the strings back to the probe support as UTF-8 encoded) I hope that there's enough commonality between the SystemTap and DTrace patches to permit common parts to be committed to Python, perhaps eventually supporting both. Having said that, I don't have any systems that can run DTrace. I've chosen to build the upcoming Fedora 13's Python 2 and Python 3 rpms with this support enabled (I'm a Python guy, not a SystemTap guy, FWIW, but we have a strong in-house SystemTap team). We've been tracking this downstream for Fedora, our detailed discussions are within this bug in our downstream tracker: https://bugzilla.redhat.com/show_bug.cgi?id=545179 along with status: https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python and documentation: https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python_2 Details ------- SystemTap supplies a "dtrace" executable implemented in Python, which mimics the "dtrace" binary. $ which dtrace /usr/bin/dtrace $ file $(which dtrace) /usr/bin/dtrace: python script text executable $ dtrace --help Usage /usr/bin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>] Where -h builds a systemtap header file from the .d file -C when used with -h, also run cpp preprocessor -o specifies an explicit output file name, the default for -G is file.o and -h is file.h -I when running cpp pass through this -I include Path -s specifies the name of the .d input file -G builds a stub file.o from file.d, which is required by some packages that use dtrace. I'm attaching a version of the patch against svn trunk. This adds a "--with-dtrace" configuration option (I filed issue 7997 to better document how to do this). Tested on a Fedora 12 box and works, with: systemtap-1.1-2.fc12.i686 systemtap-sdt-devel-1.1-1.fc12.i686 systemtap-runtime-1.1-2.fc12.i686 kernel-PAE-2.6.31.12-174.2.3.fc12.i686 kernel-PAE-devel-2.6.31.12-174.2.3.fc12.i686 kernel-PAE-debuginfo-2.6.31.12-174.2.3.fc12.i686 kernel-headers-2.6.31.12-174.2.3.fc12.i686 kernel-debuginfo-common-i686-2.6.31.12-174.2.3.fc12.i686 kernel-firmware-2.6.31.12-174.2.3.fc12.noarch Upon building --with-dtrace (but actually with systemtap), the linked binary containing ceval.o should contain a "probes" section (the following is a statically-linked build, hence the probes are in "./python"): $ eu-readelf -S ./python | grep probes [25] .probes PROGBITS 081f9d28 1b0d28 000054 0 WA 0 0 8 $ eu-readelf -x .probes python Hex dump of section [25] '.probes', 84 bytes at offset 0x1b0d28: 0x00000000 66756e63 74696f6e 5f5f656e 74727900 function__entry. 0x00000010 50524231 00000000 289d1f08 00000000 PRB1....(....... 0x00000020 a50e0f08 00000000 66756e63 74696f6e ........function 0x00000030 5f5f7265 7475726e 00000000 50524231 __return....PRB1 0x00000040 509d1f08 00000000 1a0f0f08 00000000 P............... 0x00000050 00000000 .... The above contains locations of "nop" instructions within the binary. Systemtap generates kernel modules, and it uses this information to generate modules that monkeypatch the "nop" instruction at runtime to jump to code that gathers the pertinent information (filename, lineno etc) and returns that information to the kernel, and thus back to the user-space probe process. Example of use -------------- Here's an example of running regrtest whilst probing python to see function call and return during a run of the regression test suite (all in one shell command; obviously it would be saner to split the systemtap probe into a separate function): [david@brick trunk-stap]$ stap -v -e'probe python.function.entry = process("./python").mark("function__entry") { filename = user_string($arg1); funcname = user_string($arg2); lineno = $arg3; } probe python.function.return = process("./python").mark("function__return") { filename = user_string($arg1); funcname = user_string($arg2); lineno = $arg3; } probe python.function.entry { printf("%s => %s in %s:%d\n", thread_indent(1), funcname, filename, lineno); } probe python.function.return {printf("%s <= %s in %s:%d\n", thread_indent(-1), funcname, filename, lineno);}' \ -c "./python Lib/test/regrtest.py" Output is as follow (note that I used -v to get verbose info on systemtap's passes): Pass 1: parsed user script and 65 library script(s) using 20132virt/12268res/2052shr kb, in 220usr/10sys/242real ms. Pass 2: analyzed script: 2 probe(s), 12 function(s), 2 embed(s), 2 global(s) using 24868virt/14788res/3392shr kb, in 20usr/10sys/37real ms. Pass 3: using cached /home/david/.systemtap/cache/26/stap_2689ea1aadf52992794511e0d6996aca_5352.c Pass 4: using cached /home/david/.systemtap/cache/26/stap_2689ea1aadf52992794511e0d6996aca_5352.ko Pass 5: starting run. 0 python(6769): => <module> in /home/david/coding/python-svn/trunk-stap/Lib/site.py:59 1063 python(6769): => <module> in /home/david/coding/python-svn/trunk-stap/Lib/os.py:22 2510 python(6769): => <module> in /home/david/coding/python-svn/trunk-stap/Lib/posixpath.py:11 2756 python(6769): => <module> in /home/david/coding/python-svn/trunk-stap/Lib/stat.py:4 2792 python(6769): <= <module> in /home/david/coding/python-svn/trunk-stap/Lib/stat.py:94 3015 python(6769): => <module> in /home/david/coding/python-svn/trunk-stap/Lib/genericpath.py:5 3051 python(6769): <= <module> in /home/david/coding/python-svn/trunk-stap/Lib/genericpath.py:85 3713 python(6769): => <module> in /home/david/coding/python-svn/trunk-stap/Lib/warnings.py:1 3948 python(6769): => <module> in /home/david/coding/python-svn/trunk-stap/Lib/linecache.py:6 3984 python(6769): <= <module> in /home/david/coding/python-svn/trunk-stap/Lib/linecache.py:68 (etc) Performance ----------- There are four cases here from a performance perspective: (a) the source code without the patch (b) the source code with the patch, configured --without-dtrace (c) the source code with the patch, configured --with-dtrace, but without any "live" probes using the static marker (d) the source code with the patch, configured --with-dtrace, with one or more "live" probes using the static marker The performance goals I've been working towards are: - that (a) and (b) should have identical behavior, and - that (b) and (c) should have no significant performance differences i.e. having the markers compiled in shouldn't cost anything (as a packager of Python I want to be able to supply a common build, and the vast majority of my users won't be using the markers) - that it's acceptable for (d) to have a moderate performance cost compared to (c) I used the unladen-swallow benchmark suite to examine performance. Our initial version of the patch (based on http://bugs.python.org/file12861/dtrace.diff ) showed a significant performance penalty for case (c) relative to case (b): 5-14% additional time being taken on most benchmarks, in rough (but, I believe, credible) tests. Some notes on this can be seen at https://bugzilla.redhat.com/show_bug.cgi?id=545179#c12 I believe this regression is due to the generation and storing of f_calllineno at every frame entry. We removed this and further benchmarking showed an insignificant performance difference between (b) and (c), though I wasn't as rigorous as I could have been with these benchmarks. Some notes on this can be seen at https://bugzilla.redhat.com/show_bug.cgi?id=545179#c14 Case (d) probably has a significant penalty relative to (c), but I don't regard that as a blocker. I haven't yet attempted to measure the performance impact of live probes. Other issues ------------ If I'm reading things correctly, the marker is actually instrumenting frame entry and exit, not function entry and exit, so it probably needs renamimg. There are many other possible static markers that could be implemented, but it's probably easiest to do this one first, then add more later (We brainstormed some ideas here: https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python ) In my Fedora rpm packages I also supply a "tap set" (a kind of SystemTap module, for code reuse), that modularizes parts of the tracing above to minimize boilerplate when writing a .stp script that uses the markers. This can be seen here: http://cvs.fedoraproject.org/viewvc/rpms/python/devel/libpython.stp?view=markup I also include an example script, which can be seen here: http://cvs.fedoraproject.org/viewvc/rpms/python/devel/systemtap-example.stp?view=markup Hope this is helpful; sorry if this is muddying the waters further :( Dave |
|||
msg100174 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2010-02-26 20:51 | |
Attached is the patch I'm currently applying in Fedora 13 to Python 3. The patch is actually against the 3.1.1 tarball, rather than SVN; sorry (swamped with post-pycon tasks here), as that's what I've been testing this work against. The patch is somewhat different to the Python 2 case, in that the filename and module name are stored internally by Python as unicode. The markers convert these to UTF8 for the case when the markers are in use by an active probe. Unfortunately it's now possible for this to fail (e.g. low memory). An example of using this for a non-Latin 1 script can be seen towards the end of: https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python_2 |
|||
msg106208 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-05-21 00:01 | |
Ping... We already missed the 2.7 boat. Any hope for 3.2? |
|||
msg110693 - (view) | Author: Mark Lawrence (BreamoreBoy) * | Date: 2010-07-18 21:50 | |
Comments suggest this is wanted, can our build experts get this into 3.2? |
|||
msg110699 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-07-18 23:41 | |
+1! |
|||
msg110700 - (view) | Author: Alexander Belopolsky (belopolsky) * | Date: 2010-07-19 00:07 | |
If any RedHat/Fedora people are tuned in, can you give us an update on Systemtap/DTrace support in RedHat/Fedora python? I understand that one of the main problems with bringing this upstream was the lack of uniformity between Apple and Sun approaches. These days Apple and Sun(Oracle) are increasingly unfriendly to open source, so with a third solution available from RedHat, the choice seems obvious. |
|||
msg110718 - (view) | Author: Ronald Oussoren (ronaldoussoren) * | Date: 2010-07-19 07:45 | |
On 19 Jul, 2010, at 1:07, Alexander Belopolsky wrote: > > Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment: > > If any RedHat/Fedora people are tuned in, can you give us an update on Systemtap/DTrace support in RedHat/Fedora python? I understand that one of the main problems with bringing this upstream was the lack of uniformity between Apple and Sun approaches. These days Apple and Sun(Oracle) are increasingly unfriendly to open source, so with a third solution available from RedHat, the choice seems obvious. What do you mean by increasing unfriendly in Apple's case? I haven't noted any change from them in the last few years. Not that their (perceived) unfriendlyness is relevant. I'm +1 on merging this feature having dtrace support is pretty convenient. Ronald |
|||
msg117490 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2010-09-27 23:50 | |
Updated py3k version of patch. Changes: - renamed the probepoints: "function__entry" -> "frame__entry" "function__return" -> "frame__exit" as I believe this better describes what these do - added a test suite: Lib/test/test_systemtap.py - reformatted to use 4-space indentation - slightly reworked the configure.in code Some error-handling issue remain (see the FIXMEs in the code) Tested and working on Fedora 13 x86_64, using: systemtap-1.3-1.fc13.x86_64 kernel-2.6.33.3-84.fc13.x86_64 kernel-devel-2.6.33.3-84.fc13.x86_64 kernel-headers-2.6.33.3-84.fc13.x86_64 kernel-debuginfo-2.6.33.3-84.fc13.x86_64 kernel-debuginfo-common-x86_64-2.6.33.3-84.fc13.x86_64 |
|||
msg117510 - (view) | Author: Mark Wielaard (mjw) | Date: 2010-09-28 10:36 | |
> - renamed the probepoints: > "function__entry" -> "frame__entry" > "function__return" -> "frame__exit" > as I believe this better describes what these do Are you sure you want to do this? You are right that it better describes the action that these probe points signify. But since previous versions of this patch have been used and applied people might be depending on the old names. It is not a very big issue, we can always write a tapset to translate the old to new names. But it is an interface break for anybody who might already have scripts around. |
|||
msg117514 - (view) | Author: anatoly techtonik (techtonik) | Date: 2010-09-28 12:33 | |
I don't understand the last request. Is there already a practice to depend on patches that were applied by people, but were not released, reviewed or documented in some PEP? |
|||
msg117515 - (view) | Author: Mark Wielaard (mjw) | Date: 2010-09-28 13:14 | |
The original patch was created to be as close as possible to the support that Sun and Apple added to their python implementation for Solaris and MacOS. Changing the probe point names would make the current patch slightly different for scripts written against those. That isn't a major problem, and can be worked around in user scripts or a tapset that translates the old names to new names. But it is something to consider before changing the probe names. |
|||
msg117519 - (view) | Author: anatoly techtonik (techtonik) | Date: 2010-09-28 13:57 | |
So, it was released. But why they didn't send their patches here? We could review them, or we want Sun and Apple create Python specifications unilaterally? It would be best to contact them and notify about the change. Is it something a PSF should do? |
|||
msg117521 - (view) | Author: John Levon (movement) | Date: 2010-09-28 14:01 | |
Anatoly, see above. We have done exactly that. (Honestly, when I first wrote the changes, I saw them as too hacky to ever be accepted upstream). This name change seems frivolous and will indeed break existing scripts. |
|||
msg118574 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2010-10-13 20:08 | |
Updated version of patch attached (against py3k; r85426) I've changed the names back to "function__entry" and "function__return". |
|||
msg118575 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-10-13 20:11 | |
What do we need to have this ready for Python 3.2?. The oportunity window is closing fast, and 3.3 is two years away... So now or never... |
|||
msg119263 - (view) | Author: Ronald Oussoren (ronaldoussoren) * | Date: 2010-10-21 05:58 | |
I've the same question as Jesús Cea Avión: what is needed to get this in 3.2? This would IMHO be a useful feature. |
|||
msg119270 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-10-21 09:15 | |
> I've the same question as Jesús Cea Avión: what is needed to get this > in 3.2? Same as usual: someone to review, apply, commit. Why do you ask? |
|||
msg119556 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-10-25 16:03 | |
I am trying to review this for 3.2, but I am having some issues. For instance, "include/pydtrace.d" is not present in the last patch. Please, post a patch with all the required changes in the same (patch) file. Hurry, we are still on track for 3.2. :-). Another question: I am not able to decide between Sun/Apple style, or breaking dtrace scripts compatibility completely. Anybody has an opinion about this?. Is this actually important?. Are there so many legacy dtrace scripts out there?. |
|||
msg119557 - (view) | Author: Alexander Belopolsky (belopolsky) * | Date: 2010-10-25 16:15 | |
2010/10/25 Jesús Cea Avión <report@bugs.python.org>: .. > Another question: I am not able to decide between Sun/Apple style, or breaking dtrace scripts > compatibility completely. Anybody has an opinion about this?. Is this actually important?. Are > there so many legacy dtrace scripts out there?. > I would say compatibility with Sun/Apple probes should not stand in the way of implementing this in cpython. Of course familiarity to existing users is a consideration, but in cases where self-consistency can be improved, I don't think we should be overly concerned about legacy scripts. I would estimate that 90% of future users will never have used either Sun or Apple probes, 9% will have used Apple and 1% Sun. (This is completely unscientific guess, of course.) |
|||
msg119559 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2010-10-25 16:27 | |
Updated version of the patch; this ought to contain Include/pydtrace.d: $ diffstat /tmp/py3k-add-systemtap-2010-10-25.patch Include/pydtrace.d | 10 Lib/test/test_systemtap.py | 89 ++++++ Makefile.pre.in | 10 Python/ceval.c | 75 +++++ configure | 576 +++++++++++++++++++++++---------------------- configure.in | 34 ++ pyconfig.h.in | 3 7 files changed, 522 insertions(+), 275 deletions(-) Patch contains FIXMEs (sorry), which clearly need addressing. As for the objectives, do folks here agree with the "Performance" notes in http://bugs.python.org/issue4111#msg100173 ? |
|||
msg119561 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-10-25 16:54 | |
Malcolm, does your last patch address the performance issue?. Ideally, dtrace support should be compiled in by default, so performance issues are important. Idealy, performance difference between compiling dtrace or not should be negligible. Until you actually use it, of course. |
|||
msg119562 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-10-25 17:00 | |
We need some documentation, too. Maybe a new chapter, or a new section in the debug chapter. Better the first, since this is not only for debugging, but for performance study too. |
|||
msg119563 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-10-25 17:02 | |
Compiling the code WITHOUT dtrace support gives an error: """ Undefined first referenced symbol in file __dtrace_python___function__entry ./libpython3.2m.so __dtraceenabled_python___function__return ./libpython3.2m.so __dtrace_python___function__return ./libpython3.2m.so __dtraceenabled_python___function__entry ./libpython3.2m.so ld: fatal: Symbol referencing errors. No output written to python collect2: ld returned 1 exit status """ |
|||
msg119564 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-10-25 17:12 | |
Compiling WITH dtrace... shows the same error :-???? |
|||
msg119566 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-10-25 17:26 | |
I am using Solaris 10, but the configure script detects "apple" implementation. |
|||
msg119578 - (view) | Author: Frank Ch. Eigler (fche) | Date: 2010-10-25 20:39 | |
I believe the problem jcea is experiencing is with the solaris (/linux?) branch of the configure.in: if dtrace -G -o /dev/null -s $srcdir/Include/pydtrace.d 2>/dev/null It seems solaris doesn't like the -o /dev/null part. Try specifying some real temporary file name instead. |
|||
msg119579 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2010-10-25 21:20 | |
Updated patch, removing the FIXMEs, and slightly reworking the test code. I've wrapped the whole of get_frame_marker_info with a PyErr_Fetch/PyErr_Restore pair: the PyUnicode_AsUTF8String calls could fail with a MemoryError, and we don't want to confuse the regular exception handling within ceval. I'm not sure how to write a unit test to test for this: perhaps we could corrupt the __code__ instance so that co_filename is not a PyUnicodeObject, leading to a TypeError within the function, but that's a readonly attribute. Any ideas? I've also added a unit test for a non-ASCII script (Lib/test/systemtap_sample_☠.py), containing a non-ASCII identifier (文字化け). The non-ASCII script name (Lib/test/systemtap_sample_☠.py) may be controversial: do we have anything like that in the source tree yet? Is there any risk of messing up the build across platforms, or of impacting the Hg migration? Still to-do: - address the "Undefined symbol" issues seen by jcea. - documentation - doublecheck performance |
|||
msg119580 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-10-25 21:42 | |
> I've wrapped the whole of get_frame_marker_info with a > PyErr_Fetch/PyErr_Restore pair: the PyUnicode_AsUTF8String calls could > fail with a MemoryError, and we don't want to confuse the regular > exception handling within ceval. If PyUnicode_AsUTF8String() is meant to encode a filename, you should use PyUnicode_EncodeFSDefault() instead. > The non-ASCII script name (Lib/test/systemtap_sample_☠.py) may be > controversial: do we have anything like that in the source tree yet? > Is there any risk of messing up the build across platforms, or of > impacting the Hg migration? It would be better to generate the sample dynamically, so that users with an incompatible locale or filesystem aren't prevented from checking out the source. |
|||
msg119583 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2010-10-25 22:52 | |
> It would be better to generate the sample dynamically, so that users > with an incompatible locale or filesystem aren't prevented from checking > out the source. Thanks: am attaching updated patch: I've removed Lib/test/systemtap_sample_☠.py, and now generate a similarly-named file during the test, using test.support.TEST_FN and unlink Still TODO: - address pitrou's concerns about PyUnicode_AsUTF8String from msg119580 - address the "Undefined symbol" issues seen by jcea (msg119563 onwards) - documentation - doublecheck performance - perhaps add a systemtap "tapset", and demo code using it (like I did in Fedora's python3 RPMs) - anything else I've missed :) |
|||
msg119584 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2010-10-25 23:05 | |
I should note that I've only ever been testing this with SystemTap, on Linux. I don't have a box with DTrace, and I've never directly used it. I wouldn't today be able to diagnose a buildbot failure related to DTrace (I believe I would with systemtap, fwiw). Are there any DTrace experts around who would be willing to handle the DTrace side of this? If not, would it be reasonable to make this issue be only explicitly about "systemtap"? (e.g. perhaps change the "configure" argument accordingly?) Alternatively, given that this bug originally started as an RFE about DTrace, should we split out systemtap as a separate RFE? I'm sorry if I've "muddied the waters" by doing this. For example, the only test coverage I've written (test_systemtap.py) checks for the presence of a "stap" executable, and skips the tests if it's not found. I'm not familiar enough with DTrace to create the same for DTrace. FWIW (in response to IRC question): "thread_indent" is documented here: http://sourceware.org/systemtap/SystemTap_Beginners_Guide/systemtapscript-handler.html#thread_indent It looks like it may be systemtap-specific; however the only usage is within test_systemtap.py, guarded by the presence of a "stap" binary, skipping the tests if it is not found. |
|||
msg119621 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-10-26 17:29 | |
For the record, replacing /dev/null with conftest.out in the configure test solves the detection problem (and allows Python to build cleanly). However, there is then a problem in test_systemtap (even when replacing "stap" with "dtrace") since the syntax for scripts doesn't seem compatible. In any case, the trivial fix for configure: diff -r 777b171a63ae -r 1784ac25b52e configure --- a/configure Tue Oct 26 18:50:46 2010 +0200 +++ b/configure Tue Oct 26 18:54:09 2010 +0200 @@ -9178,7 +9178,7 @@ $as_echo "$with_dtrace" >&6; } if test ! -z "$with_dtrace" then - if dtrace -G -o /dev/null -s $srcdir/Include/pydtrace.d 2>/dev/null + if dtrace -G -o conftest.out -s $srcdir/Include/pydtrace.d 2>/dev/null then $as_echo "#define WITH_DTRACE 1" >>confdefs.h diff -r 777b171a63ae -r 1784ac25b52e configure.in --- a/configure.in Tue Oct 26 18:50:46 2010 +0200 +++ b/configure.in Tue Oct 26 18:54:09 2010 +0200 @@ -2466,7 +2466,7 @@ AC_MSG_RESULT([$with_dtrace]) if test ! -z "$with_dtrace" then - if dtrace -G -o /dev/null -s $srcdir/Include/pydtrace.d 2>/dev/null + if dtrace -G -o conftest.out -s $srcdir/Include/pydtrace.d 2>/dev/null then AC_DEFINE(WITH_DTRACE, 1, [Define if you want to compile in Dtrace support]) |
|||
msg119622 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-10-26 17:29 | |
(my last message was about building on OpenSolaris) |
|||
msg121374 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-11-17 18:37 | |
Dave, we need some kind of documentation, if we expect to ship this in Python 3.2. The deadline is only 10-15 days away. Could you write something able to be in the standard documentation?. |
|||
msg122392 - (view) | Author: John Levon (movement) | Date: 2010-11-25 17:55 | |
configure.in has: AC_MSG_RESULT([$with_dtrace]) ... AC_MSG_RESULT($with_dtrace) Why twice? It looks confusing. |
|||
msg122534 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2010-11-27 16:19 | |
Some references to keep around: http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/ http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/Python26-07-dtrace.diff http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/Python-07-dtrace.diff |
|||
msg133102 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-04-06 00:42 | |
Some more references: Read the notes under the slides: https://dgl.cx/2011/01/dtrace-and-perl https://dgl.cx/dtrace http://dtrace.org/blogs/ What do we need to unblock this? |
|||
msg133114 - (view) | Author: anatoly techtonik (techtonik) | Date: 2011-04-06 07:07 | |
2011/4/6 Jesús Cea Avión <report@bugs.python.org>: > > Jesús Cea Avión <jcea@jcea.es> added the comment: > > Some more references: > > Read the notes under the slides: > https://dgl.cx/2011/01/dtrace-and-perl > > https://dgl.cx/dtrace > > http://dtrace.org/blogs/ > > What do we need to unblock this? Summarize 30+ page discussion in a new issue. Blog about it on http://blog.python.org/ -- anatoly t. |
|||
msg133147 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2011-04-06 17:02 | |
jcea: I notice that on 2011-02-22 you made these changes: assignee: dmalcolm -> dino.viehland nosy: +dino.viehland versions: +Python 3.3 -Python 3.2 Did you mean to change the assignee, or was this an accident? |
|||
msg133151 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-04-06 18:04 | |
Malcolm, it was a mistake. I only wanted to change the TARGET. I assign the issue to you again. Dino, I delete you from the nosy list now. Sorry for the inconvenience. My excuses to both. |
|||
msg147388 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-11-10 01:01 | |
Anybody still working on this?. We missed the 2.7 boat. DO NOT MISS THE 3.3 ONE!!! :-) |
|||
msg148209 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-11-23 22:16 | |
This project continues in issue #13405. |
|||
msg160449 - (view) | Author: Dave Malcolm (dmalcolm) | Date: 2012-05-11 20:59 | |
Issue #13405 covers DTrace; I've taken the liberty of also opening issue #14776 to cover SystemTap. I hope that once one of these is in the tree it will be easier to get the other one in. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:56:40 | admin | set | github: 48361 |
2021-03-31 17:30:14 | shihai1991 | set | pull_requests: + pull_request23865 |
2021-03-31 17:25:21 | shihai1991 | set | pull_requests: - pull_request23859 |
2021-03-31 13:51:03 | shihai1991 | set | nosy:
+ shihai1991 pull_requests: + pull_request23859 |
2015-09-02 19:19:44 | cburroughs | set | nosy:
+ cburroughs |
2012-05-11 20:59:48 | dmalcolm | set | messages: + msg160449 |
2011-11-23 22:16:56 | jcea | set | messages: + msg148209 |
2011-11-14 19:56:55 | jcea | set | status: open -> closed resolution: out of date superseder: Add DTrace probes |
2011-11-14 19:43:12 | jcea | link | issue13405 dependencies |
2011-11-10 01:03:38 | bretthoerner | set | nosy:
- bretthoerner |
2011-11-10 01:01:50 | jcea | set | messages: + msg147388 |
2011-06-19 01:22:24 | serverhorror | set | nosy:
+ serverhorror |
2011-04-06 18:04:08 | jcea | set | assignee: dino.viehland -> dmalcolm messages: + msg133151 nosy: - dino.viehland |
2011-04-06 17:02:47 | dmalcolm | set | messages: + msg133147 |
2011-04-06 07:07:11 | techtonik | set | messages: + msg133114 |
2011-04-06 00:42:17 | jcea | set | messages: + msg133102 |
2011-02-23 01:19:30 | jcea | set | assignee: dmalcolm -> dino.viehland nosy: + dino.viehland versions: + Python 3.3, - Python 3.2 |
2010-11-27 16:19:16 | jcea | set | messages: + msg122534 |
2010-11-25 17:55:18 | movement | set | messages: + msg122392 |
2010-11-17 18:37:10 | jcea | set | messages: + msg121374 |
2010-11-17 18:20:17 | hazmat | set | nosy:
+ hazmat |
2010-10-26 17:29:54 | pitrou | set | messages: + msg119622 |
2010-10-26 17:29:11 | pitrou | set | messages: + msg119621 |
2010-10-25 23:05:45 | dmalcolm | set | messages: + msg119584 |
2010-10-25 22:52:35 | dmalcolm | set | files:
+ py3k-add-systemtap-2010-10-25-003.patch messages: + msg119583 |
2010-10-25 21:42:33 | pitrou | set | messages: + msg119580 |
2010-10-25 21:20:51 | dmalcolm | set | files:
+ py3k-add-systemtap-2010-10-25-002.patch messages: + msg119579 |
2010-10-25 20:39:39 | fche | set | nosy:
+ fche messages: + msg119578 |
2010-10-25 20:23:39 | dhduvall | set | nosy:
+ dhduvall |
2010-10-25 17:26:49 | jcea | set | messages: + msg119566 |
2010-10-25 17:12:04 | jcea | set | messages: + msg119564 |
2010-10-25 17:02:01 | jcea | set | messages: + msg119563 |
2010-10-25 17:00:02 | jcea | set | messages: + msg119562 |
2010-10-25 16:54:57 | giampaolo.rodola | set | nosy:
- giampaolo.rodola |
2010-10-25 16:54:10 | jcea | set | messages: + msg119561 |
2010-10-25 16:27:56 | dmalcolm | set | files:
+ py3k-add-systemtap-2010-10-25.patch messages: + msg119559 |
2010-10-25 16:15:50 | belopolsky | set | messages: + msg119557 |
2010-10-25 16:03:03 | jcea | set | assignee: dmalcolm messages: + msg119556 |
2010-10-21 09:15:30 | pitrou | set | nosy:
+ pitrou messages: + msg119270 |
2010-10-21 05:58:37 | ronaldoussoren | set | messages: + msg119263 |
2010-10-13 20:11:23 | jcea | set | messages: + msg118575 |
2010-10-13 20:08:47 | dmalcolm | set | messages: + msg118574 |
2010-09-28 14:01:10 | movement | set | messages: + msg117521 |
2010-09-28 13:57:30 | techtonik | set | messages: + msg117519 |
2010-09-28 13:14:45 | mjw | set | messages: + msg117515 |
2010-09-28 12:33:14 | techtonik | set | messages: + msg117514 |
2010-09-28 10:36:15 | mjw | set | messages: + msg117510 |
2010-09-27 23:50:17 | dmalcolm | set | files:
+ py3k-add-systemtap-2010-09-27.patch messages: + msg117490 |
2010-09-03 08:17:42 | techtonik | set | nosy:
+ techtonik |
2010-09-01 19:59:59 | giampaolo.rodola | set | nosy:
+ giampaolo.rodola |
2010-07-19 07:45:03 | ronaldoussoren | set | files:
+ smime.p7s messages: + msg110718 |
2010-07-19 00:07:31 | belopolsky | set | messages: + msg110700 |
2010-07-18 23:41:18 | jcea | set | messages: + msg110699 |
2010-07-18 21:50:39 | BreamoreBoy | set | nosy:
+ BreamoreBoy messages: + msg110693 |
2010-05-21 02:21:50 | r.david.murray | set | nosy:
+ Garen, - skip.montanaro, prescod priority: normal assignee: prescod -> (no value) keywords: + patch stage: patch review |
2010-05-21 00:01:21 | jcea | set | priority: normal -> (no value) assignee: prescod versions: + Python 3.2, - Python 3.1, Python 2.7 keywords: + needs review, - patch nosy: + prescod, skip.montanaro, - Garen messages: + msg106208 |
2010-05-20 20:37:46 | skip.montanaro | set | assignee: skip.montanaro -> (no value) nosy: - skip.montanaro |
2010-04-15 21:54:28 | Garen | set | nosy:
+ Garen |
2010-02-26 20:51:31 | dmalcolm | set | files:
+ python-3.1.1-systemtap.patch messages: + msg100174 |
2010-02-26 20:45:14 | dmalcolm | set | files: + add-dtrace-or-systemtap-markers-to-trunk.patch |
2010-02-26 20:41:47 | dmalcolm | set | messages: + msg100173 |
2010-02-26 19:00:26 | jcea | set | messages: + msg100169 |
2010-02-24 15:56:00 | skip.montanaro | set | messages: + msg100041 |
2010-02-24 01:23:25 | jcea | set | messages: + msg99986 |
2010-01-15 18:18:33 | glyph | set | nosy:
+ glyph |
2010-01-15 17:45:48 | wsanchez | set | nosy:
+ wsanchez |
2009-12-08 21:27:26 | mjw | set | files:
+ stap-python.patch title: Add DTrace probes -> Add Systemtap/DTrace probes nosy: skip.montanaro, rhettinger, jcea, ronaldoussoren, belopolsky, movement, bretthoerner, laca, twleung, jbaker, robert.kern, sirg3, chrismiles, danchr, dmalcolm, mjw messages: + msg96154 components: - Installation |
2009-12-07 05:35:12 | mjw | set | nosy:
+ mjw |
2009-09-18 13:41:44 | dmalcolm | set | nosy:
+ dmalcolm |
2009-09-18 00:24:35 | jbaker | set | nosy:
+ jbaker |
2009-04-24 06:45:16 | robert.kern | set | messages: + msg86399 |
2009-04-24 05:03:00 | robert.kern | set | messages: + msg86396 |
2009-04-24 03:24:08 | robert.kern | set | messages: + msg86392 |
2009-04-24 02:03:18 | skip.montanaro | set | messages: + msg86389 |
2009-04-23 23:46:10 | jcea | set | messages: + msg86385 |
2009-04-23 23:43:33 | robert.kern | set | messages: + msg86384 |
2009-04-22 22:12:12 | robert.kern | set | messages: + msg86343 |
2009-04-22 21:26:48 | robert.kern | set | messages: + msg86338 |
2009-04-22 21:21:49 | twleung | set | files:
+ unnamed messages: + msg86337 |
2009-04-22 21:17:35 | movement | set | messages: + msg86335 |
2009-04-22 20:42:07 | robert.kern | set | messages: + msg86330 |
2009-04-22 20:24:06 | skip.montanaro | set | messages: + msg86326 |
2009-04-22 20:00:01 | robert.kern | set | messages: + msg86322 |
2009-04-22 12:10:41 | movement | set | messages: + msg86290 |
2009-04-22 07:45:21 | robert.kern | set | messages: + msg86279 |
2009-04-22 07:34:47 | laca | set | messages: + msg86278 |
2009-04-22 06:41:45 | robert.kern | set | messages: + msg86277 |
2009-04-22 06:12:58 | robert.kern | set | messages: + msg86273 |
2009-03-09 11:38:20 | danchr | set | nosy:
+ danchr |
2009-02-23 06:02:59 | chrismiles | set | nosy: + chrismiles |
2009-02-19 15:38:46 | sirg3 | set | nosy: + sirg3 |
2009-02-05 21:12:21 | movement | set | messages: + msg81229 |
2009-02-02 13:11:09 | jcea | set | nosy: + jcea |
2009-01-28 03:06:38 | twleung | set | messages: + msg80691 |
2009-01-27 23:39:51 | skip.montanaro | set | messages: + msg80679 |
2009-01-27 22:05:26 | twleung | set | messages: + msg80671 |
2009-01-27 16:50:41 | movement | set | nosy: + movement |
2009-01-25 22:00:13 | skip.montanaro | set | files:
+ dtrace.diff messages: + msg80542 |
2009-01-24 21:59:06 | brett.cannon | set | nosy: - brett.cannon |
2009-01-24 13:46:43 | skip.montanaro | set | messages: + msg80447 |
2009-01-24 05:17:54 | laca | set | messages: + msg80436 |
2009-01-23 02:00:46 | skip.montanaro | set | messages: + msg80395 |
2009-01-22 22:53:58 | skip.montanaro | set | messages: + msg80388 |
2009-01-22 21:41:29 | twleung | set | files:
+ unnamed messages: + msg80382 |
2009-01-22 21:35:05 | skip.montanaro | set | assignee: skip.montanaro versions: + Python 3.1, Python 2.7, - Python 2.6 messages: + msg80381 nosy: + ronaldoussoren |
2009-01-22 21:15:26 | rhettinger | set | nosy: + rhettinger |
2009-01-22 20:56:07 | robert.kern | set | nosy: + robert.kern |
2008-11-14 06:03:00 | laca | set | nosy:
+ laca messages: + msg75849 |
2008-11-13 19:59:01 | twleung | set | nosy:
+ twleung messages: + msg75834 |
2008-11-13 18:55:33 | brett.cannon | set | messages: + msg75832 |
2008-11-13 16:05:23 | skip.montanaro | set | messages: + msg75828 |
2008-11-13 15:09:43 | bretthoerner | set | messages: + msg75825 |
2008-11-13 03:31:44 | skip.montanaro | set | messages: + msg75811 |
2008-11-13 02:57:59 | bretthoerner | set | messages: + msg75810 |
2008-11-13 02:45:05 | skip.montanaro | set | messages: + msg75809 |
2008-11-13 02:26:51 | belopolsky | set | nosy: + belopolsky |
2008-11-12 18:36:15 | skip.montanaro | set | nosy: + skip.montanaro |
2008-10-12 18:48:20 | brett.cannon | set | nosy:
+ brett.cannon messages: + msg74676 |
2008-10-12 16:08:14 | bretthoerner | create |