classification
Title: Allow to set profile/trace function globally
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.2
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, jon, kristjan.jonsson, pitrou
Priority: normal Keywords: patch

Created on 2010-08-16 17:17 by kristjan.jonsson, last changed 2010-09-09 07:47 by asvetlov. This issue is now closed.

Files
File name Uploaded Description Edit
globaltrace.patch kristjan.jonsson, 2010-08-16 17:17
glob.patch kristjan.jonsson, 2010-08-17 15:24
Messages (10)
msg114054 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2010-08-16 17:17
issue 9609 updates _lsprof.c to be multi-stack aware.  This allows cProfile.Profile() objects to be shared by many threads and provide meaningfull results.

This update makes it more convenient to profile running, multi-threaded, applications.  By adding a set of global (cross thread) profiling hooks that override the per-thread hooks, it is possible to enable and disable profiling/tracing for the entire program.

A multithreaded python could then do something like this:
def ProfileMe(t):
  p = cProfile.Profile()
  p.enable(global=True)
  time.sleep(t)
  p.disable()
  p.print_stats()

A patch is provided, also, an updated _lsprof adding the new 'global' flag to the "enable" function.

(This paradigm is used successfully in EVE, albeit with "global" there meaning all tasklets, to do snapshot profiling of a running server.  The results are displayed on a web page served by the server backend.)
msg114110 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-17 10:47
Adding new "global" objects looks a bit complicated. Why not simply update all thread-local objects when the "global" flag is set?
msg114111 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2010-08-17 10:53
Good question.  Is that simple to do?  I didn't think to look if there were a central list of all python TLS states.
msg114112 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-17 10:58
> Good question.  Is that simple to do?  I didn't think to look if there
> were a central list of all python TLS states.

There's a linked list from the current interpreter state, which in turn
can be got from the current thread state (see Include/pystate.h).
Of course, it would be better to add macros than access the fields
directly.
msg114113 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2010-08-17 11:01
Sounds like a much simpler change.  I'll try that out instead.
msg114128 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2010-08-17 15:24
Here is a new, much simpler patch, which simply sets the flags on all the threads (accessable as a linked list of PyThreadState objects) as suggested by Antoine.

(Note that neither of the patches has the necessary enhancements to _lsprof.c to make the profiling data sensible.  That is the subject of issue 9609)
msg114137 - (view) Author: Jon Parise (jon) Date: 2010-08-17 16:43
There are a few places in the patch where you call the global version immediately followed by the local version.  For example:

+        PyEval_SetGlobalTrace(NULL, NULL);
         PyEval_SetTrace(NULL, NULL);

Isn't the local call now redundant given that the global version clears all of the threadstates already?
msg114162 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2010-08-17 20:29
Indeed it is.  This is a remnant of the previous method.  I will fix it, and add documentation.  I also think I'll leave the change to _lsprof out of this so that this change can stand on its own, irrespective of the profiling engine being used.
msg114164 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2010-08-17 20:45
I just realized that this is probably a redundant change.
We have C apis to get all the Thread states in an interpreter state (I didn't even know there was such a thing as multiple interpreter states, but there!)
This is the PyInterpreterState_ThreadHead() api et al.
From C, all that is missing is a SetTrace api that takes a thread state.

From python, the threading module provides access to all Thread objects, and each of those has a settrace/setprofile method.

To turn on global tracing from cProfile, all that is needed is to iterate over all the Thread objects.
msg114165 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2010-08-17 20:47
Setting this to invalid, since there already are APIs to do this, at least from .py code.
History
Date User Action Args
2010-09-09 07:47:35asvetlovsetnosy: + asvetlov
2010-08-20 18:17:19terry.reedysetstatus: open -> closed
keywords: patch, patch
2010-08-17 20:47:53kristjan.jonssonsetkeywords: patch, patch
resolution: not a bug
messages: + msg114165
2010-08-17 20:45:49kristjan.jonssonsetkeywords: patch, patch

messages: + msg114164
2010-08-17 20:29:56kristjan.jonssonsetkeywords: patch, patch

messages: + msg114162
2010-08-17 16:43:28jonsetnosy: + jon
messages: + msg114137
2010-08-17 15:24:25kristjan.jonssonsetkeywords: patch, patch
files: + glob.patch
messages: + msg114128
2010-08-17 11:01:18kristjan.jonssonsetkeywords: patch, patch

messages: + msg114113
2010-08-17 10:58:02pitrousetmessages: + msg114112
2010-08-17 10:53:00kristjan.jonssonsetkeywords: patch, patch

messages: + msg114111
2010-08-17 10:47:20pitrousetkeywords: patch, patch
nosy: + pitrou
messages: + msg114110

2010-08-16 17:17:13kristjan.jonssoncreate