classification
Title: threads and profiler don't work together
Type: enhancement Stage: test needed
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: ajaksu2, dbrueck, georg.brandl, nedbat, tim.peters, ysj.ray
Priority: low Keywords:

Created on 2001-02-08 15:53 by dbrueck, last changed 2010-08-22 00:05 by georg.brandl. This issue is now closed.

Messages (8)
msg53075 - (view) Author: Dave Brueck (dbrueck) Date: 2001-02-08 15:53
When a new thread is created, it doesn't inherit from the parent thread the trace and profile functions (sys_tracefunc and sys_profilefunc in PyThreadState), so multithreaded programs can't easily be profiled. 

This may be by design for safety/complexity sake, but the profiler module should still find some way to function correctly. A temporary (and performance-killing) workaround is to modify the standard profiler to hook into threads to start a new profiler for each new thread, and then merge the stats from a child thread into the parent's when the child thread ends.

Here is sample code that exhibits the problem. Stats are printed only for the main thread because the child thread has no profiling function and therefore collects no stats:

import threading, profile, time

def yo():
    for j in range(5):
        print j,

def go():
    threading.Thread(target=yo).start()
    time.sleep(1)

profile.run('go()')
msg53076 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2001-02-09 23:57
Assigned to me but reduced the priority.  I'll take a look at it, but have to suspect it will get reclassified as a Feature Request and moved into PEP 42.
msg53077 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2001-08-10 05:31
Logged In: YES 
user_id=31435

Reassigned to Fred, our current profiler expert.
msg53078 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-10-12 12:40
Logged In: YES 
user_id=849994

Turning into a feature request.
msg81808 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-02-12 19:51
Confirmed as of (around) rev69500.
msg111409 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2010-07-24 01:36
Isn't this problem solved by the threading.settrace and threading.setprofile functions?
msg111889 - (view) Author: ysj.ray (ysj.ray) Date: 2010-07-29 03:55
I don't think this problem still exists now. In the current implementation, there is no "sys_tracefunc" and "sys_profilefunc" in PyThreadState, but "c_profilefunc", "c_profileobj", "c_tracefunc", "c_traceobj" instead. When creating a new thread, the "c_profilefunc" and "c_tracefunc" are inherited from main thread, and the profile function is thread specific, it only collect profile statistic of the executing functions in its own thread, that is, each thread can profile its own executing.


I'd change the example as follows:



def child():
    def yo():
        for j in range(5):
            print(j)
    profile.runctx('yo()', globals(), locals())


def go():
    threading.Thread(target=child).start()
    time.sleep(1)

profile.runctx('go()', globals(), locals())



This will output two profile statistics, one is for main thread, another is for child thread: child(). So if you want to profile a child thread, just call profile.run() in child thread. 

So I don't think this is a problem.
msg114619 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-08-22 00:05
OK, closing as out of date.
History
Date User Action Args
2010-08-22 00:05:47georg.brandlsetstatus: open -> closed
resolution: out of date
messages: + msg114619
2010-07-29 03:55:19ysj.raysetnosy: + ysj.ray
messages: + msg111889
2010-07-24 01:36:00nedbatsetnosy: + nedbat
messages: + msg111409
2009-02-12 19:51:10ajaksu2setnosy: + ajaksu2
stage: test needed
messages: + msg81808
versions: + Python 2.7
2001-02-08 15:53:43dbrueckcreate