classification
Title: Should PyObject_Call() call the profiler on C functions, use C_TRACE() macro?
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, jdemeyer, ppperry, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2017-02-08 15:39 by vstinner, last changed 2018-08-02 07:52 by jdemeyer.

Files
File name Uploaded Description Edit
profiler_c_code.py vstinner, 2017-02-09 11:05
Messages (6)
msg287343 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-08 15:39
call_function() and do_call_core() functions of Python/ceval.c use C_TRACE() macro to call the profiler, but PyObject_Call() and similar functions like _PyObject_FastCallKeywords() don't.

Is it a bug or a deliberate choice for performance?

Since PyObject_Call() and similar functions have now fast paths, I don't think that it's still needed to have these fast paths in ceval.c too. But I kept fast paths in ceval.c because of C_TRACE().
msg287402 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-09 11:05
Output of attached profiler_c_code.py example:
---
haypo@selma$ ./python profiler_c_code.py 
(<frame object at 0xf9d788>, 'call', None)
(<frame object at 0xf9d788>, 'return', 'h')
(<frame object at 0xf9d788>, 'call', None)
(<frame object at 0xf9d788>, 'return', 'e')
(<frame object at 0xf9d788>, 'call', None)
(<frame object at 0xf9d788>, 'return', 'l')
(<frame object at 0xf9d788>, 'call', None)
(<frame object at 0xf9d788>, 'return', 'l')
(<frame object at 0xf9d788>, 'call', None)
(<frame object at 0xf9d788>, 'return', 'o')
(<frame object at 0xf9d498>, 'c_call', <built-in function setprofile>)
---

Except of sys.setprofile(), the profiler doesn't see any C calls! list() and map() are simply "hidden".

So tools like coverage miss a lot of C call? Again, is it a compromise between performance and correctness, or just a regular bug?
msg322810 - (view) Author: Jeroen Demeyer (jdemeyer) * (Python triager) Date: 2018-07-31 19:26
I always assumed that enabling profiling only from the Python bytecode interpreter was a deliberate choice.
msg322833 - (view) Author: (ppperry) Date: 2018-08-01 02:19
issue30990 is related
msg322838 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-08-01 04:17
FYI, _lsprof uses PyObject_Call()
https://github.com/python/cpython/blob/ea68d83933e6de6cabfb115ec1b8888301947369/Modules/_lsprof.c#L120-L123

If PyObject_Call() trigger profiler, lsprof should avoid recursion.
msg322925 - (view) Author: Jeroen Demeyer (jdemeyer) * (Python triager) Date: 2018-08-02 07:52
I would prefer to wait with this issue until PEP 580 has been decided. If it's accepted, it will change function calling code a lot. As I wrote in PEP 580, I was planning to have a PEP on profiling anyway.
History
Date User Action Args
2018-08-02 07:52:08jdemeyersetmessages: + msg322925
2018-08-01 04:17:06inada.naokisetmessages: + msg322838
2018-08-01 02:19:56ppperrysetnosy: + ppperry
messages: + msg322833
2018-07-31 19:26:45jdemeyersetnosy: + jdemeyer
messages: + msg322810
2017-02-09 11:05:29vstinnersetfiles: + profiler_c_code.py

messages: + msg287402
2017-02-08 15:39:38vstinnercreate