Author xdegaye
Recipients akl, amaury.forgeotdarc, asvetlov, cfbearden, davin, meador.inge, nedbat, serhiy.storchaka, xdegaye
Date 2018-05-08.20:27:40
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1525811261.17.0.682650639539.issue13044@psf.upfronthosting.co.za>
In-reply-to
Content
Nosying Serhiy as this post attempts to answer one of the questions raised in msg315588 in issue 33328:
> Is it good that the debugger is enabled at the shutdown stage?

This post has four different sections showing that the successive and incremental attempts at fixing the current code to have a destructor traced in the shutdown stage may lead to a segfault.

Section 1 - Current status:
---------------------------
A run with Python 3.6.5 of the following code proposed in the previous msg176993 gives now a different output and the C destructor is still not traced, but now it is not even called:

1 class C:
2     def __del__(self):
3         print('deleted')
4
5 c = C()
6 import pdb; pdb.set_trace()
7 x = 1

$ python test_destructor.py
> test_destructor.py(7)<module>()
-> x = 1
(Pdb) step
--Return--
> test_destructor.py(7)<module>()->None
-> x = 1
(Pdb) step
--Call--
Exception ignored in: <async_generator object _ag at 0x7f3386d18320>
Traceback (most recent call last):
  File "/usr/lib/python3.6/types.py", line 27, in _ag
  File "/usr/lib/python3.6/bdb.py", line 53, in trace_dispatch
  File "/usr/lib/python3.6/bdb.py", line 85, in dispatch_call
  File "/usr/lib/python3.6/pdb.py", line 251, in user_call
  File "/usr/lib/python3.6/pdb.py", line 351, in interaction
  File "/usr/lib/python3.6/pdb.py", line 1453, in print_stack_entry
  File "/usr/lib/python3.6/bdb.py", line 394, in format_stack_entry
TypeError: 'NoneType' object is not callable

Section 2 - Fix using PR 6730:
------------------------------
PR 6730 in issue 33446 fixes a bug in the pdb module that leaks a reference to frame->f_locals through its curframe_locals attribute.
There is no major change when the current Python master branch is run with PR 6730, the destructor is still not traced and not called.

Section 3 - Fix a bdb leak:
---------------------------
After returning from the '__main__' module, the Bdb.bdb instance keeps a reference to the module through its botframe attribute and prevents the corresponding frame from being deallocated at the end of _PyEval_EvalCodeWithName() (the frame reference count is 2 instead of 1).  The attached bdb_leak.diff patch fixes this, and when PR 6730 and bdb_leak.diff are run, the destructor is now traced but this fails with an exception in bdb:

$ /home/xavier/src/python/master/python test_destructor.py
> ./test_destructor.py(7)<module>()
-> x = 1
(Pdb) step
--Return--
> ./test_destructor.py(7)<module>()->None
-> x = 1
(Pdb) step
Exception ignored in: <function C.__del__ at 0x7ff8ffdafd48>
Traceback (most recent call last):
  File "test_destructor.py", line 3, in __del__
  File "test_destructor.py", line 3, in __del__
  File "/home/xavier/src/python/master/Lib/bdb.py", line 88, in trace_dispatch
  File "/home/xavier/src/python/master/Lib/bdb.py", line 115, in dispatch_line
  File "/home/xavier/src/python/master/Lib/pdb.py", line 262, in user_line
  File "/home/xavier/src/python/master/Lib/pdb.py", line 352, in interaction
  File "/home/xavier/src/python/master/Lib/pdb.py", line 1454, in print_stack_entry
  File "/home/xavier/src/python/master/Lib/bdb.py", line 544, in format_stack_entry
ImportError: sys.meta_path is None, Python is likely shutting down

Section 4 - Remove the lazy import:
-----------------------------------
The attached lazy_import.diff patch includes the changes in bdb_leak.diff patch and replaces the lazy imports causing the previous ImportError with imports at the start of the bdb module.  Running test_destructor.py fails now with a segfault:

$ /home/xavier/src/python/master/python test_destructor.py
> ./test_destructor.py(7)<module>()
-> x = 1
(Pdb) step
--Return--
> ./test_destructor.py(7)<module>()->None
-> x = 1
(Pdb) step
> ./test_destructor.py(3)__del__()
-> print('deleted')
Segmentation fault (core dumped)

The gdb back trace is attached at gdb_backtrace.txt.

Conclusion:
-----------
It seems that tracing should be prevented in the shutdown stage and that multiple little bugs have hidden that fact (when tracing is done with pdb).
Not sure if tracing with PyEval_SetTrace() could be allowed in that stage.
BTW coverage.py uses PyEval_SetTrace() and I have noticed that coverage.py does not trace either the destructor in test_destructor.py (nosying Ned).
History
Date User Action Args
2018-05-08 20:27:41xdegayesetrecipients: + xdegaye, amaury.forgeotdarc, nedbat, asvetlov, meador.inge, akl, serhiy.storchaka, davin, cfbearden
2018-05-08 20:27:41xdegayesetmessageid: <1525811261.17.0.682650639539.issue13044@psf.upfronthosting.co.za>
2018-05-08 20:27:41xdegayelinkissue13044 messages
2018-05-08 20:27:40xdegayecreate