classification
Title: reference leaks in pdb
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: xdegaye Nosy List: Jack Liu, blueyed, draghuram, georg.brandl, gregory.p.smith, isandler, pitrou, python-dev, r.david.murray, vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2014-02-25 11:02 by xdegaye, last changed 2019-04-19 11:10 by blueyed. This issue is now closed.

Files
File name Uploaded Description Edit
refleak.patch xdegaye, 2014-02-25 11:02 review
refleak_2.patch xdegaye, 2014-05-25 12:34 review
refleak_3.patch xdegaye, 2014-07-30 12:34 review
refleak_4.patch xdegaye, 2014-08-04 20:35 review
pdb_refleak.py xdegaye, 2016-10-02 09:02
refleak_5.patch xdegaye, 2016-10-02 16:13 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (17)
msg212171 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-02-25 11:02
After the pdb 'continue' command, the signal module owns a reference to Pdb.sigint_handler. On the next instantiation of pdb, the signal module owns a reference to a new sigint_handler method that owns a reference to the previous sigint_handler. As a consequence, the first pdb instance is never freed (as well as the frames that are referenced by this pdb instance). The following test demonstrates the problem that is fixed by the attached patch.

Run the following script:

# START of refleak.py
import sys, gc, pdb

i = 1
while i:
    pdb.set_trace()
    x = 1
    gc.collect(); print(sys.gettotalrefcount())

# END of refleak.py


With the following pdb commands:

$ python refleak.py
> /tmp/test/refleak.py(6)<module>()
-> x = 1
(Pdb) continue
95898
> /home/xavier/tmp/test/refleak.py(5)<module>()
-> pdb.set_trace()
(Pdb) continue
95983
> /tmp/test/refleak.py(6)<module>()
-> x = 1
(Pdb) continue
96068
> /tmp/test/refleak.py(5)<module>()
-> pdb.set_trace()
(Pdb) i = 0
(Pdb) continue
96153
msg212175 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-02-25 13:06
> the first pdb instance is never freed

The first pdb instance is (and all the other pdb instances) never freed until the call to PyOS_FiniInterrupts() in Py_Finalize().
msg212510 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-03-01 16:05
After applying patch 'regrtest.diff' from issue 20746, the command:

  $ ./python -m test -W -R3:3 test_pdb

succeeds now and shows there are no reference leaks.


However, with the following change in Lib/test/test_pdb.py and patch 'regrtest.diff' applied:

diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -34,7 +34,7 @@
     """This tests the custom displayhook for pdb.
 
     >>> def test_function(foo, bar):
-    ...     import pdb; pdb.Pdb(nosigint=True).set_trace()
+    ...     import pdb; pdb.Pdb().set_trace()
     ...     pass
 
     >>> with PdbTestInput([


there are reference leaks:

$ ./python -m test -R 3:3 test_pdb
[1/1] test_pdb
beginning 6 repetitions
123456
......
test_pdb leaked [200, 200, 200] references, sum=600
test_pdb leaked [43, 43, 43] memory blocks, sum=129
1 test failed:
    test_pdb


And now with the proposed patch in the previous msg 212171 'refleak.patch' of this issue that fixes the sigint_handler leak, and also patch 'regrtest.diff' applied:

$ ./python -m test -R 3:3 test_pdb
[1/1] test_pdb
beginning 6 repetitions
123456
......
1 test OK.
msg219084 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-05-25 12:34
An improved patch with a test case.
msg224265 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-07-29 22:28
It's not easy to test the patch because running test_pdb fails with the following error (I also get the error if the patch is not applied):

$ ./python -m test test_pdb test_pdb
[1/2] test_pdb
[2/2] test_pdb
test test_pdb failed -- Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/doctest.py", line 2189, in runTest
    test, out=new.write, clear_globs=False)
AssertionError: Failed doctest test for test.test_pdb.test_next_until_return_at_return_event
  File "/home/haypo/prog/python/default/Lib/test/test_pdb.py", line 603, in test_next_until_return_at_return_event

----------------------------------------------------------------------
File "/home/haypo/prog/python/default/Lib/test/test_pdb.py", line 617, in test.test_pdb.test_next_until_return_at_return_event
Failed example:
    with PdbTestInput(['break test_function_2',
                       'continue',
                       'return',
                       'next',
                       'continue',
                       'return',
                       'until',
                       'continue',
                       'return',
                       'return',
                       'continue']):
        test_function()
Expected:
    > <doctest test.test_pdb.test_next_until_return_at_return_event[1]>(3)test_function()
    -> test_function_2()
    (Pdb) break test_function_2
    Breakpoint 1 at <doctest test.test_pdb.test_next_until_return_at_return_event[0]>:1
    (Pdb) continue
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(2)test_function_2()
    -> x = 1
    (Pdb) return
    --Return--
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(3)test_function_2()->None
    -> x = 2
    (Pdb) next
    > <doctest test.test_pdb.test_next_until_return_at_return_event[1]>(4)test_function()
    -> test_function_2()
    (Pdb) continue
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(2)test_function_2()
    -> x = 1
    (Pdb) return
    --Return--
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(3)test_function_2()->None
    -> x = 2
    (Pdb) until
    > <doctest test.test_pdb.test_next_until_return_at_return_event[1]>(5)test_function()
    -> test_function_2()
    (Pdb) continue
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(2)test_function_2()
    -> x = 1
    (Pdb) return
    --Return--
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(3)test_function_2()->None
    -> x = 2
    (Pdb) return
    > <doctest test.test_pdb.test_next_until_return_at_return_event[1]>(6)test_function()
    -> end = 1
    (Pdb) continue
Got:
    > <doctest test.test_pdb.test_next_until_return_at_return_event[1]>(3)test_function()
    -> test_function_2()
    (Pdb) break test_function_2
    Breakpoint 7 at <doctest test.test_pdb.test_next_until_return_at_return_event[0]>:1
    (Pdb) continue
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(2)test_function_2()
    -> x = 1
    (Pdb) return
    --Return--
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(3)test_function_2()->None
    -> x = 2
    (Pdb) next
    > <doctest test.test_pdb.test_next_until_return_at_return_event[1]>(4)test_function()
    -> test_function_2()
    (Pdb) continue
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(2)test_function_2()
    -> x = 1
    (Pdb) return
    --Return--
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(3)test_function_2()->None
    -> x = 2
    (Pdb) until
    > <doctest test.test_pdb.test_next_until_return_at_return_event[1]>(5)test_function()
    -> test_function_2()
    (Pdb) continue
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(2)test_function_2()
    -> x = 1
    (Pdb) return
    --Return--
    > <doctest test.test_pdb.test_next_until_return_at_return_event[0]>(3)test_function_2()->None
    -> x = 2
    (Pdb) return
    > <doctest test.test_pdb.test_next_until_return_at_return_event[1]>(6)test_function()
    -> end = 1
    (Pdb) continue
msg224279 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-07-30 00:24
See issue20746 for the test_pdb failure when run multiple times.
msg224303 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-07-30 12:34
This is because breakpoints number are class attributes. With the following change, the "./python -m test test_pdb test_pdb" is ok:

$ hg diff
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -614,6 +614,8 @@
     ...     test_function_2()
     ...     end = 1
 
+    >>> from bdb import Breakpoint; Breakpoint.next = 1
+
     >>> with PdbTestInput(['break test_function_2',
     ...                    'continue',
     ...                    'return',

Attached refleak_3.patch fixes this.
msg224769 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-08-04 20:35
After refleak_3.patch, test_pdb_next_command_in_generator_for_loop is the only remaining test that sets a breakpoint without reinitializing the breakpoints numbering first.
As a result, this test may also fail in the future when tests are reordered by the unittest machinery.
refleak_4.patch adds a fix to that (and could be ignored as a problem that should be fixed in a new issue of its own).
msg277678 - (view) Author: Jack Liu (Jack Liu) Date: 2016-09-29 02:16
Will the issue be fixed in Python formal build?
I still meet the same issue with Python 3.5.1. It cost me a bit time to track down this issue.

I'm using pdb to debug a Python script, there are global variables in the script. Duo the leak of sigint_handler, calling gc.collect() doesn't release the global variables until calling PyOS_FiniInterrupts().
msg277679 - (view) Author: Jack Liu (Jack Liu) Date: 2016-09-29 02:19
Now, I have to set nosigint to True to fix the reference leaks issue for my app.
msg277863 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-02 09:02
The problem may still be reproduced by running the attached pdb_refleak.py script. Enter three times 'c' (short for the pdb 'continue' command) and the following line is printed:

    pdb 3 -> pdb 2 -> pdb 1

This shows that the third pdb instance 'pdb 3', owns indirectly a reference to the second and first pdb instances. As the _signal extension module owns a reference to 'pdb 3' through its sigint_handler() method, then it also owns indirectly a reference to 'pdb 1'.
msg277864 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-02 09:10
Oh, test_pdb does not fail anymore in refleak mode as expected (see msg212510 for the change to apply to test_pdb.py in order to reproduce that). 'hg bisect' says that this is happening since:

      The first bad revision is:
      changeset:   103552:ee0bbfd972de
      user:        Łukasz Langa <lukasz@langa.pl>
      date:        Fri Sep 09 22:21:17 2016 -0700
      summary:     Issue #18401: pdb tests don't read ~/.pdbrc anymore

And indeed, this changeset has removed entirely the doctests from the test suite :(
msg277870 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-02 09:22
> And indeed, this changeset has removed entirely the doctests from the test suite :(

Entered new issue 28338.
msg277900 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-02 16:13
Uploaded refleak_5.patch with a simpler test case.
With the patch applied, './python -m test -R 3:3 test_pdb' runs with success.

The '_previous_sigint_handler' is reset at the Pdb prompt - instead of when the signal is triggered - to handle the case where pdb stops at a new set_trace() hard breakpoint after it has been run with 'continue'.  As a side effect, this also fixes issue 22502.
msg278533 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-12 18:20
New changeset 31a2d270c0c3 by Xavier de Gaye in branch '3.5':
Issue #20766: Fix references leaked by pdb in the handling of SIGINT handlers.
https://hg.python.org/cpython/rev/31a2d270c0c3

New changeset 86a1905ea28d by Xavier de Gaye in branch '3.6':
Issue #20766: Merge with 3.5.
https://hg.python.org/cpython/rev/86a1905ea28d

New changeset 8bb426d386a5 by Xavier de Gaye in branch 'default':
Issue #20766: Merge with 3.6.
https://hg.python.org/cpython/rev/8bb426d386a5
msg279310 - (view) Author: Jack Liu (Jack Liu) Date: 2016-10-24 14:46
Good to know the fix will be included Python official builds. Thanks.
msg340540 - (view) Author: daniel hahler (blueyed) * Date: 2019-04-19 11:10
Please see https://bugs.python.org/issue36667 for a followup.

It does not look like moving it to `interaction` is relevant for fixing the leak, is it?

I think it should be restored in both places.
History
Date User Action Args
2019-04-19 11:10:21blueyedsetnosy: + blueyed
messages: + msg340540
2017-03-31 16:36:10dstufftsetpull_requests: + pull_request858
2016-10-24 14:46:25Jack Liusetmessages: + msg279310
2016-10-12 18:25:25xdegayesetstage: patch review -> resolved
2016-10-12 18:24:37xdegayesetstatus: open -> closed
resolution: fixed
2016-10-12 18:20:29python-devsetnosy: + python-dev
messages: + msg278533
2016-10-02 16:13:14xdegayesetfiles: + refleak_5.patch
versions: + Python 3.6, Python 3.7, - Python 3.4
messages: + msg277900

assignee: xdegaye
stage: patch review
2016-10-02 09:22:46xdegayesetmessages: + msg277870
2016-10-02 09:10:51xdegayesetmessages: + msg277864
2016-10-02 09:02:35xdegayesetfiles: + pdb_refleak.py

messages: + msg277863
2016-09-29 02:19:07Jack Liusetmessages: + msg277679
2016-09-29 02:16:50Jack Liusetnosy: + draghuram, isandler, r.david.murray, gregory.p.smith, Jack Liu

messages: + msg277678
versions: + Python 3.5
2014-08-04 20:35:05xdegayesetfiles: + refleak_4.patch

messages: + msg224769
2014-07-30 12:34:43xdegayesetfiles: + refleak_3.patch
2014-07-30 12:34:34xdegayesetmessages: + msg224303
2014-07-30 00:24:39pitrousetnosy: + pitrou
messages: + msg224279
2014-07-29 22:28:51vstinnersetmessages: + msg224265
2014-07-28 22:43:59vstinnersetnosy: + vstinner
2014-05-25 12:34:11xdegayesetfiles: + refleak_2.patch

messages: + msg219084
2014-03-01 16:05:39xdegayesetmessages: + msg212510
2014-02-25 13:06:51xdegayesetmessages: + msg212175
2014-02-25 11:02:34xdegayecreate