http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.6/builds/15/steps/test/logs/stdio

0:02:58 [ 28/405/2] test_atexit failed
......
beginning 6 repetitions
123456

test_atexit leaked [12, 12, 12] references, sum=36
test_atexit leaked [4, 4, 4] memory blocks, sum=12


Code of of the test:

    def test_callbacks_leak(self):
        # This test shows a leak in refleak mode if atexit doesn't
        # take care to free callbacks in its per-subinterpreter module
        # state.
        n = atexit._ncallbacks()
        code = r"""if 1:
            import atexit
            def f():
                pass
            atexit.register(f)
            del atexit
            """
        ret = support.run_in_subinterp(code)
        self.assertEqual(ret, 0)
        self.assertEqual(atexit._ncallbacks(), n)


Hum, I don't understand: the test leaks references on purpose?


To reproduce the bug, use the command:

python -m test -R 3:3 -m test_callbacks_leak test_atexit


Note: Leak isolated my bisect_test.py of bpo-29512.

I tagged this issue as easy. I consider that it's a good exercice for new contributors. Core developers: please let new contributors try to fix it since this issue is not critical, try to explain how to fix it rather than writing the fix ;-)

this issue can be executed on Linux, I think I am going to work on this one.

The failing unit test was added by:

commit 2d350fd8af29eada0c3f264a91df6ab4af4a05fd
Author: Antoine Pitrou <solipsis@pitrou.net>
Date:   Thu Aug 1 20:56:12 2013 +0200

    Issue #18619: Fix atexit leaking callbacks registered from sub-interpreters, and make it GC-aware.


Using git bisect, I found that the leak was introduced by:

commit 6b4be195cd8868b76eb6fbe166acc39beee8ce36
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Mon May 22 21:36:03 2017 -0700

    bpo-22257: Small changes for PEP 432. (#1728)
    
    PEP 432 specifies a number of large changes to interpreter startup code, including exposing a cleaner C-API. The major changes depend on a number of smaller changes. This patch includes all those smaller changes.


To run a git bisection, start with an old commit, 1 month ago: 5d7a8d0c13737fd531b722ad76c505ef47aac96a (May, 1). Spoiler: the test doesn't leak at this bisection.

git bisect reset
git bisect start

git checkout master
make && ./python -m test -R 3:3 -m test_callbacks_leak test_atexit
# test fails
git bisect bad  # bad=test fails (ref leak)

git checkout 5d7a8d0c13737fd531b722ad76c505ef47aac96a
make && ./python -m test -R 3:3 -m test_callbacks_leak test_atexit
# test pass
git bisect good  # good=test pass (no leak)

make && ./python -m test -R 3:3 -m test_callbacks_leak test_atexit
# git bisect good or bad depending on the test result

# ... continue until git bisect finds the commit ...

At the end, you should get the commit 6b4be195cd8868b76eb6fbe166acc39beee8ce36.


@Eric Snow: Please don't fix the bug, please explain how to fix it ;-)

> At the end, you should get the commit 6b4be195cd8868b76eb6fbe166acc39beee8ce36.

The commit is a giant change. So let me help you, the following change is strange. value is replaced whereas its value is non-NULL... Maybe it's the regression? ;-)

diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index 90f8551..03601ea 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -291,6 +291,9 @@ import_init(PyInterpreterState *interp, PyObject *sysmod)
 
     /* Install importlib as the implementation of import */
     value = PyObject_CallMethod(importlib, "_install", "OO", sysmod, impmod);
+    if (value != NULL)
+        value = PyObject_CallMethod(importlib,
+                                    "_install_external_importers", "");
     if (value == NULL) {
         PyErr_Print();
         Py_FatalError("Py_Initialize: importlib install failed");


Stéphane Wirtel (matrixise): "this issue can be executed on Linux, I think I am going to work on this one."

Would you like to work on a patch?

SubinterpreterTest.test_subinterps of test_capi also leaks. But it is likely the same bug than this issue (SubinterpreterTest.test_callbacks_leak() of test_atexit leaks).

Oh, 1abcf6700b4da6207fe859de40c6c1bada6b4fec introduced two more reference leaks.

I have pushed a PR, if you can check it. Thanks

-    SET_SYS_FROM_STRING_BORROW_INT_RESULT("warnoptions", warnoptions); 

Oh, it seems like the regression was introduced by me in the commit 8fea252a507024edf00d5d98881d22dc8799a8d3, see:
http://bugs.python.org/issue18520#msg201472

Or maybe it comes from recent changes made by Eric Snow and Nick Coghlan related to Python initiallization (PEP 432). I don't know.

bpo-30536 has been marked as a duplicate of this issue: [EASY] SubinterpThreadingTests.test_threads_join_2() of test_threading leaks references.

Sorry, that issue wasn't [EASY] at all! But I tried to help Stéphane & Louie as much as I could ;-)

I created bpo-30598: Py_NewInterpreter() leaks a reference on warnoptions in _PySys_EndInit().

New changeset ab1cb80b435a34e4f908c97cd2f3a7fe8add6505 by Victor Stinner (Stéphane Wirtel) in branch 'master':
bpo-30547: Fix multiple reference leaks (#1995)
https://github.com/python/cpython/commit/ab1cb80b435a34e4f908c97cd2f3a7fe8add6505

3.6 is not affected by this issue.

Thanks for taking care of this, Victor, Stéphane & Louie!

Eric Snow: "Thanks for taking care of this, Victor, Stéphane & Louie!"

You're welcome. FYI I'm trying to look more closely at the new Refleaks buildbot, so now it should become easier to catch regressions.

By the way, I like the work you did with Nick. Even if I don't understand everything, I know that we have to do it :-)