Currently, regrtest considers that a test file leaks if at least one run increased or decreased the global reference counter. The problem is that in practice, Python is full of singletons, caches, and other jokes of the garbage collector, etc.

To reduce false alarms and focus more on real bugs, I suggest to change the reference difference checker from:

     def check_rc_deltas(deltas):
        return any(deltas)

to:

     def check_rc_deltas(deltas):
        return all(delta>=1 for delta in deltas)

It would allow to ignore false positives like:

* [3, 0, 0]
* [0, 1, 0]
* [8, -8, 1]

I had to write this change to be able to bisect bpo-30775 reference leak. Without this change, my bisection tool picked the wrong tests and failed to find the tests which leaked. For example, it took a subtest of tests which was considered as leaking because the reference differences was [3, 0, 0]. This pattern more looks like a late initialization than a real leak. In the case of the bpo-30775, a leak was closer to [222, 222, 225]: much more references than just 3 :-) Many leaks for all runs :-)

Real-world examples of false positives from the x86 Gentoo Refleaks 2.7 buildbot:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%202.7/builds/32/steps/test/logs/stdio

test_nntplib leaked [0, 85, 0] references, sum=85
test_multiprocessing leaked [0, 0, -35] references, sum=-35

New changeset 48b5c422ffb03affb00c184b9a99e5537be92732 by Victor Stinner in branch 'master':
bpo-30776: reduce regrtest -R false positives (#2422)
https://github.com/python/cpython/commit/48b5c422ffb03affb00c184b9a99e5537be92732

New changeset 35d2ca2b94a6ff29e763ddb7727166f0592edfa2 by Victor Stinner in branch '3.6':
[3.6] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2441)
https://github.com/python/cpython/commit/35d2ca2b94a6ff29e763ddb7727166f0592edfa2

New changeset de1850bb03f8225cbff85f437b6e972bf9b68c2a by Victor Stinner in branch '3.5':
[3.5] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2442)
https://github.com/python/cpython/commit/de1850bb03f8225cbff85f437b6e972bf9b68c2a

New changeset fea98bfcff6ccf9142daa97677fe86c1fdf8e63e by Victor Stinner in branch '2.7':
[2.7] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2444)
https://github.com/python/cpython/commit/fea98bfcff6ccf9142daa97677fe86c1fdf8e63e

It seems like the change works well since I didn't see any more false alarms on the Refleaks buildbots. I close the issue. The change has been applied to 2.7, 3.5, 3.6 and master branches.

New changeset beeca6e1e5fd01531b1db7059498b13d07dca525 by Victor Stinner in branch 'master':
bpo-30776: regrtest: reduce memleak false positive (#2484)
https://github.com/python/cpython/commit/beeca6e1e5fd01531b1db7059498b13d07dca525

New changeset a3ca94d0504157a112a1f89bfe8be1307116fc73 by Victor Stinner in branch '3.6':
[3.6] bpo-29512, bpo-30776: Backport regrtest enhancements from master to 3.6 (#2513)
https://github.com/python/cpython/commit/a3ca94d0504157a112a1f89bfe8be1307116fc73

New changeset 0b1210739d12c26e4a161ffd494bd572d49b2483 by Victor Stinner in branch '3.5':
[3.5] bpo-29512, bpo-30764, bpo-30776: Backport regrtest enhancements from 3.6 to 3.5 (#2540)
https://github.com/python/cpython/commit/0b1210739d12c26e4a161ffd494bd572d49b2483

This change suddenly made detection of reference leaks a lot less effective.  Now if you have a single 0 in the deltas list, it will consider there is no reference leak, even though all other values might be > 0.

See https://bugs.python.org/issue31317#msg301064 for an example.

Victor, it would be nice if you could find a less brutal heuristic.

Antoine Pitrou: "Victor, it would be nice if you could find a less brutal heuristic."

My current goal is to get green buildbots. If a buildbot is always red, it's hard to notify easily regressions.

Since I rejected "false alarms" from refleak, the refleak buildbot already helped to identify a reference leak regression. I am talking about the Windows Refleak buildbot:

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.x

I'm now working on the x86 Gentoo Refleaks 3.x which raised strange issues like bpo-31217.

I am already making regrtest stricter, but slowly, without breaking buildbots. It's a slow process. That's why I'm to "fix" dangling threads for example. By the way, I'm not done with dangling threads in master yet.

I'm happy if anyone is interested to help me to work on buildbots, which means in practice fixing a new different bug everyday :-) Slowly, the remaining bugs are the most complex ones.

> Since I rejected "false alarms" from refleak, the refleak buildbot already helped to identify a reference leak regression.

Except that now it may hide actual regressions... False negatives are not better than false positives.

> Except that now it may hide actual regressions... False negatives are not better than false positives.

Please put this issue in its context. 6 months ago, there was no more Refleak buildbot and everything was broken.

If you want to make regrtest -R stricer, go ahead. But if a buildbot breaks, you will have to fix it.

I would prefer to first fix the Gentoo Refleak buildbot, since most previous builds failed.

The commit 6c2feabc5dac2f3049b15134669e9ad5af573193 of bpo-31217 fixed a major random bug in the memory block check. So maybe we can start again to look to detect more bugs by replacing all() with any() again.