This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Memory leak in dict with shared keys
Type: resource usage Stage: resolved
Components: Versions: Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x
View: 31217
Assigned To: Nosy List: methane, pitrou, rhettinger, serhiy.storchaka, vstinner, xiang.zhang
Priority: normal Keywords:

Created on 2017-08-31 20:18 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
test_leak.py vstinner, 2017-08-31 20:18
Messages (8)
msg301058 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-31 20:18
Copy attached test_leak.py to Lib/test/ and run:

haypo@selma$ ./python -m test -R 3:30 test_leak

Run tests in parallel using 1 child processes
0:00:00 load avg: 0.67 [1/1/1] test_leak failed
beginning 33 repetitions
123456789012345678901234567890123
.................................
test_leak leaked [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] memory blocks, sum=30

1 test failed:
    test_leak

Total duration: 682 ms
Tests result: FAILURE


It seems like Thread2 constructor creates a dictionary on "self.hashvalue = (...)". It's a dictionary sharing keys, but it seems like the dict table memory is not released whne the bug occurs.


Sadly, I'm only able to reproduce the bug on a 32-bit build. Since the bug depends on exact hash values, maybe the bug exists in 64-bit builds, but is less likely?


I'm not 100% sure if the bug is related to dict or set.


The bug comes from bpo-31217.
msg301060 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-31 20:22
If you cannot reproduce the bug on 32-bit build, try to set "FIXED_HASH = 0" in test_leak.py, run "./python -m test -R 3:10 test_leak" multiple times until it fails, then copy the last value of the printed list into hash_values in test_leak.py.
msg301062 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-31 21:25
I spent a lot of time in gdb, but I'm not sure anymore that the issue is related to dict shared keys.
msg301063 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-31 22:03
It's a really weird bug you found.  It depends on the exact values passed to -R:

$ ./python -m test -R 3:6 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.39 [1/1] test_haypoleak
beginning 9 repetitions
123456789
.........
test_haypoleak leaked [1, 1, 1, 1, 1, 1] memory blocks, sum=6
test_haypoleak failed

$ ./python -m test -R 2:6 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.36 [1/1] test_haypoleak
beginning 8 repetitions
12345678
........
1 test OK.
msg301064 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-31 22:04
Actually, it's not the bug that is weird, but the heuristics in -R seem completely weird now.  If I print the alloc and rc deltas:

$ ./python -m test -R 3:6 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.09 [1/1] test_haypoleak
beginning 9 repetitions
123456789
.........
alloc: [0, 0, 0, 1, 1, 1, 1, 1, 1], rc: [0, 0, 0, 0, 0, 0, 0, 0, 0]
test_haypoleak leaked [1, 1, 1, 1, 1, 1] memory blocks, sum=6
test_haypoleak failed

$ ./python -m test -R 2:6 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.10 [1/1] test_haypoleak
beginning 8 repetitions
12345678
........
alloc: [0, 0, 0, 1, 1, 1, 1, 1], rc: [0, 0, 0, 0, 0, 0, 0, 0]
1 test OK.
msg301065 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-31 22:05
s/completely weird/broken/

$ ./python -m test -R 2:20 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.10 [1/1] test_haypoleak
beginning 22 repetitions
1234567890123456789012
......................
alloc: [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], rc: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
1 test OK.
msg301067 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-31 22:11
To come back to this bug, it still fails if add a __slots__, so I don't think it's related to shared key dictionaries.
msg301068 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-31 22:23
Antoine: "Actually, it's not the bug that is weird, but the heuristics in -R seem completely weird now.  If I print the alloc and rc deltas:"

I changed it recently to reduce false alarms: see bpo-30776.


Antoine: "To come back to this bug, it still fails if add a __slots__, so I don't think it's related to shared key dictionaries."

Oh ok, in that case, I will move back to bpo-31217. I close this issue as a duplicate.
History
Date User Action Args
2022-04-11 14:58:51adminsetgithub: 75498
2017-08-31 22:23:00vstinnersetstatus: open -> closed
superseder: test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x
messages: + msg301068

resolution: duplicate
stage: resolved
2017-08-31 22:11:31pitrousetmessages: + msg301067
2017-08-31 22:05:23pitrousetmessages: + msg301065
2017-08-31 22:04:23pitrousetmessages: + msg301064
2017-08-31 22:03:22pitrousetnosy: + pitrou
messages: + msg301063
2017-08-31 21:25:15vstinnersetmessages: + msg301062
2017-08-31 20:22:38vstinnersetmessages: + msg301060
2017-08-31 20:19:31vstinnersetnosy: + serhiy.storchaka
2017-08-31 20:18:39vstinnersetfiles: + test_leak.py
2017-08-31 20:18:11vstinnercreate