valgrind reports leaks for test_zipimport #67957

rkuska · 2015-03-24T22:32:14Z

BPO	23769
Nosy	@vstinner, @serhiy-storchaka, @rkuska
Superseder	bpo-40077: Convert static types to heap types: use PyType_FromSpec()
Files	leak.py: not so minimal reproducer report: report from valgrind leak2.py leak3.py leak4.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-28.02:41:02.724>
created_at = <Date 2015-03-24.22:32:14.240>
labels = ['tests']
title = 'valgrind reports leaks for test_zipimport'
updated_at = <Date 2022-01-28.02:41:02.723>
user = 'https://github.com/rkuska'

bugs.python.org fields:

activity = <Date 2022-01-28.02:41:02.723>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-28.02:41:02.724>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2015-03-24.22:32:14.240>
creator = 'rkuska'
dependencies = []
files = ['38672', '38673', '38675', '38684', '50594']
hgrepos = []
issue_num = 23769
keywords = []
message_count = 6.0
messages = ['239193', '239194', '239196', '239238', '325731', '411963']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'rkuska']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '40077'
type = None
url = 'https://bugs.python.org/issue23769'
versions = ['Python 3.4']

rkuska · 2015-03-24T22:32:13Z

Leaks happen only when both testDoctestFile and testDoctestSuite are run.
Run with Python 3.4.2 and 3.4.1 with same result.

I have extracted those two tests into leak.py (attached).

$ valgrind --suppressions=/../cpython/Misc/valgrind-python.supp python3 leak.py
==17896== Memcheck, a memory error detector
==17896== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==17896== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==17896== Command: python3 leak.py
==17896==
==17896==
==17896== HEAP SUMMARY:
==17896== in use at exit: 1,599,328 bytes in 11,595 blocks
==17896== total heap usage: 283,757 allocs, 272,162 frees, 37,891,147 bytes allocated
==17896==
==17896== LEAK SUMMARY:
==17896== definitely lost: 30 bytes in 1 blocks
==17896== indirectly lost: 0 bytes in 0 blocks
==17896== possibly lost: 597,418 bytes in 2,319 blocks
==17896== still reachable: 1,001,880 bytes in 9,275 blocks
==17896== suppressed: 0 bytes in 0 blocks
==17896== Rerun with --leak-check=full to see details of leaked memory
==17896==
==17896== For counts of detected and suppressed errors, rerun with: -v
==17896== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Note that when I remove support.modules_cleanup(*modules_before) from leak.py valgrind reports no leaks (in original test_zipimport those are run in setUp and tearDown).

Output of
valgrind --suppressions=/home/rkuska/upstream/cpython/Misc/valgrind-python.supp --leak-check=yes -v python3 leak.py
attached as report.

rkuska · 2015-03-24T22:36:12Z

Summary for
valgrind python3 test_zipimport.py

==18608==
==18608== HEAP SUMMARY:
==18608== in use at exit: 1,596,390 bytes in 11,536 blocks
==18608== total heap usage: 343,849 allocs, 332,313 frees, 59,355,776 bytes allocated
==18608==
==18608== LEAK SUMMARY:
==18608== definitely lost: 90 bytes in 3 blocks
==18608== indirectly lost: 0 bytes in 0 blocks
==18608== possibly lost: 594,488 bytes in 2,258 blocks
==18608== still reachable: 1,001,812 bytes in 9,275 blocks
==18608== suppressed: 0 bytes in 0 blocks
==18608== Rerun with --leak-check=full to see details of leaked memory
==18608==
==18608== For counts of detected and suppressed errors, rerun with: -v
==18608== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

vstinner · 2015-03-24T23:29:32Z

When calling gc.collect() after each test, I don't see any leak anymore.

doTest() has a small leak: it prepends a path to sys.path, but it never removes it.

Try attached leak2.py: it displays something like +254 kB. Uncomment the two following lines and the output will be close to +0 kB.

    #sys.path = old_path
#gc.collect()

rkuska · 2015-03-25T09:15:11Z

I tried leak2.py with valgrind, I've uncommented the lines you mentioned.

$ valgrind python3 leak2.py
==28421== Memcheck, a memory error detector
==28421== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==28421== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==28421== Command: python3 leak2.py
==28421==
==28421==
==28421== HEAP SUMMARY:
==28421== in use at exit: 1,599,354 bytes in 11,594 blocks
==28421== total heap usage: 284,971 allocs, 273,377 frees, 37,976,898 bytes allocated
==28421==
==28421== LEAK SUMMARY:
==28421== definitely lost: 0 bytes in 0 blocks
==28421== indirectly lost: 0 bytes in 0 blocks
==28421== possibly lost: 597,482 bytes in 2,319 blocks
==28421== still reachable: 1,001,872 bytes in 9,275 blocks
==28421== suppressed: 0 bytes in 0 blocks
==28421== Rerun with --leak-check=full to see details of leaked memory
==28421==
==28421== For counts of detected and suppressed errors, rerun with: -v
==28421== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

I've removed import readline from leak2.py, everything else left as in previous run (also ofc removed tracemalloc).

$ valgrind python3 leak3.py
==28515== Memcheck, a memory error detector
==28515== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==28515== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==28515== Command: python3 leak2.py
==28515==
==28515==
==28515== HEAP SUMMARY:
==28515== in use at exit: 1,599,384 bytes in 11,595 blocks
==28515== total heap usage: 285,648 allocs, 274,053 frees, 38,126,379 bytes allocated
==28515==
==28515== LEAK SUMMARY:
==28515== definitely lost: 30 bytes in 1 blocks
==28515== indirectly lost: 0 bytes in 0 blocks
==28515== possibly lost: 597,375 bytes in 2,317 blocks
==28515== still reachable: 1,001,979 bytes in 9,277 blocks
==28515== suppressed: 0 bytes in 0 blocks
==28515== Rerun with --leak-check=full to see details of leaked memory
==28515==
==28515== For counts of detected and suppressed errors, rerun with: -v
==28515== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Duh? Why does readline have this effect?

serhiy-storchaka · 2018-09-19T07:55:23Z

zipimport has been rewritten in pure Python (bpo-25711).

vstinner · 2022-01-28T02:41:02Z

valgrind reports leaks for test_zipimport

It's not strictly a memory leak: running the same code multiple time doesn't leak memory. It's just that Python allocates memory once and never releases it (doesn't releast it at exit).

---

leak4.py: Updated and simplified script.

If the code is run 10 times, it doesn't leak. With 1, 10 or 100 loops (LOOPS=100), Python still says the same at exit:

$ ./python -I -X showrefcount leak4.py 
[1574 refs, 664 blocks]

Using cannot_deallocate2.patch of bpo-46417, I can see that multiple static types are not cleared at Python exit:
---
Cannot deallocate type 'Exception': it still has subclasses

error
Cannot deallocate type 'BaseException': it still has subclasses
Exception
Cannot deallocate type 'dict': it still has subclasses
collections.defaultdict
Cannot deallocate type 'object': it still has subclasses
dict
BaseException
itertools.accumulate
itertools.combinations
itertools.combinations_with_replacement
itertools.cycle
itertools.dropwhile
itertools.takewhile
itertools.islice
itertools.starmap
itertools.chain
itertools.compress
itertools.filterfalse
itertools.count
itertools.zip_longest
itertools.pairwise
itertools.permutations
itertools.product
itertools.repeat
itertools.groupby
itertools._grouper
itertools._tee
itertools._tee_dataobject
collections.deque
_collections._deque_iterator
_collections._deque_reverse_iterator
_collections._tuplegetter
_struct.Struct
_struct.unpack_iterator
---

The _collections, itertools and _struct extensions implement types as static types which are not cleared at Python exit. It explains why Python says that there are still 664 memory blocks allocated at Python exit (664 blocks).

I consider that this issue is a duplicate of bpo-40077.

rkuska mannequin added the tests Tests in the Lib/test dir label Mar 24, 2015

vstinner closed this as completed Jan 28, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

valgrind reports leaks for test_zipimport #67957

valgrind reports leaks for test_zipimport #67957

rkuska mannequin commented Mar 24, 2015

rkuska mannequin commented Mar 24, 2015

rkuska mannequin commented Mar 24, 2015

vstinner commented Mar 24, 2015

rkuska mannequin commented Mar 25, 2015

serhiy-storchaka commented Sep 19, 2018

vstinner commented Jan 28, 2022

valgrind reports leaks for test_zipimport #67957

valgrind reports leaks for test_zipimport #67957

Comments

rkuska mannequin commented Mar 24, 2015

rkuska mannequin commented Mar 24, 2015

rkuska mannequin commented Mar 24, 2015

vstinner commented Mar 24, 2015

rkuska mannequin commented Mar 25, 2015

serhiy-storchaka commented Sep 19, 2018

vstinner commented Jan 28, 2022