This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: valgrind reports leaks for test_zipimport
Type: Stage: resolved
Components: Tests Versions: Python 3.4
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Convert static types to heap types: use PyType_FromSpec()
View: 40077
Assigned To: Nosy List: rkuska, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2015-03-24 22:32 by rkuska, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
leak.py rkuska, 2015-03-24 22:32 not so minimal reproducer
report rkuska, 2015-03-24 22:32 report from valgrind
leak2.py vstinner, 2015-03-24 23:29
leak3.py rkuska, 2015-03-25 09:15
leak4.py vstinner, 2022-01-28 02:41
Messages (6)
msg239193 - (view) Author: Robert Kuska (rkuska) * Date: 2015-03-24 22:32
Leaks happen only when both testDoctestFile and testDoctestSuite are run.
Run with Python 3.4.2 and 3.4.1 with same result.

I have extracted those two tests into `leak.py` (attached).

> $ valgrind --suppressions=/../cpython/Misc/valgrind-python.supp python3 leak.py                                                                                            
==17896== Memcheck, a memory error detector
==17896== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==17896== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==17896== Command: python3 leak.py
==17896== 
==17896== 
==17896== HEAP SUMMARY:
==17896==     in use at exit: 1,599,328 bytes in 11,595 blocks
==17896==   total heap usage: 283,757 allocs, 272,162 frees, 37,891,147 bytes allocated
==17896== 
==17896== LEAK SUMMARY:
==17896==    definitely lost: 30 bytes in 1 blocks
==17896==    indirectly lost: 0 bytes in 0 blocks
==17896==      possibly lost: 597,418 bytes in 2,319 blocks
==17896==    still reachable: 1,001,880 bytes in 9,275 blocks
==17896==         suppressed: 0 bytes in 0 blocks
==17896== Rerun with --leak-check=full to see details of leaked memory
==17896== 
==17896== For counts of detected and suppressed errors, rerun with: -v
==17896== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Note that when I remove support.modules_cleanup(*modules_before) from leak.py valgrind reports no leaks (in original test_zipimport those are run in setUp and tearDown).

Output of  
valgrind --suppressions=/home/rkuska/upstream/cpython/Misc/valgrind-python.supp --leak-check=yes -v python3 leak.py
attached as `report`.
msg239194 - (view) Author: Robert Kuska (rkuska) * Date: 2015-03-24 22:36
Summary for
valgrind python3 test_zipimport.py

==18608== 
==18608== HEAP SUMMARY:
==18608==     in use at exit: 1,596,390 bytes in 11,536 blocks
==18608==   total heap usage: 343,849 allocs, 332,313 frees, 59,355,776 bytes allocated
==18608== 
==18608== LEAK SUMMARY:
==18608==    definitely lost: 90 bytes in 3 blocks
==18608==    indirectly lost: 0 bytes in 0 blocks
==18608==      possibly lost: 594,488 bytes in 2,258 blocks
==18608==    still reachable: 1,001,812 bytes in 9,275 blocks
==18608==         suppressed: 0 bytes in 0 blocks
==18608== Rerun with --leak-check=full to see details of leaked memory
==18608== 
==18608== For counts of detected and suppressed errors, rerun with: -v
==18608== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
msg239196 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-24 23:29
When calling gc.collect() after each test, I don't see any leak anymore.

doTest() has a small leak: it prepends a path to sys.path, but it never removes it.

Try attached leak2.py: it displays something like +254 kB. Uncomment the two following lines and the output will be close to +0 kB.

        #sys.path = old_path
    #gc.collect()
msg239238 - (view) Author: Robert Kuska (rkuska) * Date: 2015-03-25 09:15
I tried leak2.py with valgrind, I've uncommented the lines you mentioned.

> $ valgrind python3 leak2.py                                                                                                                                                                   
==28421== Memcheck, a memory error detector
==28421== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==28421== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==28421== Command: python3 leak2.py
==28421== 
==28421== 
==28421== HEAP SUMMARY:
==28421==     in use at exit: 1,599,354 bytes in 11,594 blocks
==28421==   total heap usage: 284,971 allocs, 273,377 frees, 37,976,898 bytes allocated
==28421== 
==28421== LEAK SUMMARY:
==28421==    definitely lost: 0 bytes in 0 blocks
==28421==    indirectly lost: 0 bytes in 0 blocks
==28421==      possibly lost: 597,482 bytes in 2,319 blocks
==28421==    still reachable: 1,001,872 bytes in 9,275 blocks
==28421==         suppressed: 0 bytes in 0 blocks
==28421== Rerun with --leak-check=full to see details of leaked memory
==28421== 
==28421== For counts of detected and suppressed errors, rerun with: -v
==28421== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

I've removed `import readline` from leak2.py, everything else left as in previous run (also ofc removed tracemalloc).

> $ valgrind python3 leak3.py                                                                                                                                                                  
==28515== Memcheck, a memory error detector
==28515== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==28515== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==28515== Command: python3 leak2.py
==28515== 
==28515== 
==28515== HEAP SUMMARY:
==28515==     in use at exit: 1,599,384 bytes in 11,595 blocks
==28515==   total heap usage: 285,648 allocs, 274,053 frees, 38,126,379 bytes allocated
==28515== 
==28515== LEAK SUMMARY:
==28515==    definitely lost: 30 bytes in 1 blocks
==28515==    indirectly lost: 0 bytes in 0 blocks
==28515==      possibly lost: 597,375 bytes in 2,317 blocks
==28515==    still reachable: 1,001,979 bytes in 9,277 blocks
==28515==         suppressed: 0 bytes in 0 blocks
==28515== Rerun with --leak-check=full to see details of leaked memory
==28515== 
==28515== For counts of detected and suppressed errors, rerun with: -v
==28515== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)


Duh? Why does readline have this effect?
msg325731 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-19 07:55
zipimport has been rewritten in pure Python (issue25711).
msg411963 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-01-28 02:41
> valgrind reports leaks for test_zipimport

It's not strictly a memory leak: running the same code multiple time doesn't leak memory. It's just that Python allocates memory once and never releases it (doesn't releast it at exit).

---

leak4.py: Updated and simplified script.

If the code is run 10 times, it doesn't leak. With 1, 10 or 100 loops (LOOPS=100), Python still says the same at exit:

$ ./python -I -X showrefcount leak4.py 
[1574 refs, 664 blocks]

Using cannot_deallocate2.patch of bpo-46417, I can see that multiple static types are not cleared at Python exit:
---
Cannot deallocate type 'Exception': it still has subclasses
* error
Cannot deallocate type 'BaseException': it still has subclasses
* Exception
Cannot deallocate type 'dict': it still has subclasses
* collections.defaultdict
Cannot deallocate type 'object': it still has subclasses
* dict
* BaseException
* itertools.accumulate
* itertools.combinations
* itertools.combinations_with_replacement
* itertools.cycle
* itertools.dropwhile
* itertools.takewhile
* itertools.islice
* itertools.starmap
* itertools.chain
* itertools.compress
* itertools.filterfalse
* itertools.count
* itertools.zip_longest
* itertools.pairwise
* itertools.permutations
* itertools.product
* itertools.repeat
* itertools.groupby
* itertools._grouper
* itertools._tee
* itertools._tee_dataobject
* collections.deque
* _collections._deque_iterator
* _collections._deque_reverse_iterator
* _collections._tuplegetter
* _struct.Struct
* _struct.unpack_iterator
---

The _collections, itertools and _struct extensions implement types as static types which are not cleared at Python exit. It explains why Python says that there are still 664 memory blocks allocated at Python exit (664 blocks).

I consider that this issue is a duplicate of bpo-40077.
History
Date User Action Args
2022-04-11 14:58:14adminsetgithub: 67957
2022-01-28 02:41:02vstinnersetstatus: open -> closed
files: + leak4.py
resolution: duplicate
messages: + msg411963

superseder: Convert static types to heap types: use PyType_FromSpec()
stage: resolved
2018-09-19 07:55:23serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg325731
2015-03-25 09:15:11rkuskasetfiles: + leak3.py

messages: + msg239238
2015-03-24 23:29:32vstinnersetfiles: + leak2.py
nosy: + vstinner
messages: + msg239196

2015-03-24 22:36:12rkuskasetmessages: + msg239194
2015-03-24 22:32:45rkuskasetfiles: + report
2015-03-24 22:32:14rkuskacreate