classification
Title: Refleak tests: test_doctest and test_gc are failing
Type: Stage:
Components: Interpreter Core, Tests, Windows Versions: Python 3.0
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: tim_one Nosy List: amaury.forgeotdarc, christian.heimes, gvanrossum, tim_one (4)
Priority: normal Keywords

Created on 2007-12-02 14:41 by christian.heimes, last changed 2008-01-06 22:29 by admin.

Files
File name Uploaded Description Edit Remove
gc_bug.py amaury.forgeotdarc, 2007-12-05 16:51
Messages (4)
msg58089 - (view) Author: Christian Heimes (christian.heimes) Date: 2007-12-02 14:41
I've seen the problem on Windows only. test_doctest fails and the
problem also causes test_gc to fail when it is run after test_doctest.
W/o a prior run of test_doctest test_gc doesn't fail.

File "c:\dev\python\py3k\lib\test\test_doctest.py", line 1570, in
test.test_doct
est.test_debug
Failed example:
    try: doctest.debug_src(s)
    finally: sys.stdin = real_stdin
Expected:
    > <string>(1)<module>()
    (Pdb) next
    12
    --Return--
    > <string>(1)<module>()->None
    (Pdb) print(x)
    12
    (Pdb) continue
Got:
    > c:\dev\python\py3k\lib\io.py(281)__del__()
    -> try:
    (Pdb) next
    > c:\dev\python\py3k\lib\io.py(282)__del__()
    -> self.close()
    (Pdb) print(x)
    *** NameError: NameError("name 'x' is not defined",)
    (Pdb) continue
    12
**********************************************************************
1 items had failures:
   1 of   4 in test.test_doctest.test_debug
***Test Failed*** 1 failures.
test test_doctest failed -- 1 of 418 doctests failed
test_gc
test test_gc failed -- Traceback (most recent call last):
  File "c:\dev\python\py3k\lib\test\test_gc.py", line 193, in test_saveall
    self.assertEqual(gc.garbage, [])
AssertionError: [<io.BytesIO object at 0x01237968>] != []

2 tests failed:
    test_doctest test_gc
msg58205 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) Date: 2007-12-05 00:34
After some hard debugging:
- doctest.debug_src() is unlucky enough to trigger a garbage collection
just when compiling the given code.
- gc collects unreachable objects, among them is an instance of the
class doctest._SpoofOut, which derives from io.StringIO.
- The debugger steps into io.IOBase.__del__

Some possible directions:
- Change the gc thresholds. A very temporary workaround to make the test
pass.
- Find the cycle involving the SpoofOut object, and try to break it in
doctest.py.
- Find a way to disable pdb tracing when the gc is running finalizers.
(this is what happens in 2.5: pdb does not step into a C function)
- Forget everything, and wait for the io.py object to be rewritten in C.
msg58218 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) Date: 2007-12-05 16:51
Finally I found a potential problem with the garbage collector in a
specific case:
- Some object C participates in a reference cycle, and contains a
reference to another object X.
- X.__del__ 'resurrect' the object, by saving its 'self' somewhere else.
- X contains a reference to Y, which has a __del__ method.
When collecting all this, gc.garbage == [Y] !
This is not true garbage: if you clean gc.garbage, then the next
gc.collect() clears everything.


Now, try to follow my explanation (if I correctly understand the gc
internals):
- When the cycle is garbage collected, X and Y are detected as
'unreachable with finalizers', and put in a specific 'finalizers' list.
- the cycle is broken, C is deallocated. 
- This correctly triggers X.__del__. X is removed from the 'finalizers'
list.
- when X is resurrected, it comes back to the normal gc tracking list.
- At the end, 'finalizers' contains 3 objects: X.__dict__, Y and Y.__dict__.
- Y is then considered as garbage.

I join a script which reproduces the behaviour. Note that 2.4 and 2.5
are affected too.
In py3k, the 'resurrect' seems to be caused by a (caught) exception in
TextIOWrapper.close(): the exception contains a reference to the frame,
which references the self variable.
msg58221 - (view) Author: Guido van Rossum (gvanrossum) Date: 2007-12-05 17:25
Hoping to draw Tim into this... He's the only one I know who truly
understands these issues...
History
Date User Action Args
2008-01-06 22:29:44adminsetkeywords: - py3k
versions: Python 3.0
2007-12-05 17:25:27gvanrossumsetassignee: tim_one
messages: + msg58221
nosy: + tim_one
2007-12-05 16:51:42amaury.forgeotdarcsetfiles: + gc_bug.py
nosy: + gvanrossum
messages: + msg58218
components: + Interpreter Core
2007-12-05 00:34:04amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg58205
2007-12-02 14:41:56christian.heimescreate