test_cleanup() of test_builtin failed #64798

vstinner · 2014-02-11T16:37:29Z

BPO	20599
Nosy	@pitrou, @vstinner, @serhiy-storchaka
Files	issue20599_print_flush.patch issue20599_ascii_ioencoding.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-02-13.10:51:35.219>
created_at = <Date 2014-02-11.16:37:29.288>
labels = ['type-bug', 'tests']
title = 'test_cleanup() of test_builtin failed'
updated_at = <Date 2014-03-17.06:30:56.830>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2014-03-17.06:30:56.830>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2014-02-13.10:51:35.219>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2014-02-11.16:37:29.288>
creator = 'vstinner'
dependencies = []
files = ['34057', '34058']
hgrepos = []
issue_num = 20599
keywords = ['patch']
message_count = 19.0
messages = ['210965', '210972', '211078', '211080', '211081', '211084', '211087', '211091', '211103', '211105', '211106', '211111', '211112', '211113', '211134', '211135', '211137', '211144', '213814']
nosy_count = 4.0
nosy_names = ['pitrou', 'vstinner', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20599'
versions = ['Python 3.4']

vstinner · 2014-02-11T16:37:29Z

http://buildbot.python.org/all/builders/x86%20XP-4%203.x/builds/10150/steps/test/logs/stdio

======================================================================
FAIL: test_cleanup (test.test_builtin.ShutdownTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_builtin.py", line 1621, in test_cleanup
    self.assertEqual(["before", "after"], out.decode().splitlines())
AssertionError: Lists differ: ['before', 'after'] != []

First list contains 2 additional elements.
First extra element 0:
before

['before', 'after']
+ []

vstinner · 2014-02-11T16:51:59Z

It's probably linked to recent changes from issue bpo-19255.

serhiy-storchaka · 2014-02-12T10:36:56Z

Unfortunately I can't reproduce this failure. We should look at how often this happens on buildbots. It is possible that adding flush=True in print() will solve the issue.

python-dev · 2014-02-12T10:40:50Z

New changeset 7ecee9e0dc58 by Serhiy Storchaka in branch 'default':
Try to fix test_cleanup (issue bpo-20599).
http://hg.python.org/cpython/rev/7ecee9e0dc58

serhiy-storchaka · 2014-02-12T10:53:43Z

Failed builtbots:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/4072/steps/test/logs/stdio
http://buildbot.python.org/all/builders/x86%20XP-4%203.x/builds/10154/steps/test/logs/stdio
http://buildbot.python.org/all/builders/x86%20Windows%20Server%202008%20%5BSB%5D%203.x/builds/2308/steps/test/logs/stdio

vstinner · 2014-02-12T11:24:42Z

It is possible that adding flush=True in print() will solve the issue.

I tried. It does not fix the issue.

Hint: On Windows, sys.stdout.encoding is a codec implemented in pure Python (ex: cp850). You should try to reproduce the issue on Linux using PYTHONIOENCODING=cp850.

Note: It looks like open() is no more available when C.__del__() is called.

serhiy-storchaka · 2014-02-12T12:31:38Z

Ah, thank you for your hint Victor. This patch should fix the issue.

vstinner · 2014-02-12T14:04:25Z

Sorry, issue20599_ascii_ioencoding.patch doesn't fix the issue. (I also tested issue20599_ascii_ioencoding.patch + flush=True for print, it still fails sometimes.)

serhiy-storchaka · 2014-02-12T17:15:34Z

What if use "-u" flag? "utf-8" or "utf_8" encoding?

python-dev · 2014-02-12T17:28:59Z

New changeset 10ea3125d7b8 by Victor Stinner in branch 'default':
Issue bpo-20599: Force ASCII encoding for stdout in test_cleanup() of test_builtin
http://hg.python.org/cpython/rev/10ea3125d7b8

vstinner · 2014-02-12T17:31:43Z

Sorry, issue20599_ascii_ioencoding.patch doesn't fix the issue.

Sorry again, I didn't read issue20599_ascii_ioencoding.patch carefully: it doesn't make sense to use the isolated mode and to set a Python environment variable (PYTHONIOENCODING). The purpose of the isolated mode is to ignore all Python environment variables...

I (hope I) fixed the test, but in fact, the test showed another bug: print() cannot be used during Python exit in destructor. It means for example that Python may not be able to display errors at exit.

I don't know if this issue should be closed. Maybe a new issue should be opened to address the codec issue at exit?

serhiy-storchaka · 2014-02-12T19:24:30Z

Now test fails with mystical error.

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/4074/steps/test/logs/stdio
http://buildbot.python.org/all/builders/x86%20Windows%20Server%202008%20%5BSB%5D%203.x/builds/2312/steps/test/logs/stdio

======================================================================
FAIL: test_cleanup (test.test_builtin.ShutdownTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_builtin.py", line 1628, in test_cleanup
    __cleanenv=True)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\script_helper.py", line 69, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\script_helper.py", line 55, in _assert_python
    "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is 2147483651, stderr follows:
Fatal Python error: Failed to initialize Windows random API (CryptoGen)

serhiy-storchaka · 2014-02-12T19:29:37Z

May be setting PYTHONHASHSEED will silence this error. But I afraid there is a real bug in initialization on Windows.

serhiy-storchaka · 2014-02-12T19:42:20Z

I (hope I) fixed the test, but in fact, the test showed another bug: print() cannot be used during Python exit in destructor. It means for example that Python may not be able to display errors at exit.

print() can be used during Python exit in destructor if destructor called before cleaning up the builtins and sys modules and encoding module with the encoding of sys.stdout.

Cleaning up the builtins and sys modules are delayed after wiping other modules, this decreases the chance of print() failure.

vstinner · 2014-02-13T09:51:10Z

Another failure:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/1724/steps/test/logs/stdio

======================================================================
FAIL: test_cleanup (test.test_builtin.ShutdownTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/test_builtin.py", line 1628, in test_cleanup
    __cleanenv=True)
  File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/script_helper.py", line 69, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/script_helper.py", line 55, in _assert_python
    "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is 1, stderr follows:
Shared object "libpython3.4dm.so.1" not found, required by "python"

python-dev · 2014-02-13T09:54:52Z

New changeset 488ccbee6ee6 by Victor Stinner in branch 'default':
Issue bpo-20599: Don't clear environment in test_cleanup() of test_builtin
http://hg.python.org/cpython/rev/488ccbee6ee6

vstinner · 2014-02-13T10:07:59Z

Shared object "libpython3.4dm.so.1" not found, required by "python"

This is probably is missing LD_LIBRARY_PATH environment variable.

Fatal Python error: Failed to initialize Windows random API (CryptoGen)

Oh, it remembers me that Windows requires some environment variables to start, otherwise it fails badly.

I modified the test to *not* start with an empty environment, it's not the purpose of the test.

vstinner · 2014-02-13T10:51:35Z

This is probably is missing LD_LIBRARY_PATH environment variable.
> Fatal Python error: Failed to initialize Windows random API (CryptoGen)
Oh, it remembers me that Windows requires some environment variables to start, otherwise it fails badly.

Ok, I filled issue bpo-20614 to track these bugs with __cleanenv=True.

Serhiy>print() can be used during Python exit in destructor if destructor called before cleaning up the builtins and sys modules and encoding module with the encoding of sys.stdout.
Serhiy>Cleaning up the builtins and sys modules are delayed after wiping other modules, this decreases the chance of print() failure.

Ah ok. I tested with a codec implemented in Python, it works well. So it is very specific to test_cleanup() which hack builtins and sys modules to keep a reference their.

I don't see anything else to do, I close the issue.

python-dev · 2014-03-17T06:30:56Z

New changeset 3d5154fa8413 by Serhiy Storchaka in branch '3.4':
Try to fix test_cleanup (issue bpo-20599).
http://hg.python.org/cpython/rev/3d5154fa8413

New changeset c978dffb95ac by Victor Stinner in branch '3.4':
Issue bpo-20599: Force ASCII encoding for stdout in test_cleanup() of test_builtin
http://hg.python.org/cpython/rev/c978dffb95ac

New changeset 1fb37d2d7d9d by Victor Stinner in branch '3.4':
Issue bpo-20599: Don't clear environment in test_cleanup() of test_builtin
http://hg.python.org/cpython/rev/1fb37d2d7d9d

serhiy-storchaka added tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Feb 12, 2014

vstinner closed this as completed Feb 13, 2014

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_cleanup() of test_builtin failed #64798

test_cleanup() of test_builtin failed #64798

vstinner commented Feb 11, 2014

vstinner commented Feb 11, 2014

vstinner commented Feb 11, 2014

serhiy-storchaka commented Feb 12, 2014

python-dev mannequin commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

vstinner commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

vstinner commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

python-dev mannequin commented Feb 12, 2014

vstinner commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

vstinner commented Feb 13, 2014

python-dev mannequin commented Feb 13, 2014

vstinner commented Feb 13, 2014

vstinner commented Feb 13, 2014

python-dev mannequin commented Mar 17, 2014

test_cleanup() of test_builtin failed #64798

test_cleanup() of test_builtin failed #64798

Comments

vstinner commented Feb 11, 2014

vstinner commented Feb 11, 2014

vstinner commented Feb 11, 2014

serhiy-storchaka commented Feb 12, 2014

python-dev mannequin commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

vstinner commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

vstinner commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

python-dev mannequin commented Feb 12, 2014

vstinner commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

serhiy-storchaka commented Feb 12, 2014

vstinner commented Feb 13, 2014

python-dev mannequin commented Feb 13, 2014

vstinner commented Feb 13, 2014

vstinner commented Feb 13, 2014

python-dev mannequin commented Mar 17, 2014