Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XincludeTest failure in test_xml_etree #59280

Closed
elibendersky mannequin opened this issue Jun 15, 2012 · 10 comments
Closed

XincludeTest failure in test_xml_etree #59280

elibendersky mannequin opened this issue Jun 15, 2012 · 10 comments
Labels
tests Tests in the Lib/test dir type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@elibendersky
Copy link
Mannequin

elibendersky mannequin commented Jun 15, 2012

BPO 15075
Nosy @ncoghlan, @scoder, @florentx
Superseder
  • bpo-15083: Rewrite ElementTree tests in a cleaner and safer way
    		•
    Files
  • issue15075.1.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2012-06-16.03:49:33.527>
created_at = <Date 2012-06-15.08:18:05.201>
labels = ['tests', 'type-crash']
title = 'XincludeTest failure in test_xml_etree'
updated_at = <Date 2013-01-05.15:01:11.597>
user = 'https://bugs.python.org/elibendersky'

    bugs.python.org fields:

    activity = <Date 2013-01-05.15:01:11.597>
actor = 'eli.bendersky'
assignee = 'none'
closed = True
closed_date = <Date 2012-06-16.03:49:33.527>
closer = 'python-dev'
components = ['Tests']
creation = <Date 2012-06-15.08:18:05.201>
creator = 'eli.bendersky'
dependencies = []
files = ['26020']
hgrepos = []
issue_num = 15075
keywords = ['patch']
message_count = 10.0
messages = ['162870', '162881', '162884', '162925', '162951', '162953', '162957', '162967', '178654', '179124']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'scoder', 'eli.bendersky', 'flox', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '15083'
type = 'crash'
url = 'https://bugs.python.org/issue15075'
versions = ['Python 3.3']
    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Jun 15, 2012

    The XincludeTest test-case in test_xml_etree is now skipped, because it fails in an intermittent manner. I can reproduce the failure when running full regrtest with -j1, but not -j8, and not when run individually.

    The failure is most likely due to the test itself, in the way it imports the Python ET, leaving _elementtree out.

    This may be related to http://bugs.python.org/issue14035

    @elibendersky elibendersky mannequin added tests Tests in the Lib/test dir type-crash A hard crash of the interpreter, possibly with a core dump labels Jun 15, 2012
    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Jun 15, 2012

    After some tinkering I found which test when run before test_xml_etree causes it to crash:

    $ ./python -m test.regrtest test___all__ test_xml_etree
[1/2] test___all__
[2/2] test_xml_etree
Fatal Python error: Segmentation fault

    Current thread 0x00007f771ecec700:
    File "/home/eliben/python-src/33/Lib/xml/etree/ElementTree.py", line 895 in _namespaces
    File "/home/eliben/python-src/33/Lib/xml/etree/ElementTree.py", line 843 in write
    File "/home/eliben/python-src/33/Lib/test/test_xml_etree.py", line 103 in serialize
    File "/home/eliben/python-src/33/Lib/test/test_xml_etree.py", line 2020 in test_xinclude_default
    File "/home/eliben/python-src/33/Lib/unittest/case.py", line 385 in _executeTestPart
    File "/home/eliben/python-src/33/Lib/unittest/case.py", line 440 in run
    File "/home/eliben/python-src/33/Lib/unittest/case.py", line 492 in __call__
    File "/home/eliben/python-src/33/Lib/unittest/suite.py", line 105 in run
    File "/home/eliben/python-src/33/Lib/unittest/suite.py", line 67 in __call__
    File "/home/eliben/python-src/33/Lib/unittest/suite.py", line 105 in run
    File "/home/eliben/python-src/33/Lib/unittest/suite.py", line 67 in __call__
    File "/home/eliben/python-src/33/Lib/test/support.py", line 1282 in run
    File "/home/eliben/python-src/33/Lib/test/support.py", line 1383 in _run_suite
    File "/home/eliben/python-src/33/Lib/test/support.py", line 1417 in run_unittest
    File "/home/eliben/python-src/33/Lib/test/test_xml_etree.py", line 2311 in test_main
    File "/home/eliben/python-src/33/Lib/test/regrtest.py", line 1238 in runtest_inner
    File "/home/eliben/python-src/33/Lib/test/regrtest.py", line 919 in runtest
    File "/home/eliben/python-src/33/Lib/test/regrtest.py", line 710 in main
    File "/home/eliben/python-src/33/Lib/test/regrtest.py", line 1829 in <module>
    File "/home/eliben/python-src/33/Lib/runpy.py", line 75 in _run_code
    File "/home/eliben/python-src/33/Lib/runpy.py", line 162 in _run_module_as_main
    Segmentation fault (core dumped)

    Note that this is the Python test crashing. test_xml_etree_c works fine

    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Jun 15, 2012

    As suspected, the cause is that xinclude_loader manages to somehow import the C version of ET, although test_xml_etree tries to enforce the Python version. This is probably because test___all__ imports all modules and leaves stuff in the import cache.

    What causes the segfault itself is that C Elements are mixed with Python Elements, and they're incompatible.

    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Jun 15, 2012

    Here is a patch that solves the current problem.

    A longer term solution would be to have a cleaner test plan for ET in general, without monkey-patching at all, and without state that causes test-order dependencies.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jun 16, 2012

    New changeset 5782efaa8d68 by Eli Bendersky in branch 'default':
    Make the test more resilient to test-run order (closes bpo-15075)
    http://hg.python.org/cpython/rev/5782efaa8d68

    @python-dev python-dev mannequin closed this as completed Jun 16, 2012
    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Jun 16, 2012

    Opened bpo-15083 to track the *actual* solution to this problem, which should restructure the tests to be safer.

    @florentx
    Copy link
    Mannequin

    florentx mannequin commented Jun 16, 2012

    File "/home/eliben/python-src/33/Lib/test/regrtest.py", line 710 in main
    File "/home/eliben/python-src/33/Lib/test/regrtest.py", line 1829 in <module>
    File "/home/eliben/python-src/33/Lib/runpy.py", line 75 in _run_code
    File "/home/eliben/python-src/33/Lib/runpy.py", line 162 in _run_module_as_main
    Segmentation fault (core dumped)

    Even if the ET tests are hacking sys.modules, I assume that the Python interpreter should not crash so badly.
    It reveals probably a flaw somewhere else.

    I suggest to re-open this to track the Segmentation fault.

    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Jun 16, 2012

    No need, the cause for the segfault is known. What happens is that Element objects from the Python module get intermixed with Element objects from the C module, and these are incompatible. The C module traversal functions assume they can cast Elements to a known structure (ElementObject), and when they get the invalid Elements they crash. Adding runtime checks everywhere is too costly. This situation cannot arise in a valid way.

    @scoder
    Copy link
    Contributor

    scoder commented Dec 31, 2012

    If runtime checks are needed to prevent mixing arbitrary objects into the tree, then I don't think they should be considered too costly.

    I agree with Florent that this is worth reopening. It doesn't look like a "Tests" bug to me rather a "Lib"/"XML" bug.

    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Jan 5, 2013

    I investigated a bit, and type checks were added to protect from such mixing. Please open a new patch with a speficic reproducer if you run into similar problems (interepreter crash).

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    tests Tests in the Lib/test dir type-crash A hard crash of the interpreter, possibly with a core dump
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @scoder