This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: XincludeTest failure in test_xml_etree
Type: crash Stage: resolved
Components: Tests Versions: Python 3.3
Status: closed Resolution: fixed
Dependencies: Superseder: Rewrite ElementTree tests in a cleaner and safer way
View: 15083
Assigned To: Nosy List: eli.bendersky, flox, ncoghlan, python-dev, scoder
Priority: normal Keywords: patch

Created on 2012-06-15 08:18 by eli.bendersky, last changed 2022-04-11 14:57 by admin. This issue is now closed.

File name Uploaded Description Edit
issue15075.1.patch eli.bendersky, 2012-06-15 17:57 review
Messages (10)
msg162870 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2012-06-15 08:18
The XincludeTest test-case in test_xml_etree is now skipped, because it fails in an intermittent manner. I can reproduce the failure when running full regrtest with -j1, but not -j8, and not when run individually.

The failure is most likely due to the test itself, in the way it imports the Python ET, leaving _elementtree out. 

This may be related to
msg162881 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2012-06-15 10:01
After some tinkering I found which test when run before test_xml_etree causes it to crash:

$ ./python -m test.regrtest test___all__ test_xml_etree
[1/2] test___all__
[2/2] test_xml_etree
Fatal Python error: Segmentation fault

Current thread 0x00007f771ecec700:
  File "/home/eliben/python-src/33/Lib/xml/etree/", line 895 in _namespaces
  File "/home/eliben/python-src/33/Lib/xml/etree/", line 843 in write
  File "/home/eliben/python-src/33/Lib/test/", line 103 in serialize
  File "/home/eliben/python-src/33/Lib/test/", line 2020 in test_xinclude_default
  File "/home/eliben/python-src/33/Lib/unittest/", line 385 in _executeTestPart
  File "/home/eliben/python-src/33/Lib/unittest/", line 440 in run
  File "/home/eliben/python-src/33/Lib/unittest/", line 492 in __call__
  File "/home/eliben/python-src/33/Lib/unittest/", line 105 in run
  File "/home/eliben/python-src/33/Lib/unittest/", line 67 in __call__
  File "/home/eliben/python-src/33/Lib/unittest/", line 105 in run
  File "/home/eliben/python-src/33/Lib/unittest/", line 67 in __call__
  File "/home/eliben/python-src/33/Lib/test/", line 1282 in run
  File "/home/eliben/python-src/33/Lib/test/", line 1383 in _run_suite
  File "/home/eliben/python-src/33/Lib/test/", line 1417 in run_unittest
  File "/home/eliben/python-src/33/Lib/test/", line 2311 in test_main
  File "/home/eliben/python-src/33/Lib/test/", line 1238 in runtest_inner
  File "/home/eliben/python-src/33/Lib/test/", line 919 in runtest
  File "/home/eliben/python-src/33/Lib/test/", line 710 in main
  File "/home/eliben/python-src/33/Lib/test/", line 1829 in <module>
  File "/home/eliben/python-src/33/Lib/", line 75 in _run_code
  File "/home/eliben/python-src/33/Lib/", line 162 in _run_module_as_main
Segmentation fault (core dumped)

Note that this is the Python test crashing. test_xml_etree_c works fine
msg162884 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2012-06-15 10:49
As suspected, the cause is that xinclude_loader manages to somehow import the C version of ET, although test_xml_etree tries to enforce the Python version. This is probably because test___all__ imports all modules and leaves stuff in the import cache.

What causes the segfault itself is that C Elements are mixed with Python Elements, and they're incompatible.
msg162925 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2012-06-15 17:57
Here is a patch that solves the current problem.

A longer term solution would be to have a cleaner test plan for ET in general, without monkey-patching at all, and without state that causes test-order dependencies.
msg162951 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-06-16 03:49
New changeset 5782efaa8d68 by Eli Bendersky in branch 'default':
Make the test more resilient to test-run order (closes #15075)
msg162953 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2012-06-16 03:53
Opened #15083 to track the *actual* solution to this problem, which should restructure the tests to be safer.
msg162957 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2012-06-16 07:35
>   File "/home/eliben/python-src/33/Lib/test/", line 710 in main
>   File "/home/eliben/python-src/33/Lib/test/", line 1829 in <module>
>   File "/home/eliben/python-src/33/Lib/", line 75 in _run_code
>   File "/home/eliben/python-src/33/Lib/", line 162 in _run_module_as_main
> Segmentation fault (core dumped)

Even if the ET tests are hacking sys.modules, I assume that the Python interpreter should not crash so badly.
It reveals probably a flaw somewhere else.

I suggest to re-open this to track the Segmentation fault.
msg162967 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2012-06-16 12:08
No need, the cause for the segfault is known. What happens is that Element objects from the Python module get intermixed with Element objects from the C module, and these are incompatible. The C module traversal functions assume they can cast Elements to a known structure (ElementObject), and when they get the invalid Elements they crash. Adding runtime checks everywhere is too costly. This situation cannot arise in a valid way.
msg178654 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2012-12-31 08:34
If runtime checks are needed to prevent mixing arbitrary objects into the tree, then I don't think they should be considered too costly.

I agree with Florent that this is worth reopening. It doesn't look like a "Tests" bug to me rather a "Lib"/"XML" bug.
msg179124 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-01-05 15:01
I investigated a bit, and type checks were added to protect from such mixing. Please open a new patch with a speficic reproducer if you run into similar problems (interepreter crash).
Date User Action Args
2022-04-11 14:57:31adminsetgithub: 59280
2013-01-05 15:01:11eli.benderskysetmessages: + msg179124
2012-12-31 08:34:58scodersetnosy: + scoder
messages: + msg178654
2012-06-16 12:08:36eli.benderskysetmessages: + msg162967
2012-06-16 07:35:37floxsetmessages: + msg162957
2012-06-16 03:53:16eli.benderskysetsuperseder: Rewrite ElementTree tests in a cleaner and safer way
messages: + msg162953
2012-06-16 03:49:33python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg162951

resolution: fixed
stage: test needed -> resolved
2012-06-15 17:57:31eli.benderskysetfiles: + issue15075.1.patch
keywords: + patch
messages: + msg162925
2012-06-15 10:49:32eli.benderskysetmessages: + msg162884
2012-06-15 10:01:43eli.benderskysetmessages: + msg162881
2012-06-15 08:34:31ncoghlansetnosy: + ncoghlan
2012-06-15 08:18:05eli.benderskycreate