classification
Title: test_thread leaks a core dump on PPC64 AIX 3.x
Type: Stage:
Components: Tests Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Michael.Felt, vstinner, xtreak
Priority: normal Keywords:

Created on 2019-03-12 16:14 by vstinner, last changed 2019-08-03 10:43 by Michael.Felt.

Files
File name Uploaded Description Edit
recurse.py vstinner, 2019-08-01 14:43
Messages (14)
msg337772 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-12 16:14
PPC64 AIX 3.x:
https://buildbot.python.org/all/#/builders/10/builds/2224

0:12:47 [160/420/1] test_threading failed (env changed)
...
Ran 158 tests in 12.023s

OK (skipped=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['core']


I don't know if it's a regression or if it occurred in the past.

Michael: Can you please try to reproduce the issue? Any idea if it's new or not?

You can try to identify which test creates a core dump using:

$ ./python -m test.bisect_cmd --fail-env-changed test_thread -v

The command is supposed to find which test creates the "core" file.
msg337775 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-03-12 16:39
See also https://bugs.python.org/issue35828#msg337076 where test_multiprocessing_fork seemed to leave a core dump. Maybe one test core dumps and leaves the report to the other causing env changed?
msg338178 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-18 09:04
David Edelsohn: Would you mind to have a look at this issue?
msg338975 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-03-27 16:52
I have looked at this briefly.

I would consider it a regression, or a spurious incident as the bot run before (https://buildbot.python.org/all/#/builders/10/builds/2223) and after (https://buildbot.python.org/all/#/builders/10/builds/2225) do not show this error. It may also be spurious - as there are other users on that bot system and the cause could be a side-effect of something else.

FYI: The error message is something similar to what I see when building using xlc - there is a core dump - but I cannot get a copy of the core dump to evaluate it.

See: https://bugs.python.org/issue35828#msg336134 - where I ask for a hint on how to get the code that reports the ENV change to save/copy a core dump.
msg340391 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-17 12:02
The same bug is back:
https://buildbot.python.org/all/#/builders/10/builds/2443

Warning -- files was modified by test_threading
  Before: []
  After:  ['core']
msg340395 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-04-17 13:41
On 17/04/2019 14:02, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> The same bug is back:
> https://buildbot.python.org/all/#/builders/10/builds/2443
>
> Warning -- files was modified by test_threading
>   Before: []
>   After:  ['core']
>
> ----------

What can I do to "copy" the core dump before normal testing removes it?

I have had a similar message regarding the multiprocessing tests, but in
those cases I expect it is the "client" thread complaining about a
"server" thread crash.

>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue36273>
> _______________________________________
>
msg343012 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-05-21 11:14
Again - how can I get the core (dump) mentioned in the error message. When I force this situation I have several core dumps - not "the one".
msg348844 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-01 09:12
> Again - how can I get the core (dump) mentioned in the error message. When I force this situation I have several core dumps - not "the one".

You have to contact the owner of the buildbot worker. From a build, you can click on the Worker tab. For PPC64 AIX, I see:

   David Edelsohn <dje.gcc@gmail.com>
msg348845 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-01 09:13
test_threading has a similar issue, still on PPC64 AIX 3.x:
https://buildbot.python.org/all/#builders/10/builds/3171

test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... FAIL
test_releasing_unacquired_lock (test.test_threading.ThreadingExceptionTests) ... ok
test_start_thread_again (test.test_threading.ThreadingExceptionTests) ... ok
test_init_immutable_default_args (test.test_threading.TimerTests) ... ok

======================================================================
FAIL: test_recursion_limit (test.test_threading.ThreadingExceptionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_threading.py", line 1088, in test_recursion_limit
    self.assertEqual(p.returncode, 0, "Unexpected error: " + stderr.decode())
AssertionError: -11 != 0 : Unexpected error:

----------------------------------------------------------------------

Ran 168 tests in 12.623s

FAILED (failures=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['core']
msg348846 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-01 09:15
Same test_threading crash on POWER6 AIX 3.x:
https://buildbot.python.org/all/#/builders/161/builds/1430

This build was triggered by the change of bpo-18049 which is supposed to only impact macOS, but it does touch test_recursion_limit... can it be related to the regression?

https://github.com/python/cpython/commit/1a057bab0f18d6ad843ce321d1d77a4819497ae4

test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... FAIL
test_releasing_unacquired_lock (test.test_threading.ThreadingExceptionTests) ... ok
test_start_thread_again (test.test_threading.ThreadingExceptionTests) ... ok
test_init_immutable_default_args (test.test_threading.TimerTests) ... ok

======================================================================
FAIL: test_recursion_limit (test.test_threading.ThreadingExceptionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_threading.py", line 1088, in test_recursion_limit
    self.assertEqual(p.returncode, 0, "Unexpected error: " + stderr.decode())
AssertionError: -11 != 0 : Unexpected error: 

----------------------------------------------------------------------

Ran 168 tests in 12.623s

FAILED (failures=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['core']
msg348854 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-08-01 13:40
On 01/08/2019 11:15, STINNER Victor wrote:
> FAILED (failures=1)
> Warning -- files was modified by test_threading
>   Before: []
>   After:  ['core']

Thanks. I'll look again (on my bot) and other test systems.

What I assume has not been clear about my question - from the output I
see that there is, somewhere, a file named 'core' - but I can never find
it - and my assumption is that the test environment is cleaning up after
itself.

So, rephrased - is there a way to stop running test cycle when there is
an environment error, especially with a core dump, so that it can be
examined for further clues.

Michael
msg348861 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-01 14:43
"""
What I assume has not been clear about my question - from the output I
see that there is, somewhere, a file named 'core' - but I can never find
it - and my assumption is that the test environment is cleaning up after
itself.
"""

Right, regrtest removes all temporary directories used for tests. You should try to extract the code run by the test and run it manually.

For example for test_recursion_limit(), you can use attached recurse.py.
msg348949 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-08-03 10:39
resolved via issue18049
msg348950 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-08-03 10:43
correction - issue18049 was related to test_threading. cannot say for sure that "this" is resolved by 18049. apologies for noise.
History
Date User Action Args
2019-08-03 10:43:27Michael.Feltsetmessages: + msg348950
2019-08-03 10:39:59Michael.Feltsetmessages: + msg348949
2019-08-01 14:43:26vstinnersetfiles: + recurse.py

messages: + msg348861
2019-08-01 13:40:24Michael.Feltsetmessages: + msg348854
2019-08-01 09:15:47vstinnersetmessages: + msg348846
2019-08-01 09:13:10vstinnersetmessages: + msg348845
2019-08-01 09:12:56vstinnersetmessages: + msg348844
2019-05-21 11:14:43Michael.Feltsetmessages: + msg343012
2019-04-17 13:41:04Michael.Feltsetmessages: + msg340395
2019-04-17 12:02:30vstinnersetmessages: + msg340391
2019-03-27 16:52:20Michael.Feltsetmessages: + msg338975
2019-03-18 09:04:05vstinnersetmessages: + msg338178
2019-03-12 16:39:08xtreaksetnosy: + xtreak
messages: + msg337775
2019-03-12 16:14:04vstinnercreate