PPC64 AIX 3.x:
https://buildbot.python.org/all/#/builders/10/builds/2224

0:12:47 [160/420/1] test_threading failed (env changed)
...
Ran 158 tests in 12.023s

OK (skipped=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['core']


I don't know if it's a regression or if it occurred in the past.

Michael: Can you please try to reproduce the issue? Any idea if it's new or not?

You can try to identify which test creates a core dump using:

$ ./python -m test.bisect_cmd --fail-env-changed test_thread -v

The command is supposed to find which test creates the "core" file.

See also https://bugs.python.org/issue35828#msg337076 where test_multiprocessing_fork seemed to leave a core dump. Maybe one test core dumps and leaves the report to the other causing env changed?

David Edelsohn: Would you mind to have a look at this issue?

I have looked at this briefly.

I would consider it a regression, or a spurious incident as the bot run before (https://buildbot.python.org/all/#/builders/10/builds/2223) and after (https://buildbot.python.org/all/#/builders/10/builds/2225) do not show this error. It may also be spurious - as there are other users on that bot system and the cause could be a side-effect of something else.

FYI: The error message is something similar to what I see when building using xlc - there is a core dump - but I cannot get a copy of the core dump to evaluate it.

See: https://bugs.python.org/issue35828#msg336134 - where I ask for a hint on how to get the code that reports the ENV change to save/copy a core dump.

The same bug is back:
https://buildbot.python.org/all/#/builders/10/builds/2443

Warning -- files was modified by test_threading
  Before: []
  After:  ['core']

On 17/04/2019 14:02, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> The same bug is back:
> https://buildbot.python.org/all/#/builders/10/builds/2443
>
> Warning -- files was modified by test_threading
>   Before: []
>   After:  ['core']
>
> ----------

What can I do to "copy" the core dump before normal testing removes it?

I have had a similar message regarding the multiprocessing tests, but in
those cases I expect it is the "client" thread complaining about a
"server" thread crash.

>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue36273>
> _______________________________________
>

Again - how can I get the core (dump) mentioned in the error message. When I force this situation I have several core dumps - not "the one".

> Again - how can I get the core (dump) mentioned in the error message. When I force this situation I have several core dumps - not "the one".

You have to contact the owner of the buildbot worker. From a build, you can click on the Worker tab. For PPC64 AIX, I see:

   David Edelsohn <dje.gcc@gmail.com>

test_threading has a similar issue, still on PPC64 AIX 3.x:
https://buildbot.python.org/all/#builders/10/builds/3171

test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... FAIL
test_releasing_unacquired_lock (test.test_threading.ThreadingExceptionTests) ... ok
test_start_thread_again (test.test_threading.ThreadingExceptionTests) ... ok
test_init_immutable_default_args (test.test_threading.TimerTests) ... ok

======================================================================
FAIL: test_recursion_limit (test.test_threading.ThreadingExceptionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_threading.py", line 1088, in test_recursion_limit
    self.assertEqual(p.returncode, 0, "Unexpected error: " + stderr.decode())
AssertionError: -11 != 0 : Unexpected error:

----------------------------------------------------------------------

Ran 168 tests in 12.623s

FAILED (failures=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['core']

Same test_threading crash on POWER6 AIX 3.x:
https://buildbot.python.org/all/#/builders/161/builds/1430

This build was triggered by the change of bpo-18049 which is supposed to only impact macOS, but it does touch test_recursion_limit... can it be related to the regression?

https://github.com/python/cpython/commit/1a057bab0f18d6ad843ce321d1d77a4819497ae4

test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... FAIL
test_releasing_unacquired_lock (test.test_threading.ThreadingExceptionTests) ... ok
test_start_thread_again (test.test_threading.ThreadingExceptionTests) ... ok
test_init_immutable_default_args (test.test_threading.TimerTests) ... ok

======================================================================
FAIL: test_recursion_limit (test.test_threading.ThreadingExceptionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_threading.py", line 1088, in test_recursion_limit
    self.assertEqual(p.returncode, 0, "Unexpected error: " + stderr.decode())
AssertionError: -11 != 0 : Unexpected error: 

----------------------------------------------------------------------

Ran 168 tests in 12.623s

FAILED (failures=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['core']

On 01/08/2019 11:15, STINNER Victor wrote:
> FAILED (failures=1)
> Warning -- files was modified by test_threading
>   Before: []
>   After:  ['core']

Thanks. I'll look again (on my bot) and other test systems.

What I assume has not been clear about my question - from the output I
see that there is, somewhere, a file named 'core' - but I can never find
it - and my assumption is that the test environment is cleaning up after
itself.

So, rephrased - is there a way to stop running test cycle when there is
an environment error, especially with a core dump, so that it can be
examined for further clues.

Michael

"""
What I assume has not been clear about my question - from the output I
see that there is, somewhere, a file named 'core' - but I can never find
it - and my assumption is that the test environment is cleaning up after
itself.
"""

Right, regrtest removes all temporary directories used for tests. You should try to extract the code run by the test and run it manually.

For example for test_recursion_limit(), you can use attached recurse.py.

resolved via issue18049

correction - issue18049 was related to test_threading. cannot say for sure that "this" is resolved by 18049. apologies for noise.

I didn't see this failure on PPC64 AIX 3.x recently, I close the issue.

"""
OK (skipped=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['core']
"""

That's maybe bpo-40068 "test_threading: ThreadJoinOnShutdown.test_reinit_tls_after_fork() crash with Python 3.8 on AIX".