Lib/trace.cover is not removed by the clean target, and apparently this file is always created in the source tree, not in the build tree. It should be created in the build tree, if that's not possible, it should be cleaned in any case.

It seems that this file is left by test_trace after commit 47ab154.

PR 8841 prevents creating Lib/trace.cover when run the trace module.

New changeset c406d5c by Serhiy Storchaka in branch 'master':
bpo-34171: Prevent creating Lib/trace.cover when run the trace module. (GH-8841)
c406d5c

New changeset 80e9fed by Miss Islington (bot) in branch '3.7':
bpo-34171: Prevent creating Lib/trace.cover when run the trace module. (GH-8841)
80e9fed

Several buildbots are failing due to commit c406d5c:

https://buildbot.python.org/all/#/builders/40/builds/709
https://buildbot.python.org/all/#/builders/21/builds/1459
https://buildbot.python.org/all/#/builders/13/builds/1476
https://buildbot.python.org/all/#/builders/53/builds/1470
https://buildbot.python.org/all/#/builders/133/builds/477
https://buildbot.python.org/all/#/builders/145/builds/327
https://buildbot.python.org/all/#/builders/85/builds/1302
https://buildbot.python.org/all/#/builders/141/builds/358
https://buildbot.python.org/all/#/builders/126/builds/564
https://buildbot.python.org/all/#/builders/147/builds/213
https://buildbot.python.org/all/#/builders/119/builds/611
https://buildbot.python.org/all/#/builders/116/builds/590
https://buildbot.python.org/all/#/builders/143/builds/233
https://buildbot.python.org/all/#/builders/93/builds/1072
https://buildbot.python.org/all/#/builders/3/builds/1257
https://buildbot.python.org/all/#/builders/32/builds/1331
https://buildbot.python.org/all/#/builders/115/builds/520
https://buildbot.python.org/all/#/builders/128/builds/571
https://buildbot.python.org/all/#/builders/99/builds/1335
https://buildbot.python.org/all/#/builders/121/builds/497
https://buildbot.python.org/all/#/builders/113/builds/540
https://buildbot.python.org/all/#/builders/154/builds/166

Sample error:

======================================================================
FAIL: test_cover_files_written_no_highlight (test.test_trace.TestCoverageCommandLineOutput)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_trace.py", line 393, in test_cover_files_written_no_highlight
    self.assertFalse(os.path.exists(tracecoverpath))
AssertionError: True is not false

Ran 21 tests in 17.796s
FAILED (failures=1)
test test_trace failed
1 test failed again:
test_trace

Seems this is a temporary issue. Tests are failed due to trace.cover files left from previous runs. Recreating the working space from zero should fix the issue, but I don't know how to do this on all buildbots.

@zach.ware Is there a way to clean the workspace of these buildbots programmatically/from the UI? I can do it myself if you tell me how it must be done.

No, there isn't. However, python/buildmaster-config#43 should help; I'll try to get that deployed later today.

It looks like python/buildmaster-config#43 won't actually help; I changed it to use method='clean' because we don't want to blow away the externals directory on Windows buildbots, and *.cover is in .gitignore so [Lib/trace.cover](https://github.com/python/cpython/blob/main/Lib/trace.cover) will not be removed.

Can we manually ssh into the buildbots from the buildbot master and delete the file?

It may be easier to remove "trace.cover" before testing every time than clean up all buildbots once.

Can we manually ssh into the buildbots from the buildbot master and delete the file?

No, there is no access to the workers except for the connection created by each worker for the purpose of buildbot communication (and individually by each worker owner).

It may be easier to remove "trace.cover" before testing every time than clean up all buildbots once.

I agree. Alternately, the test could be changed to avoid that assertion; it's strange that the test could fail just because a file could happen to exist outside of the test's control.

New changeset b44a1d4 by Serhiy Storchaka in branch 'master':
bpo-34171: Fix test_trace. (GH-8940)
b44a1d4

New changeset e3f2082 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7':
bpo-34171: Fix test_trace. (GH-8940) (GH-8951)
e3f2082