Lib/trace.cover is not removed by the clean target, and apparently this file is always created in the source tree, not in the build tree.  It should be created in the build tree, if that's not possible, it should be cleaned in any case.

It seems that this file is left by test_trace after commit 47ab15470d72367694d7758004067313ae022f0e.

PR 8841 prevents creating Lib/trace.cover when run the trace module.

New changeset c406d5cd74002964a64c3eb7d9e2445a7fd3a03f by Serhiy Storchaka in branch 'master':
bpo-34171: Prevent creating Lib/trace.cover when run the trace module. (GH-8841)
https://github.com/python/cpython/commit/c406d5cd74002964a64c3eb7d9e2445a7fd3a03f

New changeset 80e9fedcf579537c5bc51ff3c9c18d14d9598564 by Miss Islington (bot) in branch '3.7':
bpo-34171: Prevent creating Lib/trace.cover when run the trace module. (GH-8841)
https://github.com/python/cpython/commit/80e9fedcf579537c5bc51ff3c9c18d14d9598564

Several buildbots are failing due to commit c406d5cd74002964a64c3eb7d9e2445a7fd3a03f:

https://buildbot.python.org/all/#/builders/40/builds/709
https://buildbot.python.org/all/#/builders/21/builds/1459
https://buildbot.python.org/all/#/builders/13/builds/1476
https://buildbot.python.org/all/#/builders/53/builds/1470
https://buildbot.python.org/all/#/builders/133/builds/477
https://buildbot.python.org/all/#/builders/145/builds/327
https://buildbot.python.org/all/#/builders/85/builds/1302
https://buildbot.python.org/all/#/builders/141/builds/358
https://buildbot.python.org/all/#/builders/126/builds/564
https://buildbot.python.org/all/#/builders/147/builds/213
https://buildbot.python.org/all/#/builders/119/builds/611
https://buildbot.python.org/all/#/builders/116/builds/590
https://buildbot.python.org/all/#/builders/143/builds/233
https://buildbot.python.org/all/#/builders/93/builds/1072
https://buildbot.python.org/all/#/builders/3/builds/1257
https://buildbot.python.org/all/#/builders/32/builds/1331
https://buildbot.python.org/all/#/builders/115/builds/520
https://buildbot.python.org/all/#/builders/128/builds/571
https://buildbot.python.org/all/#/builders/99/builds/1335
https://buildbot.python.org/all/#/builders/121/builds/497
https://buildbot.python.org/all/#/builders/113/builds/540
https://buildbot.python.org/all/#/builders/154/builds/166

Sample error:

======================================================================
FAIL: test_cover_files_written_no_highlight (test.test_trace.TestCoverageCommandLineOutput)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_trace.py", line 393, in test_cover_files_written_no_highlight
    self.assertFalse(os.path.exists(tracecoverpath))
AssertionError: True is not false
----------------------------------------------------------------------
Ran 21 tests in 17.796s
FAILED (failures=1)
test test_trace failed
1 test failed again:
    test_trace

Seems this is a temporary issue. Tests are failed due to trace.cover files left from previous runs. Recreating the working space from zero should fix the issue, but I don't know how to do this on all buildbots.

@zach.ware Is there a way to clean the workspace of these buildbots programmatically/from the UI? I can do it myself if you tell me how it must be done.

No, there isn't.  However, https://github.com/python/buildmaster-config/pull/43 should help; I'll try to get that deployed later today.

It looks like https://github.com/python/buildmaster-config/pull/43 won't actually help; I changed it to use `method='clean'` because we don't want to blow away the `externals` directory on Windows buildbots, and `*.cover` is in `.gitignore` so `Lib/trace.cover` will not be removed.

Can we manually ssh into the buildbots from the buildbot master and delete the file?

It may be easier to remove "trace.cover" before testing every time than clean up all buildbots once.

> Can we manually ssh into the buildbots from the buildbot master and delete the file?

No, there is no access to the workers except for the connection created by each worker for the purpose of buildbot communication (and individually by each worker owner).

> It may be easier to remove "trace.cover" before testing every time than clean up all buildbots once.

I agree.  Alternately, the test could be changed to avoid that assertion; it's strange that the test could fail just because a file could happen to exist outside of the test's control.

New changeset b44a1d4f71daf4edb948b9878f82a60891e4a2e1 by Serhiy Storchaka in branch 'master':
bpo-34171: Fix test_trace. (GH-8940)
https://github.com/python/cpython/commit/b44a1d4f71daf4edb948b9878f82a60891e4a2e1

New changeset e3f20828f6b7f59baa2d25437176ee32ae7700b0 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7':
bpo-34171: Fix test_trace. (GH-8940) (GH-8951)
https://github.com/python/cpython/commit/e3f20828f6b7f59baa2d25437176ee32ae7700b0