This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: edelsohn-* buildbot worker failing with: No space left on device
Type: Stage: resolved
Components: Tests Versions: Python 3.10
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Buildbot: workers detached every minute and "no space left on device" issue
View: 41642
Assigned To: Nosy List: David.Edelsohn, cstratak, vstinner
Priority: normal Keywords:

Created on 2020-08-27 12:46 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (10)
msg375988 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-08-27 12:46
Issue similar to bpo-41642 "RHEL and fedora buildbots fail due to disk space error", multiple edelsohn-* buildbot worker are failing with: No space left on device.
msg375989 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-08-27 13:02
Example of affected workers.

edelsohn-fedora-rawhide-z: s390x Fedora Rawhide Refleaks 3.8
https://buildbot.python.org/all/#/builders/56/builds/8
msg375990 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-08-27 13:02
David: Would you mind to have a look?
msg375996 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-08-27 15:36
"xtreak	set nosy: + David.Edelsohn"

Strange. I'm quite sure that I added him to the nosy list when I wrote my previous comment.
msg375998 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2020-08-27 15:49
Note: I found that all my Fedora and RHEL8 buildbots have issues with connecting to the master buildbot server.

Not sure if a related package got updated and prompts the disconnects (as the RHEL7 ones are fine), or if it's relevant to the master buildbot server migration.
msg375999 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2020-08-27 15:56
It really would be better to discuss this in the builtbot mailing list and not Python issue tracker.

I have been seeing problems with many buildbots for the past few weeks, not just the s390x bots.  I receive messages that all of the bots disconnected, but the bots themselves are fine.  I also find many temporary files left in the bots, filling the /tmp filesystem.

I believe that something else has degraded.
msg376000 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-08-27 15:58
> It really would be better to discuss this in the builtbot mailing list and not Python issue tracker.

Pablo, me and a few otheres are using the Python bug tracker to track buildbot issues. But I'm fine with also discussing these issues on python-buildbot list. I suggested to Charalampos to write there:
https://mail.python.org/archives/list/python-buildbots@python.org/thread/R4CPNQFTOHBKU5VHCOPM2OBXWY5JC5SC/
msg376001 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2020-08-27 16:03
If they want to discuss here, fine, but the failing bot is a symptom not the problem.  The Buidlbot infrastructure is unstable, causing disconnects and leaving un-removed files.  If one looks at the workers, one can see many of the architectures in Exception states.

I would appreciate a discussion of the underlying issue and not "edelsohn-* buildbot workers failing".
msg376002 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-08-27 16:08
> I would appreciate a discussion of the underlying issue and not "edelsohn-* buildbot workers failing".

FYI Charalampos, Pablo and me are investigating the root issues.
msg376005 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-08-27 16:26
Hum, there are many affected workers. I will reuse bpo-41642 instead.
History
Date User Action Args
2022-04-11 14:59:35adminsetgithub: 85814
2020-08-27 16:26:35vstinnersetstatus: open -> closed
superseder: Buildbot: workers detached every minute and "no space left on device" issue
messages: + msg376005

resolution: duplicate
stage: resolved
2020-08-27 16:08:22vstinnersetmessages: + msg376002
2020-08-27 16:03:42David.Edelsohnsetmessages: + msg376001
2020-08-27 15:58:21vstinnersetmessages: + msg376000
2020-08-27 15:56:31David.Edelsohnsetmessages: + msg375999
2020-08-27 15:49:08cstrataksetnosy: + cstratak
messages: + msg375998
2020-08-27 15:36:05vstinnersetmessages: + msg375996
2020-08-27 14:58:12xtreaksetnosy: + David.Edelsohn
2020-08-27 13:02:47vstinnersetmessages: + msg375990
2020-08-27 13:02:04vstinnersetmessages: + msg375989
2020-08-27 12:46:41vstinnercreate