classification
Title: Remove Travis CI macOS job: rely on buildbots
Type: Stage: resolved
Components: Build Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, ned.deily, pitrou, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2017-09-05 23:24 by vstinner, last changed 2018-02-27 20:40 by pitrou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3367 merged vstinner, 2017-09-05 23:25
PR 5858 merged pitrou, 2018-02-24 20:28
Messages (15)
msg301398 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-05 23:24
Currently, Travis CI has a job running tests on macOS. Sadly, there is a bug on Travis: *sometimes*, the macOS job blocks a Pull Request to be merged even if macOS is marked as "Allowed Failure". If you cancel the job, it's even worse: Travis CI is marked a failed and the PR cannot be merged.

I tried to cancel the build and restart it, but 2 hours later, the macOS job was still queued and I was still unable to merge my PR.

We cannot work around the Travis CI issue. We are getting more and more pull requests, especially this week with the CPython sprint, so the macOS queue is getting even longer.

I discussed the issue with Zachary Ware and we agreed to "disable" ("remove" in practice) the macOS job for this week, and rely on the macOS buildbots instead.

We have 3 macOS buildbots (x86 Tiger 3.x, x86-64 El Capitan 3.x, x86-64 Sierra 3.x) and I worked hard last months to get buildbots green. So we should be quickly notified of failures. There is now the buildbot-status mailing list for this purpose!

https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/

I already reported the issue twice on python-committers mailing list in June and September (this year):

* https://mail.python.org/pipermail/python-committers/2017-September/004824.html
* https://mail.python.org/pipermail/python-committers/2017-June/004661.html
msg301408 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-05 23:50
Failures of the two Allowed Failure jobs, coverage and macOS, are not reported to GitHub PR. For example, the coverage job is failing on Python 3.6 since May 2016 at least, and nobody noticed.

I don't think that it's worth it to waste Travis CI resources if nobody notices failures.

See https://github.com/python/core-workflow/issues/101 for the discussion on Allowed Failures on Travis CI.
msg301411 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-09-06 00:09
Since Travis CI time is a bottleneck for the ongoing sprint, I agree that it makes sense to disable the macOS build checks for now.  Longer term, it would be nice to resolve the problem but we certainly could rely on our own macOS buildbots; I'm not aware of many (any?) instances yet where the Travis macOS instance has caught a problem.  I think Brett should make the call on whether to permanently disable this after he gets back.
msg301413 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-09-06 00:12
I will note that my on-going research on rewriting the buildbot config makes it look like we should be able to securely enable building PRs by checking for the 'awaiting merge' label.
msg301421 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-06 00:57
New changeset 501b324d3a940d26e0021a38aae8d896a30fbcff by Victor Stinner in branch 'master':
bpo-31355: Travis CI: remove the macOS job (#3367)
https://github.com/python/cpython/commit/501b324d3a940d26e0021a38aae8d896a30fbcff
msg301430 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-09-06 01:43
New changeset 11453524ed26ee449275c32bedfd86ef19dd91ee by Ned Deily (Victor Stinner) in branch '3.6':
bpo-31355: Travis CI: remove the macOS job (#3367)
https://github.com/python/cpython/commit/11453524ed26ee449275c32bedfd86ef19dd91ee
msg301431 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-09-06 01:45
New changeset 2bc83afb30850f682487ffa560c9e3663788baaa by Ned Deily (Victor Stinner) in branch '2.7':
bpo-31355: Travis CI: remove the macOS job (#3367)
https://github.com/python/cpython/commit/2bc83afb30850f682487ffa560c9e3663788baaa
msg301466 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-06 12:21
> I'm not aware of many (any?) instances yet where the Travis macOS instance has caught a problem.

It did help me on some delicate signals/threading issues.
msg302550 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-09-19 17:48
Should this be closed?
msg302815 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-23 22:33
Yeah, I consider that the issue can now be closed. If someone really wants runnings checks on macOS, we should wait until Travis CI solves their issues, or find a different CI provider.

https://blog.travis-ci.com/2017-09-22-macos-update
msg312631 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-02-23 11:35
The situation with macOS builds on Travis-CI is now much better (no more long waiting queues) so we might revisit this decision.
msg312647 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-02-23 17:07
I'm willing to give it another go. Do you want to open a PR to turn it back on and see how the timing looks, Antoine?
msg312761 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-02-24 21:47
Last macOS job took 13 minutes and there was almost no wait.  That looks ok.  I'm not sure it's required to backport to 3.6 and 2.7 since those branches typically have less churn.
msg312831 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-02-25 18:36
Awesome, thanks for taking the time to verify all of that! And I agree with your logic to not bother backporting.
msg313023 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-02-27 20:40
New changeset d7687eb4b66c9f675b112eff1169326a1766c111 by Antoine Pitrou in branch 'master':
bpo-31355: Travis-CI: re-enable macOS job (#5858)
https://github.com/python/cpython/commit/d7687eb4b66c9f675b112eff1169326a1766c111
History
Date User Action Args
2018-02-27 20:40:55pitrousetstatus: open -> closed
stage: patch review -> resolved
2018-02-27 20:40:39pitrousetmessages: + msg313023
2018-02-25 18:36:52brett.cannonsetmessages: + msg312831
2018-02-24 21:47:01pitrousetmessages: + msg312761
2018-02-24 20:28:43pitrousetkeywords: + patch
stage: resolved -> patch review
pull_requests: + pull_request5633
2018-02-23 17:07:10brett.cannonsetstatus: pending -> open

messages: + msg312647
2018-02-23 11:35:44pitrousetstatus: closed -> pending

messages: + msg312631
2017-09-23 22:33:01vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg302815

stage: resolved
2017-09-19 17:48:19brett.cannonsetnosy: + brett.cannon
messages: + msg302550
2017-09-06 12:21:37pitrousetnosy: + pitrou
messages: + msg301466
2017-09-06 01:45:29ned.deilysetmessages: + msg301431
2017-09-06 01:43:16ned.deilysetmessages: + msg301430
2017-09-06 01:14:13vstinnersettitle: Remove Travis CI job: rely on buildbots -> Remove Travis CI macOS job: rely on buildbots
2017-09-06 00:57:16vstinnersetmessages: + msg301421
2017-09-06 00:12:04zach.waresetmessages: + msg301413
2017-09-06 00:09:22ned.deilysetmessages: + msg301411
2017-09-05 23:50:34vstinnersetmessages: + msg301408
2017-09-05 23:25:10vstinnersetpull_requests: + pull_request3377
2017-09-05 23:24:41vstinnercreate