classification
Title: AppVeyor builds interrupted before tests complete
Type: Stage: resolved
Components: Tests Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, steve.dower, vstinner
Priority: normal Keywords: patch

Created on 2018-06-04 00:53 by vstinner, last changed 2018-06-06 09:44 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 7368 closed vstinner, 2018-06-04 01:04
PR 7364 merged zach.ware, 2018-06-04 01:10
PR 7369 merged miss-islington, 2018-06-04 01:14
PR 7379 merged ned.deily, 2018-06-04 04:24
PR 7380 merged ned.deily, 2018-06-04 04:41
Messages (15)
msg318590 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-04 00:53
While looking at AppVeyor history, I saw two builds which seem like interrupted in the middle of the test

3.6:
https://ci.appveyor.com/project/python/cpython/build/3.6build16876/job/t9nyt59wkwcn68nk

...
0:04:32 [312/407] test_genexps passed -- running: test_largefile (2 min 21 sec), test_mmap (4 min 30 sec)
0:04:32 [313/407] test_weakset passed -- running: test_largefile (2 min 21 sec), test_mmap (4 min 30 sec)
0:04:32 [314/407] test_fractions passed -- running: test_largefile (2 min 21 sec), test_mmap (4 min 31 sec)

3.8:
https://ci.appveyor.com/project/python/cpython/build/3.8build16878

...
0:06:07 [290/416] test_frame passed -- running: test_io (4 min 1 sec), test_mmap (5 min 45 sec)
0:06:09 [291/416] test_codecmaps_cn passed -- running: test_io (4 min 3 sec), test_mmap (5 min 47 sec)
fetching http://www.pythontest.net/unicode/gb-18030-2000.xml ...
	fetching http://www.pythontest.net/unicode/EUC-CN.TXT ...
	fetching http://www.pythontest.net/unicode/CP936.TXT ...


Terry Reedy also noticed the issue:
https://mail.python.org/pipermail/python-committers/2018-June/005527.html


https://appveyor.statuspage.io/ says that all services are operational.
msg318592 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-04 00:56
See also PR 7365" href="issue33765">bpo-33765: "AppVeyor didn't start on my PR 7365".
msg318593 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 00:57
Clicking on the Messages tab:

https://ci.appveyor.com/project/python/cpython/build/3.8build16878/messages

"Build exceeded allowed resource quotas. Fix your build to consume less resources or contact AppVeyor support to request quotas increase."
msg318594 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 00:59
Looks like this was another case of test_mmap on Windows taking a long time.  If the change has now been made to disable tests with "largefile" resources, test_mmap should no longer be a hangup.
msg318595 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 01:01
We discussed this earlier: https://bugs.python.org/issue33355#msg317587
msg318596 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-04 01:02
> Looks like this was another case of test_mmap on Windows taking a long time.

Ah yes, bpo-33355.
msg318597 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-04 01:07
> We discussed this earlier: https://bugs.python.org/issue33355#msg317587

At that time, I didn't know that AppVeyor was impacted. In my mind, the bug was specific to buildbots on slow cloud providers (slow cloud flavors).
msg318598 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 01:13
New changeset 47a6c79f0986ce9a84d0efc3c565b28255042911 by Ned Deily (Zachary Ware) in branch 'master':
bpo-33764: Appveyor fixes (GH-7364)
https://github.com/python/cpython/commit/47a6c79f0986ce9a84d0efc3c565b28255042911
msg318600 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 01:17
New changeset 37343a2db8763a114e63ac63a249603b1df2554d by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-33764: Appveyor fixes (GH-7364) (GH-7369)
https://github.com/python/cpython/commit/37343a2db8763a114e63ac63a249603b1df2554d
msg318603 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 02:42
Steve, what do you think about a similar change to the VSTS Windows CI builds to disable the "longfile" resource for the tests?  test_mmap seems to take a long time.
msg318626 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 04:47
New changeset b8f2fb52007f6d0d9eacfb459a9e813678f55f7e by Ned Deily in branch '3.6':
bpo-33764: Appveyor fixes (GH-7364) (GH-7379)
https://github.com/python/cpython/commit/b8f2fb52007f6d0d9eacfb459a9e813678f55f7e
msg318628 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 04:52
New changeset 1cbdbf35db16f2b06c0974b711c4d8b840c8a264 by Ned Deily in branch '2.7':
bpo-33764: Appveyor fixes (GH-7364) (GH-7380)
https://github.com/python/cpython/commit/1cbdbf35db16f2b06c0974b711c4d8b840c8a264
msg318639 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-04 08:12
I asked AppVeyor to increase our quotas:
https://help.appveyor.com/discussions/problems/14532-cpython-exceeded-allowed-resource-quotas-what-are-these-quotas-can-them-be-increased
msg318652 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-06-04 13:25
Are we sure we haven't actually regressed something here and it's actually taking more resources? I can't tell from this discussion if it's memory CPU, disk or time that we're hitting problems.

I'll have to look at the VSTS build history to see whether it needs to be disabled there too. I haven't been watching much over the last week.
msg318806 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-06 09:44
Steve Dower: "Are we sure we haven't actually regressed something here and it's actually taking more resources? (...)"

According to AppVeyor support, there was an issue on their side:

"Regarding disk quota -- we are extremely sorry. This was not your fault, but side effect of build worker image update. You should be good now."

https://help.appveyor.com/discussions/problems/14532-cpython-exceeded-allowed-resource-quotas-what-are-these-quotas-can-them-be-increased

Moreover, AppVeyor donated us a second parallel job!

"We decided to donate additional parallel job to the python account. Let us know how it goes."

I close the issue. AppVeyor is back!
History
Date User Action Args
2018-06-06 09:44:09vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg318806

stage: patch review -> resolved
2018-06-04 13:25:20steve.dowersetmessages: + msg318652
2018-06-04 08:12:28vstinnersetmessages: + msg318639
2018-06-04 04:52:40ned.deilysetmessages: + msg318628
2018-06-04 04:47:41ned.deilysetmessages: + msg318626
2018-06-04 04:41:29ned.deilysetpull_requests: + pull_request7007
2018-06-04 04:24:17ned.deilysetpull_requests: + pull_request7006
2018-06-04 02:42:11ned.deilysetnosy: + steve.dower
messages: + msg318603
2018-06-04 01:17:02ned.deilysetmessages: + msg318600
2018-06-04 01:14:50miss-islingtonsetpull_requests: + pull_request6996
2018-06-04 01:13:33ned.deilysetmessages: + msg318598
2018-06-04 01:10:34zach.waresetpull_requests: + pull_request6995
2018-06-04 01:07:48vstinnersetmessages: + msg318597
2018-06-04 01:04:20vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request6994
2018-06-04 01:02:18vstinnersetmessages: + msg318596
2018-06-04 01:01:32ned.deilysetmessages: + msg318595
2018-06-04 00:59:27ned.deilysetmessages: + msg318594
2018-06-04 00:57:10ned.deilysetnosy: + ned.deily
messages: + msg318593
2018-06-04 00:56:03vstinnersetmessages: + msg318592
2018-06-04 00:53:32vstinnercreate