This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Restore caching of externals on Windows buildbots
Type: Stage:
Components: Build, Tests, Windows Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: jeremy.kloth, jkloth, pablogsal, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2022-02-18 17:00 by jkloth, last changed 2022-04-11 14:59 by admin.

Messages (5)
msg413494 - (view) Author: Jeremy Kloth (jkloth) * Date: 2022-02-18 17:00
A recent change to the buildmaster config effectively disabled the caching of the externals for Windows buildbots:
   https://github.com/python/buildmaster-config/pull/255

If the caching is desired, a simple change to the buildmaster config is needed (define EXTERNALS_DIR in the build environment).  Or, to continue with fetching them each run, the buildbot scripts in Tools\buildbot can be simplified.

Once a course of action is determined I can develop the requisite PR(s) in the appropriate tracker.
msg413498 - (view) Author: Jeremy Kloth (jkloth) * Date: 2022-02-18 17:52
I personally would like to see caching restored so as to keep the duration of buildbot runs as low as possible.  The repeated fetching effectively doubles compilation time for my Win11 builder.
msg413650 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-21 12:34
Would it be possible to create a download cache somewhere outside the Python source tree, so "git clean -fdx" would not remove this cache?

Some CIs implement such cache. Does buildbot have helpers for that?

buildbot provides for example:

* LRUCache: https://docs.buildbot.net/2.4.0/full.html#module-buildbot.util.lru
* State: https://docs.buildbot.net/2.4.0/full.html#keeping-state
* Server cache: https://docs.buildbot.net/2.4.0/full.html#caches

In this case, one option would be to enhance PCbuild/get_external.py to add support for a cache directory.
msg413655 - (view) Author: Jeremy Kloth (jeremy.kloth) Date: 2022-02-21 15:00
> Would it be possible to create a download cache somewhere outside the Python source tree, so "git clean -fdx" would not remove this cache?

I was thinking of locating it next to the checkout directory.  The
current structure is:

[worker root]
-- [builder root]
---- [checkout]

I propose to add the externals directory within the builder root, so
each branch would still have a unique copy.
msg413680 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-02-21 21:00
> one option would be to enhance PCbuild/get_external.py to add support 
for a cache directory.

It should already have this support - set the EXTERNALS_DIR environment 
variable before building. I use this in my own builds.

Though if there are specific platforms worth integrating with, this 
could no doubt be further improved.
History
Date User Action Args
2022-04-11 14:59:56adminsetgithub: 90945
2022-02-21 21:00:00steve.dowersetmessages: + msg413680
2022-02-21 15:00:07jeremy.klothsetnosy: + jeremy.kloth
messages: + msg413655
2022-02-21 12:34:07vstinnersetmessages: + msg413650
2022-02-18 17:52:35jklothsetmessages: + msg413498
2022-02-18 17:00:52jklothsetnosy: + vstinner, pablogsal
2022-02-18 17:00:25jklothcreate