classification
Title: Update Windows build batch scripts
Type: enhancement Stage: resolved
Components: Build, Windows Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: zach.ware Nosy List: BreamoreBoy, db3l, ibecker, jkloth, loewis, python-dev, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: buildbot, patch

Created on 2014-07-02 19:16 by zach.ware, last changed 2014-10-15 18:40 by db3l. This issue is now closed.

Files
File name Uploaded Description Edit
better_bats.diff zach.ware, 2014-07-02 19:16 review
devguide_update.diff zach.ware, 2014-07-02 19:22
fix_pgo_vcvarsall.diff ibecker, 2014-07-17 18:05
fix_pgo_vcvarsall_rightway.diff ibecker, 2014-07-17 19:47 review
test.bat db3l, 2014-09-04 23:01
Messages (25)
msg222118 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-07-02 19:16
I am aware of a few open issues with the build scripts provided for Windows (both the Tools/buildbot scripts and the .bat files provided in PCbuild), such as #9973 and #9822, and there are some other issues that bother me but that I haven't opened issues for.  Here's a patch which takes care of all of the issues I'm aware of, by almost completely rewriting most of the scripts.

An overview of the changes:

- Tools/buildbot/external*.bat are dead, replaced by PCbuild/get_externals.bat.  Since those scripts have been suggested for use by normal users by PCbuild/readme.txt and the devguide, stubs for Tools/buildbot/external.bat and external-amd64.bat still exist, which display a message suggesting PCbuild/get_externals.bat and then pass all arguments on to that script.

- PCbuild/get_externals.bat is a complete rewrite of Tools/buildbot/external-common.bat, and is written for users rather than buildbots.  It takes '-c/--clean' or '--clean-only' options, which will gently remove^W^W nuke from orbit all existing external library sources.

- PCbuild/build.bat has been updated to set up the environment itself, and to allow arguments to be passed to MSbuild, such as "/m" (though anything containing [ =,] (and possibly other characters) will require quotes to pass through properly, due to batch's wonderful argument handling abilities).  It has also grown an '-e' option, to pull in external libs using get_externals.bat.

- Tools/buildbot/build.bat has been updated to use PCbuild/build.bat, and Tool/buildbot/build-amd64.bat has been updated to use Tools/buildbot/build.bat.

- PCbuild/env.bat has been updated to allow for non-standard VS install locations, but really, it's obsolete with build.bat setting the environment itself.  I would prefer to eliminate PCbuild/build_env.bat and PCbuild/env.bat, but I don't know whether they're an integral part of other people's workflow and don't want to mess that up if they are.  There doesn't seem to be any harm from running the build with the environment set up twice.

- Tools/buildbot/clean.bat has been updated as I suggested in #9973, it has a set list of kill_python exes to try if they exist, then uses `hg purge` to clean *everything* out.  Tools/buildbot/clean-amd64.bat is now just a stub calling Tools/buildbot/clean.bat.

- Tools/buildbot/buildmsi.bat has a few updates, but I can't properly test them (having not yet managed to get an MSI built at all).  The script is currently unused anyway, as far as I know.

- Tools/buildbot/test[-amd64].bat have been updated to use Tools/scripts/run_tests.py, just like the UNIX buildbots.  run_tests.py has been updated to use subprocess.call instead of os.execv on Windows.

- All scripts have been made cwd-agnostic, by which I mean they should run the same way no matter where they are called from.  Also, every effort is made to not pollute the environment (even with VS setup).  They should also be a bit more robust against paths with spaces, but I make no guarantees about that.

- PCbuild/readme.txt has been updated to suggest PCbuild/get_externals.bat instead of Tools/buildbot/external*.bat, and has an added "Quick Start Guide" at the top.  A patch for the devguide will follow.

I think these changes make things simpler, with fewer places to update when options are changed, compilers upgraded, libraries updated, etc.  With this change, the simplest way to build on Windows becomes "PCbuild\build.bat -e" (or add "-d" for debug).  This is still not exactly what I want eventually (see #16895, which is now long out of date :), but is a solid step in that direction.

Most of these changes could be backported to 3.4, and at least a good portion of those to 2.7.  I'm not exactly sure how worth it that would be, though.

Thoughts?
msg222119 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-07-02 19:22
Here's the devguide patch.
msg222123 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-07-02 20:53
Looks pretty good. I'm happy to see more move into PCBuild - ideally, people building a Python release should never have to look anywhere else.

buildmsi.bat can probably go away completely if the buildbots aren't using it. 3.5 will eventually have a .wixproj to build the MSI and there'll be a buildrelease.bat or similar under tools/ to keep the single entry point.

As part of the VC14 change there'll be changes to the batch files, but as far as entry points go they'll still be there. I want to move most of the functionality into an MSBuild script (currently pcbuild.proj in my sandbox) since that is generally more flexible than cmd.exe, but until then this looks like a great improvement. Not quite the 'make' equivalent you have in #16895, but that will be easy to write when "make (\w+)" translates into "msbuild pcbuild.proj /t:\1".

I don't see any value in backporting to 2.7. I've got my own scripts for that which make doing a release very straightforward, and I'm happy to keep it that way. That said, the easier we make it for people to build from source, the sooner we can stop doing binary releases for 2.7.

Consider me +0.5 on taking this change, but it's only less than +1 because I'm already working on the next major iteration and so *I* don't need them.
msg222436 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-07-07 06:56
All fine with me. As for buildmsi.bat: note that we used to have a "daily msi" builder that was using the script. It took too much effort to keep it running.
msg222469 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-07-07 15:55
With a vote and a half in favor, I'll go ahead with this probably later today.  There is one change needed to the `clean` script; the `hg purge` call should ignore Lib/test/data to avoid downloading the codecmaps test files every time.  I may backport to 3.4 and/or 2.7 later on, but will not do so until default has been happy with it for a while.

I'll leave buildmsi.bat in place for now; I'm interested in trying to resurrect the daily MSI builder some day.  Possibly not before Steve converts us to WIX :)
msg222496 - (view) Author: Roundup Robot (python-dev) Date: 2014-07-07 18:41
New changeset c3022e2606df by Zachary Ware in branch 'default':
Issue #21907: Improved the batch scripts provided for building Python.
http://hg.python.org/cpython/rev/c3022e2606df
msg222497 - (view) Author: Roundup Robot (python-dev) Date: 2014-07-07 18:45
New changeset ed3fa77804f8 by Zachary Ware in branch 'default':
Issue #21907: Update the instructions for building on Windows
http://hg.python.org/devguide/rev/ed3fa77804f8
msg222510 - (view) Author: Roundup Robot (python-dev) Date: 2014-07-07 20:11
New changeset f0a5be139717 by Zachary Ware in branch 'default':
Issue #21907: Exit with the correct return code
http://hg.python.org/cpython/rev/f0a5be139717
msg222564 - (view) Author: Roundup Robot (python-dev) Date: 2014-07-08 14:42
New changeset 06589e81fd56 by Zachary Ware in branch 'default':
Issue #21907: Make the buildbot clean script always return 0.
http://hg.python.org/cpython/rev/06589e81fd56
msg222566 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-07-08 14:51
The XP buildbot seems to have choked on this change, I suspect because of the 'clean' script exiting with a non-zero error code.  I'm not certain why it can't recover, though.  David, can you tell what's going on with it?
msg222574 - (view) Author: David Bolen (db3l) Date: 2014-07-08 18:35
Interesting - it's got a "Visual Studio Just-In-Time Debugger" dialog on the screen for an unhandled win32 exception in the compiler (cl.exe).

That's a dialog my pop-up AutoIt script wasn't expecting, so I've added it to the other (RTL error) checks, and it'll clear automatically going forward.  But I'm not sure what triggered the error.

-- David
msg222575 - (view) Author: David Bolen (db3l) Date: 2014-07-08 18:56
Ok, the last spurt of exceptions on the XP buildbot in the 3.x branch were all related to the fact that somehow the .hg folder in the 3.x branch build tree was missing.  The rest of the build files seemed present.  I've removed the build tree completely to let a new checkout take place and restarted the most recent build.

There does also appear to have been a test file beneath the build tree that the new clean script couldn't remove (build/test_python_580/@test_580_tmp-?L???.py).  I was able to remove it myself interactively from cmd.exe, but I've had problems like this myself when cleaning up temp files on the buildbots sometimes, and believe it has something to do with filename character encodings, depending on the tool being used to reference the file.  That might explain the non-zero error on XP though, and I agree that making sure such failures don't propagate seems right.

-- David
msg222592 - (view) Author: David Bolen (db3l) Date: 2014-07-08 23:49
I'm seeing an apparent side-effect of the new clean script (or it seems to correlate).  Since it started running, the Windows buildbots (both 7 and XP) seem to always perform a full clone of the master repository for each build rather than just a pull/update.  That's a noticeable performance difference (15-20 minutes per clone rather than the more typical 1-2 minutes for pull/update) and seems like it should be unnecessary.

I'm not quite sure how the buildbot task makes the decision between the two approaches, but could the new clean be removing a file that's relevant to how the decision is made?
msg222621 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-07-09 15:29
Thanks for getting the XP bot running again, it looks like it's now having issues with how the tests are now run (some issue with how sys.stdin is set up in a subprocess); I'm not sure if that should be fixed by reverting back to not using run_tests.py, or fixing stdin in a subprocess on XP (which is now officially unsupported by 3.5 anyway...).

As for the fact that your bots are now doing full clones instead of pulls is disturbing and must be fixed, but I'm not sure how that happened.  The AMD64 Windows 7 bot is still doing pulls, so the new clean script shouldn't be a problem.  Do you have any insight into what happened on the x86 Win7 bot's build 8503[1]?  That should have been the first build with this issue's changes, but it didn't even try to update at all.

Build 10744[2] on the XP bot seems to have tried to clobber the build directory on update after a successful clean on the previous build, which doesn't make sense to me either (though the 'test' step did time out, which might play into it).

[1] http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/8503
[2] http://buildbot.python.org/all/builders/x86%20XP-4%203.x/builds/10744
msg223351 - (view) Author: Ingolf Becker (ibecker) * Date: 2014-07-17 18:05
A couple of issues: 
PCbuild/build_pgo.bat: 
- there a quite a few incorrect "" which break the call to rmpyc.py
- some paths were not cwd-agnostic
- all paths should now be whitespace aware
- The PGI python.exe requires the pgodb100.dll to be in the PATH when executed. vcvarsall.bat sets the path correctly.

PCbuild/build_pgo.bat:
- incorrect parameter for vcvarsall.bat for x64: x86_amd64 == x86!!

With these changes build_pgo.bat works.
msg223355 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-07-17 19:06
As you can probably tell, I wasn't on a machine where I could test build_pgo.bat before I committed changes to it (oops).

It looks like your patch is reversed, would you mind fixing it?  I can tell you it will need to change, though; build.bat needs to use 'x86_amd64' for the argument to vcvarsall.bat for building 64-bit to support cross-compiling on 32-bit Windows.  It may work out easiest for build_pgo.bat to just not use build.bat.
msg223363 - (view) Author: Ingolf Becker (ibecker) * Date: 2014-07-17 19:47
This should now be the correct way.
I did not need to touch build.bat indeed. I got confused by all the different combinations! I have removed the calls from build_pgo.bat to build.bat and changed the it to a native x64 build - the PGO requires the code to be run natively, so cross compiling makes no sense.
There was also one \ too many in the pginstrument.props file, which caused a path to end on a \\, which broke the build process.
msg223473 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-19 20:24
Does this render the patch against build_pgo.bat on #17667 obsolete?
msg224467 - (view) Author: Roundup Robot (python-dev) Date: 2014-08-01 04:49
New changeset 60c61ea64021 by Zachary Ware in branch 'default':
Issue #21907: Further improvments to build_pgo.bat.  Patch by Ingolf Becker.
http://hg.python.org/cpython/rev/60c61ea64021
msg224468 - (view) Author: Roundup Robot (python-dev) Date: 2014-08-01 04:59
New changeset 4c1d543135ef by Zachary Ware in branch 'default':
Issue #21907: Avoid using double quotes to check argument values.
http://hg.python.org/cpython/rev/4c1d543135ef
msg226387 - (view) Author: David Bolen (db3l) Date: 2014-09-04 23:01
While troubleshooting an issue with test_distutils consistently failing on my XP buildbot, I narrowed it down to the test.bat change to use run_tests.py.  I don't yet know fully what's happening, but after replacing the new test.bat with the older version (that calls rt.bat) everything works fine.

Doing so also appears to clean up the stdin encoding problem (which was causing a consistent failure in test_capi).

As suggested by Zachary in msg222621, and given the status of XP going forward, I think it may be best, at least for now, to revert back to the traditional test.bat/rt.bat combination under XP, which has been stable for a long time.  At least until some further work can be done to discover what's happening with the new approach, assuming it's desired at some point under XP.

Attached is a simple test.bat update that reverts to the older rt.bat approach but only under XP.
msg229060 - (view) Author: Roundup Robot (python-dev) Date: 2014-10-11 06:44
New changeset f592a4073672 by Zachary Ware in branch 'default':
Issue #21907: Try to make the pre-Vista Windows buildbots happier.
https://hg.python.org/cpython/rev/f592a4073672
msg229218 - (view) Author: Roundup Robot (python-dev) Date: 2014-10-13 05:18
New changeset 1d5485471457 by Zachary Ware in branch 'default':
Issue #21907: Partially revert changes to Windows buildbot scripts.
https://hg.python.org/cpython/rev/1d5485471457
msg229252 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-10-13 16:40
After the last round of changes, the buildbots appear to be mostly happy.  If anybody else wants to backport the changes, I'd be happy to review and commit, but I'll leave the backporting itself to whoever wants to do it.  In the meantime, closing the issue.
msg229491 - (view) Author: David Bolen (db3l) Date: 2014-10-15 18:40
Just thought I'd add a note here that after the most recent changes, my buildbots also appear to be back to quicker hg pulls rather than clones at the start of the process (see msg222592).  Still not sure why that behavior changed, but we're back to the previous behavior.
History
Date User Action Args
2014-10-15 18:40:00db3lsetmessages: + msg229491
2014-10-13 16:40:48zach.waresetstatus: open -> closed
resolution: fixed
messages: + msg229252

stage: patch review -> resolved
2014-10-13 05:18:58python-devsetmessages: + msg229218
2014-10-11 06:44:57python-devsetmessages: + msg229060
2014-09-04 23:01:51db3lsetfiles: + test.bat

messages: + msg226387
2014-08-01 04:59:54python-devsetmessages: + msg224468
2014-08-01 04:49:03python-devsetmessages: + msg224467
2014-07-19 20:24:40BreamoreBoysetnosy: + BreamoreBoy
messages: + msg223473
2014-07-17 19:47:37ibeckersetfiles: + fix_pgo_vcvarsall_rightway.diff

messages: + msg223363
2014-07-17 19:06:38zach.waresetmessages: + msg223355
2014-07-17 18:05:57ibeckersetfiles: + fix_pgo_vcvarsall.diff
nosy: + ibecker
messages: + msg223351

2014-07-09 15:29:17zach.waresetmessages: + msg222621
2014-07-08 23:49:28db3lsetmessages: + msg222592
2014-07-08 18:56:15db3lsetmessages: + msg222575
2014-07-08 18:35:43db3lsetmessages: + msg222574
2014-07-08 14:51:28zach.waresetnosy: + db3l
messages: + msg222566
2014-07-08 14:42:29python-devsetmessages: + msg222564
2014-07-07 20:11:23python-devsetmessages: + msg222510
2014-07-07 19:14:23zach.warelinkissue9822 superseder
2014-07-07 18:54:23zach.warelinkissue9973 superseder
2014-07-07 18:45:44python-devsetmessages: + msg222497
2014-07-07 18:41:40python-devsetnosy: + python-dev
messages: + msg222496
2014-07-07 15:55:13zach.waresetmessages: + msg222469
2014-07-07 06:56:52loewissetmessages: + msg222436
2014-07-04 15:52:17jklothsetnosy: + jkloth
2014-07-02 20:53:09steve.dowersetmessages: + msg222123
2014-07-02 19:22:56zach.waresetfiles: + devguide_update.diff

messages: + msg222119
2014-07-02 19:16:42zach.warecreate