classification
Title: [2.7] OpenSSL compilation fails on AMD64 Windows7 SP1 VS9.0 2.7
Type: Stage: resolved
Components: Build, SSL, Windows Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, jkloth, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2017-05-15 08:46 by vstinner, last changed 2017-06-28 00:49 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1805 merged jkloth, 2017-05-25 00:28
Messages (22)
msg293687 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-15 08:46
See http://bugs.python.org/issue30350#msg293663

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%20VS9.0%202.7/builds/83/steps/compile/logs/stdio

Build started: Project: _ssl, Configuration: Debug|x64
...
No Perl installation was found. Existing Makefiles are used.
...
IOError: [Errno 2] No such file or directory: 'crypto\\buildinf_amd64.h'
msg293735 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-05-15 19:47
Perl should not be required - we should be checking these files in to our clone on svn.p.o for use when building. (Perl is only required by whoever creates the source tree, normally Zach.)

Perhaps it got missed in the latest update?
msg293737 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-05-15 19:50
Perl is not required when using the PCbuild/ build files on any branch.  It is currently required to use the PC/VS9.0 build files, at least for 64-bit builds, because I was not able to make it work (and not break the new system) in the time I had to get it done.
msg293781 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-16 20:46
I don't really care if Jeremy installs Perl on the buildbot or if someone succeed to fix the dependency on the svn repository. I only care of seeing a green buildbot ;-)
msg293784 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-05-16 20:49
The simple fix from my point of view is for Jeremy to install Perl :).  But since he also mentioned having a patch ready in bpo-30350, I'd like to see it before declaring that installing Perl is what should be done.
msg294419 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-05-25 00:42
Submitted PR-1805 that should restore perl-less building.

Also some open issues as noted in the PR.
msg294422 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-05-25 01:11
I tried out PR1805 with Perl unavailable, and it seemed to build fine with `PC\VS9.0\build.bat -e -d -p x64`.  However, trying `PC\VS9.0\amd64\python_d.exe -m test.ssltests` segfaults.  Can you reproduce?
msg294425 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-05-25 02:50
On my VM, ssltests pass, but both the Perl and non-Perl segfault in
test_ssl (and many others).  My VM has just VS2008 installed, does
VS2010 come with an updated SDK?
msg294437 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-05-25 06:16
OK, testing with the 7.1 SDK installed changes the crashes around, but something is really amiss.  I'd like to do more testing, but it will need to wait until morning here.
msg294479 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-05-25 14:13
Ok, sounds good.  The two points you raised on the PR also sound fine, as far as I can tell.
msg294508 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-05-25 19:15
Updated PR.  It seems that in my testing back and forth, some build artifacts were affecting my outcomes.  That's what I get for cutting corners...

Per Zach's comment, I've changed to VS project files to use the prepare_ssl.py include directories (include{suffix} vs inc{suffix}).  This simplifies build_ssl.py further.

I have, however, left the "fix_buildinf()" code in place since it requires an update to the checked-in external OpenSSL before it can be removed.
msg295311 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-06-06 23:51
Just a status check, as I would like to see my buildbot go green again.  (This along with the PR on issue30339).
msg295866 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-13 08:26
Any progress on this issue? The following buildbot still fails to build Python 2.7:
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%20VS9.0%202.7
msg296314 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-06-19 11:45
This is ready to be merged, pending any comments from Zach.
msg296488 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-20 20:53
New changeset ebbccea997e03f60fec8308f2fb9e89a11521a74 by Victor Stinner (Jeremy Kloth) in branch '2.7':
bpo-30368: Update build_ssl.py to restore Perl-less building (#1805)
https://github.com/python/cpython/commit/ebbccea997e03f60fec8308f2fb9e89a11521a74
msg296492 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-20 21:57
I checked AMD64 Windows7 SP1 VS9.0 2.7. Before the commit, the build 129 failed badly when building ssl:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%20VS9.0%202.7/builds/129/steps/compile/logs/stdio

IOError: [Errno 2] No such file or directory: 'crypto\\buildinf_amd64.h'

In the first build with the commit, the compilation of ssl succeeded!

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%20VS9.0%202.7/builds/130

"_ssl - 0 error(s), 375 warning(s)"

... but the compilation step still failed, for other reasons. I see that 2 projects failed:

..\..\Modules\expat\xmlparse.c(13) : fatal error C1083: Cannot open include file: 'sys/time.h': No such file or directory
_elementtree - 1 error(s), 9 warning(s)

and

..\..\Modules\expat\xmlparse.c(13) : fatal error C1083: Cannot open include file: 'sys/time.h': No such file or directory
pyexpat - 1 error(s), 1 warning(s)

It may be related to the recent update of libexpat from 2.1.1 to 2.2.0?
msg296493 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-20 22:03
In a previous build, build 111 (June 6), the compilation of _elementtree succeeded:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%20VS9.0%202.7/builds/111/steps/compile/logs/stdio
_elementtree - 0 error(s), 10 warning(s)

I also see that the commit 0e4571a68a7f48e8469ef05b04ba3463d3fd82c0 which upgraded libexpat introduced "#ifdef WIN32". I see that libexpat 2.2.1, the new version, replace "#ifdef WIN32" with "#ifdef _WIN32". I tested manually and it seems to fix the compilation of xmlparse.c at least! So bpo-30694 which update expat from 2.2.0 to 2.2.1 should fix this buildbot.
msg296495 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-20 22:07
So, I made final tests with the merged commit. I manually uninstalled (Active)Perl. Without the commit, the compilation of ssl failed on my Windows VM since I'm using VS 2008.

(1) using build.bat

git clean -fdx
In the VS2008 shell, type: PC\VS9.0\build.bat -e -d -p x64
=> you get a python_d.exe with a working _ssl module!

(2) using VS GUI

git clean -fdx
PCbuild\get_externals.bat
Open VS 2008, build the project
=> compilation of _ssl fails with a permission error about a temporay object file... I don't understand it
Rebuild the project
=> compilation of _ssl works, python_d.exe has a working _ssl module!

Except of the "temporary" (but reproductible) compilation error in the VS GUI, I confirm that the change fixes this issue!

I prefer to keep the issue open until the buildbot is repaired (expat issue).
msg296539 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-06-21 10:36
Added PR to issue29591 to address issue building Expat.  In short, the project files were not updated along with the copy of Expat.

I cannot reproduce the GUI build problem.  I would need to see the build logs to attempt to resolve.
msg296551 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-21 13:16
> Added PR to issue29591 to address issue building Expat.  In short, the project files were not updated along with the copy of Expat.

I would prefer to first fix the new vulnerabilities, but upgrading expat to 2.2.1, and then review your change.

=> https://github.com/python/cpython/pull/2312
msg296561 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-21 15:21
Oh cool, with my commit 2ada64d2a073f85f135461833952dbe8d656810d "[2.7] bpo-30694: Upgrade Modules/expat/ to libexpat 2.2.1 (#2300)", and obvious Jeremy's previous fix!, the compile step of "AMD64 Windows7 SP1 VS9.0 2.7" succeeded for the first time since a long time!
msg297087 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 00:49
The initial issue ("[2.7] OpenSSL compilation fails on AMD64 Windows7 SP1 VS9.0 2.7") is now fixed, so I close the issue. Thanks for the help everyone who helped me on this one!
History
Date User Action Args
2017-06-28 00:49:25vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg297087

stage: resolved
2017-06-21 15:21:07vstinnersetmessages: + msg296561
2017-06-21 13:16:31vstinnersetmessages: + msg296551
2017-06-21 10:36:52jklothsetmessages: + msg296539
2017-06-20 22:07:40vstinnersetmessages: + msg296495
2017-06-20 22:03:04vstinnersetmessages: + msg296493
2017-06-20 21:57:15vstinnersetmessages: + msg296492
2017-06-20 20:53:43vstinnersetmessages: + msg296488
2017-06-19 11:45:27jklothsetmessages: + msg296314
2017-06-13 08:26:26vstinnersetmessages: + msg295866
2017-06-06 23:51:37jklothsetmessages: + msg295311
2017-05-25 19:15:27jklothsetmessages: + msg294508
2017-05-25 14:13:22zach.waresetmessages: + msg294479
2017-05-25 06:16:54jklothsetmessages: + msg294437
2017-05-25 02:50:17jklothsetmessages: + msg294425
2017-05-25 01:11:05zach.waresetmessages: + msg294422
2017-05-25 00:42:20jklothsetmessages: + msg294419
2017-05-25 00:28:11jklothsetpull_requests: + pull_request1888
2017-05-16 20:49:24zach.waresetassignee: christian.heimes ->
messages: + msg293784
2017-05-16 20:46:38vstinnersetmessages: + msg293781
2017-05-15 19:50:16zach.waresetmessages: + msg293737
2017-05-15 19:47:42steve.dowersetmessages: + msg293735
2017-05-15 08:46:08vstinnercreate