classification
Title: ssl build under Windows always rebuilds OpenSSL
Type: behavior Stage: resolved
Components: Build, Windows Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brian.curtin, eric.araujo, loewis, ocean-city, pitrou, tim.golden
Priority: low Keywords: patch

Created on 2010-08-09 23:04 by pitrou, last changed 2011-08-13 16:44 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
py3k_supress_rebuild_of_openssl.patch ocean-city, 2010-09-18 01:21
py3k_openssl.patch ocean-city, 2010-09-20 04:55
py3k_openssl_v2.patch ocean-city, 2010-09-21 15:21
py3k_better_build_ssl.zip ocean-city, 2010-09-24 09:26
Messages (20)
msg113497 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-09 23:04
The build steps for _ssl and _hashlib under Windows always rebuild the same intermediate files, even if no change was recorded since the last build. This makes building the whole project longer than it should be.

Here is the (trimmed) log:

------ Début de la génération : Projet : _ssl, Configuration : Debug Win32 ------
Exécution d'un événement avant génération...
Found a working perl at 'C:\Perl\bin\perl.exe'
Found an SSL directory at '..\..\openssl-1.0.0a'
Executing ssl makefiles: nmake /nologo -f "ms\nt.mak"
Building OpenSSL
 copy ".\crypto\buildinf.h" "tmp32\buildinf.h"
        1 fichier(s) copié(s).
 copy ".\crypto\opensslconf.h" "inc32\openssl\opensslconf.h"
        1 fichier(s) copié(s).
 nasmw -f win32 -o tmp32\x86cpuid.obj tmp32\x86cpuid.asm
 nasmw -f win32 -o tmp32\md5-586.obj tmp32\md5-586.asm
[... snip many asm files ...]
 nasmw -f win32 -o tmp32\wp-mmx.obj tmp32\wp-mmx.asm
 lib /nologo /out:out32\libeay32.lib @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nm92.tmp
 link /nologo /subsystem:console /opt:ref /debug /out:out32\md4test.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nm97.tmp
 IF EXIST out32\md4test.exe.manifest mt -nologo -manifest out32\md4test.exe.manifest -outputresource:out32\md4test.exe;1
 link /nologo /subsystem:console /opt:ref /debug /out:out32\md5test.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nm99.tmp
 IF EXIST out32\md5test.exe.manifest mt -nologo -manifest out32\md5test.exe.manifest -outputresource:out32\md5test.exe;1
 link /nologo /subsystem:console /opt:ref /debug /out:out32\shatest.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nm9B.tmp
[... snip many exe files ...]
 IF EXIST out32\openssl.exe.manifest mt -nologo -manifest out32\openssl.exe.manifest -outputresource:out32\openssl.exe;1
[44763 refs]
Compilation en cours...
_ssl.c
..\Modules\_ssl.c(583) : warning C4090: '=' : qualificateurs 'const' différents
..\Modules\_ssl.c(930) : warning C4090: '=' : qualificateurs 'const' différents
Édition des liens en cours...
   Création de la bibliothèque Z:\py3k\__svn__\PCbuild\\_ssl_d.lib et de l'objet Z:\py3k\__svn__\PCbuild\\_ssl_d.exp
Le journal de génération a été enregistré à l'emplacement "file://Z:\py3k\__svn__\PCbuild\Win32-temp-Debug\_ssl\BuildLog.htm"
_ssl - 0 erreur(s), 2 avertissement(s)
msg116744 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-18 01:21
I hope attached patch will fix this. I used shutil.copy2
to retain modified time. (Actually, I tested on VS8.0 with
equivalent change, I didn't test this file directly)
msg116745 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-18 01:25
Here is the result of second run.

------ ビルド開始: プロジェクト: _ssl, 構成: Debug Win32 ------
ビルド前のイベントを実行しています...
Can not find a suitable PERL:
 NO perl interpreters were found on this machine at all!
 Please install ActivePerl and ensure it appears on your path
No Perl installation was found. Existing Makefiles are used.
Found a working perl at 'None'
Found an SSL directory at '..\..\..\openssl-1.0.0a'
Executing ssl makefiles: nmake /nologo -f "ms\nt.mak"
Building OpenSSL
[40162 refs]
コンパイルしています...
_ssl.c
..\..\Modules\_ssl.c(583) : warning C4090: '=' : 異なる 'const' 修飾子です。
..\..\Modules\_ssl.c(930) : warning C4090: '=' : 異なる 'const' 修飾子です。
リンクしています...
   ライブラリ E:\python-dev\py3k\PC\VS8.0\\_ssl_d.lib とオブジェクト E:\python-dev\py3k\PC\VS8.0\\_ssl_d.exp を作成中
ビルドログは "file://E:\python-dev\py3k\PC\VS8.0\Win32-temp-Debug\_ssl\BuildLog.htm" に保存されました。
_ssl - エラー 0、警告 2
========== ビルド: 1 正常終了、0 失敗、7 更新、0 スキップ ==========
msg116808 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-18 16:10
The patch works here (VS 2008).
msg116852 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-19 09:24
Thanks. I've committed in r84902(py3k).
msg116908 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-20 04:39
I'll revert my commits because I noticed after perl source
like openssl-1.0.0a/crypto/x86cpuid.pl was modified,
unnecessary rebuild happened again.

Appropriate fix will be "only copies *.asm if there are not
in tmp32/" as comments in PCBuild/build_ssl.py saids.(Probably
this copying is not needed only when Perl is available,
I suppose)
msg116909 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-20 04:42
- only when
+ when

P.S.
I cannot figure out how to solve this error.
http://www.python.org/dev//buildbot/builders/x86%20Windows7%203.x/builds/1593/steps/compile/logs/stdio
msg116910 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-20 04:55
This is the patch I'm thinking of. I confirmed this works
on VS8.0. (After replaced "..\\.." with "..\\..\\..")

E:\PYTHON~1\py3k\PC\VS8.0>python_d.exe build_ssl.py Release Win32 -a
Found a working perl at 'C:\Perl\bin\perl.exe'
Found an SSL directory at '..\..\..\openssl-1.0.0a'
Executing ssl makefiles: nmake /nologo -f "ms\nt.mak"
Building OpenSSL
        copy ".\crypto\buildinf.h" "tmp32\buildinf.h"
        Copied one file.
        copy ".\crypto\opensslconf.h" "inc32\openssl\opensslconf.h"
        Copied one file.
[45178 refs]

By this change, two header files are still copied. I feel
it's strange header files are modified but none of sources
are not re-built.
msg116971 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-20 20:58
> This is the patch I'm thinking of. I confirmed this works
> on VS8.0. (After replaced "..\\.." with "..\\..\\..")

Works here too (VS 2008)!
msg117064 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-21 15:17
Thank you for the test.

Well, I noticed py3k_openssl.patch can suppress rebuild
of OpenSSL nicely, but from IDE, _ssl and _hashlib
are always rebuilt. It doesn't take so much time though.

With py3k_openssl_v2.patch, we can supress OpenSSL and
python project rebuild nicely, but we cannot rebuild
_ssl and _hashlib from IDE after OpenSSL sources are
modified. We need to call "python_d build_ssl.py ..."
manually.

I cannot find the best way for this issue. :-(
msg117065 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-21 15:21
> Well, I noticed py3k_openssl.patch can suppress rebuild
> of OpenSSL nicely, but from IDE, _ssl and _hashlib
> are always rebuilt. It doesn't take so much time though.
> 
> With py3k_openssl_v2.patch, we can supress OpenSSL and
> python project rebuild nicely, but we cannot rebuild
> _ssl and _hashlib from IDE after OpenSSL sources are
> modified. We need to call "python_d build_ssl.py ..."
> manually.

I think it's better to rebuild everytime than have to invoke a console
command manually.
Also, as you say, building _ssl and hashlib is fast. The problem with
rebuilding OpenSSL is that it's very slow.
msg117066 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-21 15:23
I want to commit py3k_openssl.patch for now, because
it is much better than before.
msg117068 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-21 15:24
Oops, I missed your post. Thank you, I'll commit like that.
msg117093 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-21 18:35
I committed py3k_openssl.patch in r84957(py3k).
I won't merge into release27-maint nor release31-maint
because they are built against openssl-0.9.x and
don't have *.asm copy code.
msg117186 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-23 13:48
Hello. Probably I found the better solution for this issue.

* build_ssl.py always runs before _ssl.vcproj or _hashlib.vcproj
  is built. (If entire solution is built, run only onece)

* When OpenSSL sources are modified, because build_ssl.py always runs,
  correctly OpenSSL are rebuilt.

* Module/_ssl.c or Module/_hashlib.c are rebuilt if they or
  include files refereed by them are not modified.

Can you try this? (Only Debug|Win32 and Release|Win32 are implemented)
msg117189 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-23 13:54
- Module/_ssl.c or Module/_hashlib.c are rebuilt
+ Module/_ssl.c or Module/_hashlib.c won't be rebuilt
msg117262 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-24 06:47
Sorry, previous one contained experimental code for another issue.
I've attached correct one.
msg117276 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-24 12:23
I'm not really able to review this kind of things. Please commit if you think it is ok.
msg117306 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-24 16:38
Thank you, I've committed in r84993(py3k)
I'll close this issue after merging to other branches.
msg141829 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-09 15:49
Can this be backported, or just closed?
History
Date User Action Args
2011-08-13 16:44:09pitrousetstatus: open -> closed
resolution: fixed
stage: resolved
2011-08-09 15:49:13eric.araujosetnosy: + eric.araujo
messages: + msg141829
2010-09-24 16:38:20ocean-citysetmessages: + msg117306
2010-09-24 12:23:03pitrousetmessages: + msg117276
2010-09-24 09:26:42ocean-citysetfiles: + py3k_better_build_ssl.zip
2010-09-24 09:26:21ocean-citysetfiles: - py3k_better_build_ssl.zip
2010-09-24 09:26:11ocean-citysetfiles: - py3k_better_build_ssl.zip
2010-09-24 06:47:13ocean-citysetfiles: + py3k_better_build_ssl.zip

messages: + msg117262
2010-09-23 13:54:07ocean-citysetmessages: + msg117189
2010-09-23 13:48:32ocean-citysetstatus: closed -> open
resolution: fixed -> (no value)
versions: + Python 3.1, Python 2.7
2010-09-23 13:48:06ocean-citysetfiles: + py3k_better_build_ssl.zip

messages: + msg117186
2010-09-22 23:37:38ocean-citysetstatus: open -> closed
resolution: fixed
versions: - Python 3.1, Python 2.7
2010-09-21 18:35:17ocean-citysetmessages: + msg117093
2010-09-21 15:24:38ocean-citysetmessages: + msg117068
2010-09-21 15:23:27ocean-citysetmessages: + msg117066
2010-09-21 15:21:51ocean-citysetfiles: + py3k_openssl_v2.patch
2010-09-21 15:21:29pitrousetmessages: + msg117065
2010-09-21 15:19:16ocean-citysetfiles: - py3k_openssl_v2.patch
2010-09-21 15:17:10ocean-citysetfiles: + py3k_openssl_v2.patch

messages: + msg117064
2010-09-20 20:58:04pitrousetmessages: + msg116971
2010-09-20 04:57:02ocean-citysetfiles: + py3k_openssl.patch

messages: + msg116910
2010-09-20 04:42:22ocean-citysetmessages: + msg116909
2010-09-20 04:39:09ocean-citysetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg116908
2010-09-19 09:24:50ocean-citysetstatus: open -> closed
resolution: fixed
messages: + msg116852

versions: + Python 3.1, Python 2.7
2010-09-18 16:10:36pitrousetmessages: + msg116808
2010-09-18 01:25:23ocean-citysetmessages: + msg116745
2010-09-18 01:21:16ocean-citysetfiles: + py3k_supress_rebuild_of_openssl.patch

nosy: + ocean-city
messages: + msg116744

keywords: + patch
2010-08-09 23:04:50pitroucreate