classification
Title: Python 3.6 compilation fails on AppVeyor: libeay.lib was created with an older compiler
Type: Stage: resolved
Components: Build, Windows Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: paul.moore, steve.dower, tim.golden, vstinner, xtreak, zach.ware
Priority: normal Keywords: patch

Created on 2018-09-04 09:31 by vstinner, last changed 2018-09-21 22:13 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9066 merged zach.ware, 2018-09-05 13:51
Messages (14)
msg324576 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-04 09:31
Example on my PR:
https://github.com/python/cpython/pull/9057

https://ci.appveyor.com/project/python/cpython/build/3.6build21480/job/lge7r4qknx0t0tlv

LINK : fatal error C1047: The object or library file 'C:\projects\cpython\PCBuild\win32\libeay.lib' was created with an older compiler than other objects; rebuild old objects and libraries [C:\projects\cpython\PCbuild\_hashlib.vcxproj]
LINK : fatal error LNK1257: code generation failed [C:\projects\cpython\PCbuild\_hashlib.vcxproj]
LINK : fatal error C1047: The object or library file 'C:\projects\cpython\PCBuild\win32\libeay.lib' was created with an older compiler than other objects; rebuild old objects and libraries [C:\projects\cpython\PCbuild\_ssl.vcxproj]
LINK : fatal error LNK1257: code generation failed [C:\projects\cpython\PCbuild\_ssl.vcxproj]
Command exited with code 1
msg324593 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-09-04 16:07
Works on VSTS build. I suspect AppVeyor is caching files between builds, but I'm not familiar enough with the configuration to fix that.
msg324613 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-09-05 08:30
If it's due to cache then there is a REST API to clear the cache manually and someone with access to Appveyor credentials can give it a try to clear the cache and see if it's fixed.

Ref : https://github.com/appveyor/ci/issues/985

Log in and execute the below from the console as per https://github.com/appveyor/ci/issues/985#issuecomment-279199811

$.ajax({
    url: 'https://ci.appveyor.com/api/projects/<username>/<project>/buildcache',
    type: 'DELETE'})

A workaround : https://help.appveyor.com/discussions/questions/1310-delete-cache


Thanks
msg324616 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-05 09:32
REST API for the AppVeyor build cache:
https://www.appveyor.com/docs/build-cache/#rest-api

Authentication of the REST API:
https://www.appveyor.com/docs/api/#authentication

I tried:

$ curl -H "Authorization: Bearer <MY_PRIVATE_TOKEN>" -X "DELETE" https://ci.appveyor.com/api/projects/python/cpython/buildcache; echo

{"message":"You do not have required permissions to perform this action."}

Oh. I'm not allowed to clear the build cache :-(

I'm logged as "python" and my role is "Super User". If a Super User is not allowed to clear the cache, who is allowed to do that? Maybe my URL is wrong? The documentation says:

DELETE https://ci.appveyor.com/api/projects/{accountName}/{projectSlug}/buildcache
msg324617 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2018-09-05 09:46
I don't see an appveyor.yml file in the CPython repository. How are
the appveyor builds configured? I was going to take a look, but need
to see the config :-(
On Wed, 5 Sep 2018 at 10:32, STINNER Victor <report@bugs.python.org> wrote:
>
>
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> REST API for the AppVeyor build cache:
> https://www.appveyor.com/docs/build-cache/#rest-api
>
> Authentication of the REST API:
> https://www.appveyor.com/docs/api/#authentication
>
> I tried:
>
> $ curl -H "Authorization: Bearer <MY_PRIVATE_TOKEN>" -X "DELETE" https://ci.appveyor.com/api/projects/python/cpython/buildcache; echo
>
> {"message":"You do not have required permissions to perform this action."}
>
> Oh. I'm not allowed to clear the build cache :-(
>
> I'm logged as "python" and my role is "Super User". If a Super User is not allowed to clear the cache, who is allowed to do that? Maybe my URL is wrong? The documentation says:
>
> DELETE https://ci.appveyor.com/api/projects/{accountName}/{projectSlug}/buildcache
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue34575>
> _______________________________________
msg324618 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-05 09:48
Ah! If I user my personal account which has a different token, I'm able to list my roles using the REST API.

So it seems that I lack some permissions on the "python" account of AppVeyor. Who owns this account?
msg324619 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-09-05 09:48
Another way will be to invalidate the cache by manually updating cleanup-cache.txt in the end. But this requires a commit that has to be reverted later to enable caching and I don't think it's worthy to pollute git history for this.

https://help.appveyor.com/discussions/questions/1310-delete-cache#comment_36916917


> You can have any file as dependency that will be triggering cache invalidation, even "cleanup-cache.txt" say in the root of your repo:
> cache:
>     - my_cached_folder -> cleanup-cache.txt
> Whenever CRC32 of cleanup-cache.txt is changed the cache is invalidated, i.e. not restored in the beginning of the build.

@paul.moore It's here : https://github.com/python/cpython/blob/master/.github/appveyor.yml


Thanks
msg324621 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-09-05 10:00
One another way will be to set environment variable to skip the cache restore and trigger a build to see if cache is the actual problem. I don't know how configurable environment variables are from the UI to trigger a specific build with the set of environment variable values.

https://www.appveyor.com/docs/build-cache/#skipping-cache-operations-for-specific-build

> Skipping cache operations for specific build

> You can skip cache restore or save stages with the following tweak environment variables:

> APPVEYOR_CACHE_SKIP_RESTORE - set to true to disable cache restore
> APPVEYOR_CACHE_SKIP_SAVE - set to true to disable cache update


Thanks
msg324622 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-05 10:03
> APPVEYOR_CACHE_SKIP_RESTORE - set to true to disable cache restore
> APPVEYOR_CACHE_SKIP_SAVE - set to true to disable cache update

I see these as well, but it would only be a temporary solution and may make build much slower, whereas AppVeyor is already a bottleneck in our workflow. AppVeyor only give us two jobs in parallel which take 10-30 min, whereas it's common that we have much more new pull requests per hour.
msg324625 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-09-05 10:21
Ah ok, I thought disabling the cache with APPVEYOR_CACHE_SKIP_RESTORE as true will trigger a successful build and in the end it will store the new set of artifacts from the recent compiler to the cache and then we can toggle the environment variable (APPVEYOR_CACHE_SKIP_RESTORE) so that for the next build artifacts built from old compiler are replaced with the newer ones resembling a cache clear. Sorry I might be wrong here and it's a workaround that I thought will work as in the other CI systems. I will be happy if it's solved through API then with the right credentials.

Thanks
msg324626 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2018-09-05 10:30
One thought - appveyor.yml says that the cache of externals depends on PCBuild. Could it be that appveyor is caching PCBuild to verify whether it's changed (and hence whether to use the cached externals)? I know that sounds bizarre, but I'm not entirely sure why externals should be dependent on the *whole* of PCBuild anyway - maybe we should change it to depend only on PCBuild/get_externals.bat?

Clutching at straws here, there's definitely nothing obvious I can see though.
msg324633 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-05 14:09
Discussion on python-dev:
https://mail.python.org/pipermail/python-dev/2018-September/155075.html
msg324635 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2018-09-05 14:42
New changeset 635461fca5e90c6e091f1e5b46adafc0d28bf0e2 by Zachary Ware in branch '3.6':
[3.6] bpo-34575: Build with only VS2015 on AppVeyor (GH-9066)
https://github.com/python/cpython/commit/635461fca5e90c6e091f1e5b46adafc0d28bf0e2
msg326049 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-21 22:13
It seems like the bug has been fixed. Thanks Zachary Ware for the fix, and thanks everybody for helping to fix this cache issue ;-)
History
Date User Action Args
2018-09-21 22:13:34vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg326049

stage: patch review -> resolved
2018-09-05 14:42:56zach.waresetmessages: + msg324635
2018-09-05 14:09:11vstinnersetmessages: + msg324633
2018-09-05 13:51:03zach.waresetkeywords: + patch
stage: patch review
pull_requests: + pull_request8530
2018-09-05 10:30:08paul.mooresetmessages: + msg324626
2018-09-05 10:21:08xtreaksetmessages: + msg324625
2018-09-05 10:03:41vstinnersetmessages: + msg324622
2018-09-05 10:00:48xtreaksetmessages: + msg324621
2018-09-05 09:48:09xtreaksetmessages: + msg324619
2018-09-05 09:48:04vstinnersetmessages: + msg324618
2018-09-05 09:46:53paul.mooresetmessages: + msg324617
2018-09-05 09:32:07vstinnersetmessages: + msg324616
2018-09-05 08:30:40xtreaksetnosy: + xtreak
messages: + msg324613
2018-09-04 16:07:09steve.dowersetmessages: + msg324593
2018-09-04 09:31:27vstinnercreate