classification
Title: Enable manylinux1 builds on Pipelines for CI testing
Type: Stage: needs patch
Components: Cross-Build Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, barry, steve.dower, terry.reedy, zach.ware
Priority: normal Keywords: patch, patch, patch

Created on 2019-01-08 04:19 by steve.dower, last changed 2019-01-22 20:33 by steve.dower.

Pull Requests
URL Status Linked Edit
PR 11493 merged steve.dower, 2019-01-10 00:39
PR 11493 merged steve.dower, 2019-01-10 00:39
PR 11493 merged steve.dower, 2019-01-10 00:39
PR 11651 merged steve.dower, 2019-01-22 19:25
PR 11651 merged steve.dower, 2019-01-22 19:25
PR 11651 merged steve.dower, 2019-01-22 19:25
Messages (6)
msg333209 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-08 04:19
Azure Pipelines can now support container jobs: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=vsts&tabs=yaml

I experimented with enabling a manylinux1 build a while back, which should now be able to use identical steps to the POSIX build. With the new syntax, we can enable CI (and perhaps PR?) builds using the snippet below:

- job: ManyLinux1_CI_Tests
  displayName: ManyLinux1 CI Tests
  dependsOn: Prebuild
  condition: |
    and(
        and(
            succeeded(),
            eq(variables['manylinux'], 'true')
        ),
        eq(dependencies.Prebuild.outputs['tests.run'], 'true')
    )

  resources:
    containers:
    - container: manylinux1
      image: dockcross:manylinux-x64

  pool:
    vmImage: ubuntu-16.04

  container: manylinux1

  variables:
    testRunTitle: '$(build.sourceBranchName)-manylinux1'
    testRunPlatform: manylinux1

  steps:
  - template: ./posix-steps.yml


I don't have time right now to test this change, but someone else might. It's certainly going to be easier for someone to test it by adding this to the PR build first (or set up a build on your own Pipelines instance).

Maybe there are other more relevant containers we should be testing in?
msg333891 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-17 18:27
I have made the changes I suggested (though correctly...), but ultimately we need to create our own Docker image suitable for running these tests.

So for now, I'm proposing in my PR to make most of the change, as well as a few other Pipelines/test-related improvements, but to leave this open in case someone wants to come in later with a suitable image. At that point, the only change necessary to enable the tests will be to add a "posix_deps_yum.sh" script, update the image name/tag and change the manylinux variable to 'true'.

PR 11493 also fixes a missing LICENSE.txt file in the app store package, which was causing an idlelib test to fail as the fallback text only has one line. *Way* too obscure a failure for my liking, but at least we had a test there, so thanks, Terry :)
msg333904 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-01-17 22:30
I believe it was Louie Lu's idea, #30290, to check more than one line.
I am glad it helped.
msg334227 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-22 18:50
New changeset 28f6cb34f602b9796987904a607dceaf2e4a9e78 by Steve Dower in branch 'master':
bpo-35683: Improve Azure Pipelines steps (GH-11493)
https://github.com/python/cpython/commit/28f6cb34f602b9796987904a607dceaf2e4a9e78
msg334235 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-22 20:31
New changeset 128efcade63480b5860a6d045a41ba4abf5eea2f by Steve Dower in branch '3.7':
bpo-35683: Improve Azure Pipelines steps (GH-11493)
https://github.com/python/cpython/commit/128efcade63480b5860a6d045a41ba4abf5eea2f
msg334236 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-22 20:33
As mentioned above, those changes are other improvements that were worth taking, and about half of the required manylinux1 changes. But we probably need to maintain our own manylinux image for building/running CPython tests, if we want to do it. The existing images are all designed for having CPython already present to build a range of binaries, which is not at all our use case.
History
Date User Action Args
2019-01-22 20:33:21steve.dowersetkeywords: patch, patch, patch

messages: + msg334236
stage: patch review -> needs patch
2019-01-22 20:31:34steve.dowersetmessages: + msg334235
2019-01-22 19:26:06steve.dowersetpull_requests: + pull_request11440
2019-01-22 19:25:56steve.dowersetpull_requests: + pull_request11439
2019-01-22 19:25:45steve.dowersetpull_requests: + pull_request11438
2019-01-22 18:50:04steve.dowersetmessages: + msg334227
2019-01-17 22:30:31terry.reedysetkeywords: patch, patch, patch

messages: + msg333904
2019-01-17 18:27:19steve.dowersetkeywords: patch, patch, patch
nosy: + terry.reedy
messages: + msg333891

2019-01-10 00:40:05steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request11012
2019-01-10 00:39:57steve.dowersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11011
2019-01-10 00:39:50steve.dowersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11010
2019-01-08 19:05:03brett.cannonsettitle: Enable manylinux1 builds on Pipelines -> Enable manylinux1 builds on Pipelines for CI testing
2019-01-08 04:19:11steve.dowercreate