classification
Title: Deprecate sys._mercurial and create sys._git
Type: Stage: patch review
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: benjamin.peterson, brett.cannon, haypo, larry, matrixise, ned.deily, steve.dower
Priority: high Keywords: patch

Created on 2016-07-22 21:43 by brett.cannon, last changed 2017-03-27 16:29 by ned.deily.

Files
File name Uploaded Description Edit
issue27593.diff brett.cannon, 2016-09-10 18:49 review
issue27593-with-indent-3.7.diff matrixise, 2016-10-20 15:00 review
issue27593-with-indent-3.7-2.diff matrixise, 2016-10-20 16:44 review
Pull Requests
URL Status Linked Edit
PR 262 merged steve.dower, 2017-02-23 22:12
PR 446 merged ned.deily, 2017-03-04 03:56
PR 448 merged steve.dower, 2017-03-04 05:31
PR 449 merged steve.dower, 2017-03-04 05:31
PR 450 merged steve.dower, 2017-03-04 05:32
PR 451 merged steve.dower, 2017-03-04 05:50
PR 454 merged ned.deily, 2017-03-04 06:04
PR 455 merged ned.deily, 2017-03-04 06:33
PR 744 merged ned.deily, 2017-03-21 03:16
PR 745 merged ned.deily, 2017-03-21 04:06
PR 746 merged ned.deily, 2017-03-21 04:12
Messages (40)
msg271037 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-07-22 21:43
This is part of the GitHub migration. And by "deprecate" sys._mercurial I mean fill it with default values (https://www.python.org/dev/peps/pep-0512/#deprecate-sys-mercurial).
msg275524 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-09-10 01:12
Get the current branch: `git rev-parse --abbrev-ref HEAD`
msg275525 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-09-10 01:13
Get the current revision: `git rev-parse HEAD`
msg275544 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-09-10 02:27
https://github.com/python/cpython/blob/master/Modules/getbuildinfo.c will need to be updated on top of configure.ac and Makefile.pre.in.
msg275671 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-09-10 18:49
Here is a patch to add sys._git and update Py_GetBuildInfo() as necessary. I would like to see this in Python 3.6 as 3.6.0 will be released using Mercurial but (hopefully) subsequent releases will be on git, so any reviews will be appreciated.

I will hold off on removing sys._mercurial until 3.7 as that will be the first release that doesn't span version control systems.
msg275825 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-09-11 18:13
Thanks for the patch, Brett.  Having done a quick test, I'd like to review this in more detail before applying but that will have to wait until after b1.  If you want, you can reassign this to me.
msg275831 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-09-11 19:05
Sounds good.

On Sun, Sep 11, 2016, 11:13 Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily added the comment:
>
> Thanks for the patch, Brett.  Having done a quick test, I'd like to review
> this in more detail before applying but that will have to wait until after
> b1.  If you want, you can reassign this to me.
>
> ----------
> priority: release blocker -> deferred blocker
> stage:  -> patch review
> versions: +Python 3.7
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27593>
> _______________________________________
>
msg279037 - (view) Author: St├ęphane Wirtel (matrixise) * Date: 2016-10-20 15:00
Hello Brett,

after my review, I propose this patch where I have added the missing indents and remove the ./configure file (because this one can be generated by autoreconf)
msg279044 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-10-20 16:38
The inclusion of changes to ./configure is on purpose since the file is in the repository and not everyone has autoconf installed to necessarily re-generate the file (and we have had issues with people using really old versions of autoconf in the past).
msg279045 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-20 16:40
I'll get to this one shortly.
msg279046 - (view) Author: St├ęphane Wirtel (matrixise) * Date: 2016-10-20 16:44
Hi Brett,

With your comment, I have added the modified ./configure.

Here is the last patch including the indentation.
msg287715 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-02-13 17:40
This is now blocking all releases for all versions.

I'll try and make some time to update the Windows build scripts and project files, but if someone else gets there first feel free to post a PR.
msg287725 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-02-13 21:14
Yeah, I'll get to the non-Windows parts shortly.  I'd been waiting for the transition to happen.
msg288490 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-02-23 22:13
It looks to me like we want:

branch=`git name-rev --name-only HEAD`
revision=`git rev-parse HEAD`
tag=`git name-rev --tags --name-only HEAD`

Unless we're planning on leaving out the tag?

My PR 262 makes the Windows build changes in master, but doesn't change getbuildinfo.c.
msg288574 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-02-25 18:05
I purposefully left the tag out because I don't think it's useful (a tag is just a conveniently named hash and all of our tags are version numbers and that's already available in other parts of the sys module).
msg288719 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-02-28 18:23
Just updated my PR to remove the GITTAG variable, so we'll just go with GITBRANCH and GITVERSION.

(Any value in trying to extract the URL of the remote? That's probably going to be a bit flimsy, but might help more clearly identify forks.)
msg288771 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-01 18:25
I had thought about trying to pull down the remote but it did seems unnecessarily messy when the SHA1 hash should match across various remotes only when the commits actually match.
msg288950 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-04 04:12
I finally took a close look at this and I think the approach Brett advocated here is a bit too simple and does not cover the use case I had in mind when I suggested we needed to have a replacement for sys._mercurial.  Previously when building from a source repo, if the repo was checked out to a tag (like a release tag, say, "v3.6.0"), that tag showed up in sys._mercurial and sys.version making it very easy to identify release builds.  For example, with the macOS pythons we provide on python.org, you see versions like this:

$ /usr/local/bin/python3.5 -c 'import sys;print(sys.version)'
3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:
$ /usr/local/bin/python3.5
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

Having the tag, if available, is very important, IMO.  Also, the original patch didn't deal with the platform module which also uses the SCM info.

Since we've now made the transition to git, it seems to me there's no reason to complicate the code by trying to support either hg or git; we don't support svn anymore.  So the PR is simpler and follows very much the hg support.  I think this also means that Steve's PR for the Windows installer should be changed back to use tags again.  I'd like to get this into 3.6.1 if possible.   Better late than never!
msg288964 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-04 06:21
Here's some sample output with the change:

C:\build\cpython36>PCbuild\win32\python_d.exe
Python 3.6.0+ (3.6:95c50e5aed9e5683676e18349dd94b11901a66b3, Mar  4 2017, 06:08:54) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._git
('CPython', '3.6', '95c50e5aed9e5683676e18349dd94b11901a66b3')

I wonder whether we should shorten the revision hash for the copyright string? I think sys._git[2] should keep the whole output.
msg288965 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-04 06:23
copyright==>version string (sys.version)

Also, that part is clearly not release blocking. My PR for 2.7 is waiting for Travis to catch up, so once that's done and Ned has finished his backports (or Larry and Ben have backported to their versions?) we can lower the priority on this.
msg288966 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-04 06:25
Yeah, shortening the hash in the version string might be nice.  Feel free to do so.  It can probably wait for 3.6.2.
msg288967 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-04 06:27
GitHub trims to the first 7 characters. I see no harm in doing that for sys.version (but not tonight)
msg288988 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-04 17:30
git actually does not always shorten to 7 characters. In git 2.11 (I think) they shorten to the shortest length to guarantee uniqueness. So `git rev-parse --short HEAD` gives a hash of de04644627 which is 10 characters.
msg288994 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-04 20:42
I just did my build for 3.6.1rc1 and it looks like `git name-rev --name-only --tags HEAD` returns "3.6.1rc1^0" for some reason.

Perhaps "git describe" is the better command to use here? From what I've seen of that, it should do "tag if tagged, else short hash" by default, and we can do "git describe --dirty=<suffix>" to get an extra marker for when it has been modified. (Though this may mean taking GITTAG back out of our scripts.)

Otherwise, I think the releases are no longer blocked on this, and we can take a bit of time to figure out the right format.
msg289911 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-21 04:49
Based on Brett's and Steve's feedback, I've pushed some tweaks to the captured git info.  With the changes, we now use --short form of git hash.  And we use output from "git describe --all --always --dirty" for the tag.  I added --all and --always for better results with development builds; the results should be the same for release (tagged) builds - I think.  I'm going to cherry pick this into 3.6.1 final and we can see how this works for development builds and further tweak as necessary.  Steve, note that I was brave and modified the Windows builds; yay, AppVeyor.  I'll also cherrypick these to 2.7 soon.

Expected outputs:
1. previous hg
2. previous git
3. updated git

Release (tagged) build:
1. Python 3.7.0a0 (v3.7.0a0:4def2a2901a5, ...
2. Python 3.7.0a0 (v3.7.0a0^0:05f53735c8912f8df1077e897f052571e13c3496, ...
3. Python 3.7.0a0 (v3.7.0a0:05f53735c8, ...

Development build:
1. Python 3.7.0a0 (default:41df79263a11, ...
2. Python 3.7.0a0 (master:05f53735c8912f8df1077e897f052571e13c3496, ...
3. Python 3.7.0a0 (heads/master-dirty:05f53735c8, ...

"dirty" means the working tree has uncommitted changes.
See "git help describe" for more info.
msg289943 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-21 16:51
The output LGTM.
msg290121 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-24 20:09
New changeset 8c18fbeed1c7721b67f1726a6e9c41acef823135 by Ned Deily in branch '3.6':
bpo-27593: Revise git SCM build info. (#744) (#745)
https://github.com/python/cpython/commit/8c18fbeed1c7721b67f1726a6e9c41acef823135
msg290125 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-24 20:17
New changeset ed512cba78af211da4d83cbb7cc533c39176f374 by Ned Deily in branch '3.5':
bpo-27593: Revise git SCM build info. (#744) (#746)
https://github.com/python/cpython/commit/ed512cba78af211da4d83cbb7cc533c39176f374
msg290126 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-24 20:17
New changeset e9213d929d7b0075539e87416f6a6fb86c27454b by Ned Deily in branch '3.6':
bpo-27593: Revise git SCM build info. (#744) (#745)
https://github.com/python/cpython/commit/e9213d929d7b0075539e87416f6a6fb86c27454b
msg290127 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-24 20:17
New changeset 554626ada769abf82a5dabe6966afa4265acb6a6 by Ned Deily in branch 'master':
bpo-27593: Revise git SCM build info. (#744)
https://github.com/python/cpython/commit/554626ada769abf82a5dabe6966afa4265acb6a6
msg290299 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-24 22:55
New changeset 6120484e46886fbc798c85a523bfe196faa1f60f by Steve Dower in branch '2.7':
bpo-27593: Updates Windows build to use information from git (#262) (#448)
https://github.com/python/cpython/commit/6120484e46886fbc798c85a523bfe196faa1f60f
msg290305 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-24 22:56
New changeset a2edd3ae4074952ce77d9319da2dbb2a47300c27 by Ned Deily in branch '3.5':
[3.5] bpo-27593: Get SCM build info from git instead of hg. (#446) (#454) (#455)
https://github.com/python/cpython/commit/a2edd3ae4074952ce77d9319da2dbb2a47300c27
msg290307 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-24 22:56
New changeset 95c50e5aed9e5683676e18349dd94b11901a66b3 by Ned Deily in branch '3.6':
[3.6] bpo-27593: Get SCM build info from git instead of hg. (#446) (#454)
https://github.com/python/cpython/commit/95c50e5aed9e5683676e18349dd94b11901a66b3
msg290309 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-24 22:58
New changeset d3e1e9df724d97ab83113c2d5fa15179d1dcd560 by Steve Dower in branch '3.6':
bpo-27593: Updates Windows build to use information from git (#262) (#450)
https://github.com/python/cpython/commit/d3e1e9df724d97ab83113c2d5fa15179d1dcd560
msg290310 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-24 22:59
New changeset cf445f10560483d38485204cf46ff1d0adcb4192 by Steve Dower in branch '3.5':
bpo-27593: Updates Windows build to use information from git (#262) (#449)
https://github.com/python/cpython/commit/cf445f10560483d38485204cf46ff1d0adcb4192
msg290311 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-24 22:59
New changeset 5c4b0d063aba0a68c325073f5f312a2c9f40d178 by Ned Deily in branch 'master':
bpo-27593: Get SCM build info from git instead of hg. (#446)
https://github.com/python/cpython/commit/5c4b0d063aba0a68c325073f5f312a2c9f40d178
msg290312 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-24 22:59
New changeset a0c07d2edd345d2867f97ac31822c9544f9cbcf0 by Steve Dower in branch 'master':
bpo-27593: Updates Windows build to use information from git (#262)
https://github.com/python/cpython/commit/a0c07d2edd345d2867f97ac31822c9544f9cbcf0
msg290600 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-03-27 12:10
The 3.5, 3.6 and master branches of Python have a new sys._git attribute and lost their sys._mercurial attribute. But the 2.7 branch is still stuck at sys._mercurial: version and tag are empty.

Should we backport sys._git from master to 2.7 and remove sys._mercurial? Maybe on 2.7 we can keep sys._mercurial.
msg290610 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-27 13:55
It needs to be backported there too, or we need to unmigrate 2.7 back to mercurial. Otherwise we can't make a release.
msg290626 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-27 16:29
I'll be doing backports to 2.7 for this and some other things shortly.
History
Date User Action Args
2017-03-27 16:29:28ned.deilysetmessages: + msg290626
2017-03-27 13:55:39steve.dowersetmessages: + msg290610
2017-03-27 12:10:07hayposetnosy: + haypo
messages: + msg290600
2017-03-24 22:59:32steve.dowersetmessages: + msg290312
2017-03-24 22:59:26ned.deilysetmessages: + msg290311
2017-03-24 22:59:13steve.dowersetmessages: + msg290310
2017-03-24 22:58:49steve.dowersetmessages: + msg290309
2017-03-24 22:56:59ned.deilysetmessages: + msg290307
2017-03-24 22:56:44ned.deilysetmessages: + msg290305
2017-03-24 22:55:03steve.dowersetmessages: + msg290299
2017-03-24 20:17:30ned.deilysetmessages: + msg290127
2017-03-24 20:17:20ned.deilysetmessages: + msg290126
2017-03-24 20:17:13ned.deilysetmessages: + msg290125
2017-03-24 20:09:33ned.deilysetmessages: + msg290121
2017-03-21 16:51:38brett.cannonsetmessages: + msg289943
2017-03-21 04:49:01ned.deilysetmessages: + msg289911
2017-03-21 04:12:50ned.deilysetpull_requests: + pull_request660
2017-03-21 04:06:55ned.deilysetpull_requests: + pull_request659
2017-03-21 03:16:05ned.deilysetpull_requests: + pull_request658
2017-03-17 21:05:13larrysetpull_requests: - pull_request578
2017-03-17 21:00:31larrysetpull_requests: + pull_request578
2017-03-09 20:14:57ned.deilylinkissue29775 superseder
2017-03-04 20:42:34steve.dowersetpriority: release blocker -> high

messages: + msg288994
2017-03-04 17:30:13brett.cannonsetmessages: + msg288988
2017-03-04 06:33:44ned.deilysetpull_requests: + pull_request379
2017-03-04 06:27:07steve.dowersetmessages: + msg288967
2017-03-04 06:25:16ned.deilysetmessages: + msg288966
2017-03-04 06:23:38steve.dowersetmessages: + msg288965
2017-03-04 06:21:53steve.dowersetmessages: + msg288964
2017-03-04 06:04:03ned.deilysetpull_requests: + pull_request378
2017-03-04 05:50:30steve.dowersetpull_requests: + pull_request377
2017-03-04 05:32:02steve.dowersetpull_requests: + pull_request376
2017-03-04 05:31:30steve.dowersetpull_requests: + pull_request375
2017-03-04 05:31:09steve.dowersetpull_requests: + pull_request374
2017-03-04 04:12:14ned.deilysetmessages: + msg288950
2017-03-04 03:56:07ned.deilysetpull_requests: + pull_request372
2017-03-01 18:25:40brett.cannonsetmessages: + msg288771
2017-02-28 18:23:16steve.dowersetmessages: + msg288719
2017-02-25 18:05:05brett.cannonsetmessages: + msg288574
2017-02-23 22:13:17steve.dowersetmessages: + msg288490
2017-02-23 22:12:31steve.dowersetpull_requests: + pull_request234
2017-02-13 21:14:39ned.deilysetmessages: + msg287725
2017-02-13 17:40:58steve.dowersetpriority: deferred blocker -> release blocker
versions: + Python 2.7, Python 3.5
nosy: + larry, benjamin.peterson, steve.dower

messages: + msg287715
2016-10-20 16:44:31matrixisesetfiles: + issue27593-with-indent-3.7-2.diff

messages: + msg279046
2016-10-20 16:40:53ned.deilysetassignee: brett.cannon -> ned.deily
messages: + msg279045
2016-10-20 16:38:24brett.cannonsetmessages: + msg279044
2016-10-20 15:00:14matrixisesetfiles: + issue27593-with-indent-3.7.diff
nosy: + matrixise
messages: + msg279037

2016-09-11 19:05:30brett.cannonsetmessages: + msg275831
2016-09-11 18:13:11ned.deilysetpriority: release blocker -> deferred blocker

stage: patch review
messages: + msg275825
versions: + Python 3.7
2016-09-10 18:49:59brett.cannonsetfiles: + issue27593.diff
priority: normal -> release blocker

versions: + Python 3.6
keywords: + patch
nosy: + ned.deily

messages: + msg275671
2016-09-10 02:27:22brett.cannonsetmessages: + msg275544
2016-09-10 01:13:43brett.cannonsetmessages: + msg275525
2016-09-10 01:12:51brett.cannonsetassignee: brett.cannon
messages: + msg275524
2016-07-22 21:43:25brett.cannoncreate