Deprecate sys._mercurial and create sys._git #71780

brettcannon · 2016-07-22T21:43:25Z

BPO	27593
Nosy	@brettcannon, @vstinner, @larryhastings, @benjaminp, @ned-deily, @zooba, @matrixise
PRs	bpo-27593: Updates Windows build to use information from git #262 bpo-27593: Get SCM build info from git instead of hg. #446 bpo-27593: Updates Windows build to use information from git (#262) #448 bpo-27593: Updates Windows build to use information from git (#262) #449 bpo-27593: Updates Windows build to use information from git (#262) #450 bpo-27593: Fixes git command #451 [3.6] bpo-27593: Get SCM build info from git instead of hg. (#446) #454 [3.5] bpo-27593: Get SCM build info from git instead of hg. (#446) #455 bpo-27593: Revise git SCM build info. #744 [3.6] bpo-27593: Revise git SCM build info. (#744) #745 [3.5] bpo-27593: Revise git SCM build info. (#744) #746 bpo-27593: Get SCM build info from git instead of hg #1327 bpo-27593: Revise git SCM build info. (#744) (#746) #1392
Files	issue27593.diff issue27593-with-indent-3.7.diff issue27593-with-indent-3.7-2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2017-05-02.16:32:17.680>
created_at = <Date 2016-07-22.21:43:25.488>
labels = ['3.7', 'library']
title = 'Deprecate sys._mercurial and create sys._git'
updated_at = <Date 2017-05-02.20:52:09.301>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2017-05-02.20:52:09.301>
actor = 'vstinner'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2017-05-02.16:32:17.680>
closer = 'brett.cannon'
components = ['Library (Lib)']
creation = <Date 2016-07-22.21:43:25.488>
creator = 'brett.cannon'
dependencies = []
files = ['44535', '45152', '45154']
hgrepos = []
issue_num = 27593
keywords = ['patch']
message_count = 46.0
messages = ['271037', '275524', '275525', '275544', '275671', '275825', '275831', '279037', '279044', '279045', '279046', '287715', '287725', '288490', '288574', '288719', '288771', '288950', '288964', '288965', '288966', '288967', '288988', '288994', '289911', '289943', '290121', '290125', '290126', '290127', '290299', '290305', '290307', '290309', '290310', '290311', '290312', '290600', '290610', '290626', '292461', '292772', '292778', '292785', '292801', '292805']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'steve.dower', 'matrixise']
pr_nums = ['262', '446', '448', '449', '450', '451', '454', '455', '744', '745', '746', '1327', '1392']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue27593'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

brettcannon · 2016-07-22T21:43:25Z

This is part of the GitHub migration. And by "deprecate" sys._mercurial I mean fill it with default values (https://www.python.org/dev/peps/pep-0512/#deprecate-sys-mercurial).

brettcannon · 2016-09-10T01:12:52Z

Get the current branch: git rev-parse --abbrev-ref HEAD

brettcannon · 2016-09-10T01:13:43Z

Get the current revision: git rev-parse HEAD

brettcannon · 2016-09-10T02:27:22Z

https://github.com/python/cpython/blob/master/Modules/getbuildinfo.c will need to be updated on top of configure.ac and Makefile.pre.in.

brettcannon · 2016-09-10T18:49:59Z

Here is a patch to add sys._git and update Py_GetBuildInfo() as necessary. I would like to see this in Python 3.6 as 3.6.0 will be released using Mercurial but (hopefully) subsequent releases will be on git, so any reviews will be appreciated.

I will hold off on removing sys._mercurial until 3.7 as that will be the first release that doesn't span version control systems.

ned-deily · 2016-09-11T18:13:12Z

Thanks for the patch, Brett. Having done a quick test, I'd like to review this in more detail before applying but that will have to wait until after b1. If you want, you can reassign this to me.

brettcannon · 2016-09-11T19:05:30Z

Sounds good.

On Sun, Sep 11, 2016, 11:13 Ned Deily <report@bugs.python.org> wrote:

Ned Deily added the comment:

Thanks for the patch, Brett. Having done a quick test, I'd like to review
this in more detail before applying but that will have to wait until after
b1. If you want, you can reassign this to me.

----------
priority: release blocker -> deferred blocker
stage: -> patch review
versions: +Python 3.7

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue27593\>

matrixise · 2016-10-20T15:00:15Z

Hello Brett,

after my review, I propose this patch where I have added the missing indents and remove the ./configure file (because this one can be generated by autoreconf)

brettcannon · 2016-10-20T16:38:24Z

The inclusion of changes to ./configure is on purpose since the file is in the repository and not everyone has autoconf installed to necessarily re-generate the file (and we have had issues with people using really old versions of autoconf in the past).

ned-deily · 2016-10-20T16:40:54Z

I'll get to this one shortly.

matrixise · 2016-10-20T16:44:31Z

Hi Brett,

With your comment, I have added the modified ./configure.

Here is the last patch including the indentation.

zooba · 2017-02-13T17:40:59Z

This is now blocking all releases for all versions.

I'll try and make some time to update the Windows build scripts and project files, but if someone else gets there first feel free to post a PR.

ned-deily · 2017-02-13T21:14:39Z

Yeah, I'll get to the non-Windows parts shortly. I'd been waiting for the transition to happen.

zooba · 2017-02-23T22:13:17Z

It looks to me like we want:

branch=git name-rev --name-only HEAD
revision=git rev-parse HEAD
tag=git name-rev --tags --name-only HEAD

Unless we're planning on leaving out the tag?

My PR 262 makes the Windows build changes in master, but doesn't change getbuildinfo.c.

brettcannon · 2017-02-25T18:05:05Z

I purposefully left the tag out because I don't think it's useful (a tag is just a conveniently named hash and all of our tags are version numbers and that's already available in other parts of the sys module).

zooba · 2017-02-28T18:23:16Z

Just updated my PR to remove the GITTAG variable, so we'll just go with GITBRANCH and GITVERSION.

(Any value in trying to extract the URL of the remote? That's probably going to be a bit flimsy, but might help more clearly identify forks.)

brettcannon · 2017-03-01T18:25:41Z

I had thought about trying to pull down the remote but it did seems unnecessarily messy when the SHA1 hash should match across various remotes only when the commits actually match.

ned-deily · 2017-03-04T04:12:14Z

I finally took a close look at this and I think the approach Brett advocated here is a bit too simple and does not cover the use case I had in mind when I suggested we needed to have a replacement for sys._mercurial. Previously when building from a source repo, if the repo was checked out to a tag (like a release tag, say, "v3.6.0"), that tag showed up in sys._mercurial and sys.version making it very easy to identify release builds. For example, with the macOS pythons we provide on python.org, you see versions like this:

$ /usr/local/bin/python3.5 -c 'import sys;print(sys.version)'
3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:
$ /usr/local/bin/python3.5
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

Having the tag, if available, is very important, IMO. Also, the original patch didn't deal with the platform module which also uses the SCM info.

Since we've now made the transition to git, it seems to me there's no reason to complicate the code by trying to support either hg or git; we don't support svn anymore. So the PR is simpler and follows very much the hg support. I think this also means that Steve's PR for the Windows installer should be changed back to use tags again. I'd like to get this into 3.6.1 if possible. Better late than never!

zooba · 2017-03-04T06:21:53Z

Here's some sample output with the change:

C:\build\cpython36>PCbuild\win32\python_d.exe
Python 3.6.0+ (3.6:95c50e5aed9e5683676e18349dd94b11901a66b3, Mar  4 2017, 06:08:54) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._git
('CPython', '3.6', '95c50e5aed9e5683676e18349dd94b11901a66b3')

I wonder whether we should shorten the revision hash for the copyright string? I think sys._git[2] should keep the whole output.

zooba · 2017-03-04T06:23:39Z

copyright==>version string (sys.version)

Also, that part is clearly not release blocking. My PR for 2.7 is waiting for Travis to catch up, so once that's done and Ned has finished his backports (or Larry and Ben have backported to their versions?) we can lower the priority on this.

ned-deily · 2017-03-04T06:25:17Z

Yeah, shortening the hash in the version string might be nice. Feel free to do so. It can probably wait for 3.6.2.

zooba · 2017-03-04T06:27:07Z

GitHub trims to the first 7 characters. I see no harm in doing that for sys.version (but not tonight)

brettcannon · 2017-03-04T17:30:13Z

git actually does not always shorten to 7 characters. In git 2.11 (I think) they shorten to the shortest length to guarantee uniqueness. So git rev-parse --short HEAD gives a hash of de04644 which is 10 characters.

zooba · 2017-03-04T20:42:35Z

I just did my build for 3.6.1rc1 and it looks like git name-rev --name-only --tags HEAD returns "3.6.1rc1^0" for some reason.

Perhaps "git describe" is the better command to use here? From what I've seen of that, it should do "tag if tagged, else short hash" by default, and we can do "git describe --dirty=<suffix>" to get an extra marker for when it has been modified. (Though this may mean taking GITTAG back out of our scripts.)

Otherwise, I think the releases are no longer blocked on this, and we can take a bit of time to figure out the right format.

ned-deily · 2017-03-21T04:49:01Z

Based on Brett's and Steve's feedback, I've pushed some tweaks to the captured git info. With the changes, we now use --short form of git hash. And we use output from "git describe --all --always --dirty" for the tag. I added --all and --always for better results with development builds; the results should be the same for release (tagged) builds - I think. I'm going to cherry pick this into 3.6.1 final and we can see how this works for development builds and further tweak as necessary. Steve, note that I was brave and modified the Windows builds; yay, AppVeyor. I'll also cherrypick these to 2.7 soon.

Expected outputs:

previous hg
previous git
updated git

Release (tagged) build:

Python 3.7.0a0 (v3.7.0a0:4def2a2901a5, ...
Python 3.7.0a0 (v3.7.0a0^0:05f53735c8912f8df1077e897f052571e13c3496, ...
Python 3.7.0a0 (v3.7.0a0:05f53735c8, ...

Development build:

Python 3.7.0a0 (default:41df79263a11, ...
Python 3.7.0a0 (master:05f53735c8912f8df1077e897f052571e13c3496, ...
Python 3.7.0a0 (heads/master-dirty:05f53735c8, ...

"dirty" means the working tree has uncommitted changes.
See "git help describe" for more info.

brettcannon · 2017-03-21T16:51:38Z

The output LGTM.

ned-deily · 2017-03-24T20:09:33Z

New changeset 8c18fbe by Ned Deily in branch '3.6':
bpo-27593: Revise git SCM build info. (#744) (#745)
8c18fbe

ned-deily · 2017-03-24T20:17:13Z

New changeset ed512cb by Ned Deily in branch '3.5':
bpo-27593: Revise git SCM build info. (#744) (#746)
ed512cb

ned-deily · 2017-03-24T20:17:21Z

New changeset e9213d9 by Ned Deily in branch '3.6':
bpo-27593: Revise git SCM build info. (#744) (#745)
e9213d9

ned-deily · 2017-03-24T20:17:30Z

New changeset 554626a by Ned Deily in branch 'master':
bpo-27593: Revise git SCM build info. (#744)
554626a

zooba · 2017-03-24T22:55:03Z

New changeset 6120484 by Steve Dower in branch '2.7':
bpo-27593: Updates Windows build to use information from git (#262) (#448)
6120484

ned-deily · 2017-03-24T22:56:45Z

New changeset a2edd3a by Ned Deily in branch '3.5':
[3.5] bpo-27593: Get SCM build info from git instead of hg. (#446) (#454) (#455)
a2edd3a

ned-deily · 2017-03-24T22:56:59Z

New changeset 95c50e5 by Ned Deily in branch '3.6':
[3.6] bpo-27593: Get SCM build info from git instead of hg. (#446) (#454)
95c50e5

zooba · 2017-03-24T22:58:50Z

New changeset d3e1e9d by Steve Dower in branch '3.6':
bpo-27593: Updates Windows build to use information from git (#262) (#450)
d3e1e9d

zooba · 2017-03-24T22:59:14Z

New changeset cf445f1 by Steve Dower in branch '3.5':
bpo-27593: Updates Windows build to use information from git (#262) (#449)
cf445f1

ned-deily · 2017-03-24T22:59:26Z

New changeset 5c4b0d0 by Ned Deily in branch 'master':
bpo-27593: Get SCM build info from git instead of hg. (#446)
5c4b0d0

zooba · 2017-03-24T22:59:33Z

New changeset a0c07d2 by Steve Dower in branch 'master':
bpo-27593: Updates Windows build to use information from git (#262)
a0c07d2

vstinner · 2017-03-27T12:10:07Z

The 3.5, 3.6 and master branches of Python have a new sys._git attribute and lost their sys._mercurial attribute. But the 2.7 branch is still stuck at sys._mercurial: version and tag are empty.

Should we backport sys._git from master to 2.7 and remove sys._mercurial? Maybe on 2.7 we can keep sys._mercurial.

zooba · 2017-03-27T13:55:40Z

It needs to be backported there too, or we need to unmigrate 2.7 back to mercurial. Otherwise we can't make a release.

ned-deily · 2017-03-27T16:29:29Z

I'll be doing backports to 2.7 for this and some other things shortly.

vstinner · 2017-04-27T16:19:38Z

Ned Deily: "I'll be doing backports to 2.7 for this and some other things shortly."

It seems like Ned was busy or forgot 2.7, so I wrote a change: #1327

My change doesn't touch platform.py. platform._sys_version() uses sys.subversion but not sys._hg. If someone wants to enhance the platform module of Python 2.7, I suggest to open a new issue. There is a non-zero risk of breaking the backward compatibility, so I skipped my turn for that one :-)

vstinner · 2017-05-02T14:55:53Z

New changeset 2c7085f by Victor Stinner in branch '2.7':
bpo-27593: Get SCM build info from git instead of hg (bpo-1327)
2c7085f

vstinner · 2017-05-02T15:22:30Z

With #1392 Python 2.7 should up to date with Python master, and this issue should be fixed. Or did I miss something?

brettcannon · 2017-05-02T16:32:18Z

I think that covers it!

vstinner · 2017-05-02T19:50:11Z

New changeset fd6c8bb by Victor Stinner in branch '2.7':
bpo-27593: Revise git SCM build info. (#744) (#746) (bpo-1392)
fd6c8bb

vstinner · 2017-05-02T20:52:09Z

I think that covers it!

Cool, good job ;-)

brettcannon added the stdlib Python modules in the Lib dir label Jul 22, 2016

brettcannon self-assigned this Sep 10, 2016

brettcannon added the release-blocker label Sep 10, 2016

ned-deily added 3.7 (EOL) end of life deferred-blocker and removed release-blocker labels Sep 11, 2016

ned-deily assigned ned-deily and unassigned brettcannon Oct 20, 2016

zooba added release-blocker and removed deferred-blocker labels Feb 13, 2017

zooba removed the release-blocker label Mar 4, 2017

brettcannon closed this as completed May 2, 2017

ezio-melotti transferred this issue from another repository Apr 10, 2022

Deprecate sys._mercurial and create sys._git #71780

Deprecate sys._mercurial and create sys._git #71780

Comments

brettcannon commented Jul 22, 2016

brettcannon commented Jul 22, 2016

brettcannon commented Sep 10, 2016

brettcannon commented Sep 10, 2016

brettcannon commented Sep 10, 2016

brettcannon commented Sep 10, 2016

ned-deily commented Sep 11, 2016

brettcannon commented Sep 11, 2016

matrixise commented Oct 20, 2016

brettcannon commented Oct 20, 2016

ned-deily commented Oct 20, 2016

matrixise commented Oct 20, 2016

zooba commented Feb 13, 2017

ned-deily commented Feb 13, 2017

zooba commented Feb 23, 2017

brettcannon commented Feb 25, 2017

zooba commented Feb 28, 2017

brettcannon commented Mar 1, 2017

ned-deily commented Mar 4, 2017

zooba commented Mar 4, 2017

zooba commented Mar 4, 2017

ned-deily commented Mar 4, 2017

zooba commented Mar 4, 2017

brettcannon commented Mar 4, 2017

zooba commented Mar 4, 2017

ned-deily commented Mar 21, 2017

brettcannon commented Mar 21, 2017

ned-deily commented Mar 24, 2017

ned-deily commented Mar 24, 2017

ned-deily commented Mar 24, 2017

ned-deily commented Mar 24, 2017

zooba commented Mar 24, 2017

ned-deily commented Mar 24, 2017

ned-deily commented Mar 24, 2017

zooba commented Mar 24, 2017

zooba commented Mar 24, 2017

ned-deily commented Mar 24, 2017

zooba commented Mar 24, 2017

vstinner commented Mar 27, 2017

zooba commented Mar 27, 2017

ned-deily commented Mar 27, 2017

vstinner commented Apr 27, 2017

vstinner commented May 2, 2017

vstinner commented May 2, 2017

brettcannon commented May 2, 2017

vstinner commented May 2, 2017

vstinner commented May 2, 2017