classification
Title: platform.libc_ver() returns incorrect version number
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Thomas.Waldmann, benjamin.peterson, doko, lemburg, ned.deily, njs, serhiy.storchaka, vstinner
Priority: release blocker Keywords: 3.6regression, 3.7regression, patch

Created on 2016-03-12 00:36 by Thomas.Waldmann, last changed 2018-12-05 14:23 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
results-20160709-224820.csv njs, 2016-07-10 06:00
platform-no-distutils.diff doko, 2018-08-20 16:00 platform-not-using-distutils
Pull Requests
URL Status Linked Edit
PR 7684 merged serhiy.storchaka, 2018-06-13 17:51
PR 8193 merged serhiy.storchaka, 2018-07-09 08:55
PR 8195 merged miss-islington, 2018-07-09 09:56
PR 8196 merged serhiy.storchaka, 2018-07-09 10:17
PR 8356 merged serhiy.storchaka, 2018-07-20 17:42
PR 8952 merged miss-islington, 2018-08-27 10:30
PR 8970 merged serhiy.storchaka, 2018-08-28 09:36
PR 8971 closed serhiy.storchaka, 2018-08-28 09:37
PR 8973 merged serhiy.storchaka, 2018-08-28 10:20
PR 9061 merged miss-islington, 2018-09-04 14:31
PR 9062 closed serhiy.storchaka, 2018-09-04 14:43
PR 10868 merged vstinner, 2018-12-03 12:57
Messages (36)
msg261624 - (view) Author: Thomas Waldmann (Thomas.Waldmann) Date: 2016-03-12 00:36
platform.libc_ver() is trivially broken as it uses string comparison internally to determine the maximum libc version number (which is obviously broken as "2.9" > "2.10").
msg261648 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-03-12 13:34
True. At the time the code was written, this was not an issue :-)

Is the libc version information documented somewhere ? If so, we could probably add a better parser for it.
msg261649 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-03-12 13:35
Adding other Python versions as well, since this is a bug.
msg270073 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2016-07-10 06:00
We just ran into this in pip -- https://github.com/pypa/pip/pull/3836

I'd really recommend dropping the current "grovel through the binary doing a regex search" strategy -- it's incredibly error prone, and AFAICT doesn't really give any value. This code OTOH reliably lets you detect glibc and gives the exact version number with no fuss:
   https://github.com/pypa/pip/blob/master/pip/utils/glibc.py#L9

What about non-glibc systems? Unfortunately the current libc_ver() turns out not to work well for those either. Attached is a CSV file showing the return value of ~1.2 billion calls to platform.libc_ver() by the last 6 months of pip users. You can see that the current code basically never returns anything useful for non-glibc platforms. (The one exception is that it seems to be able to detect uclibc 0.9.32 and label it as "libc 0.9.32".)

Don't get me wrong: it'd be really really useful if there were some way to detect and distinguish between the common non-glibc libcs like musl/bionic/uclibc/..., but I'm not sure how to do that -- and unfortunately the current code definitely doesn't do the job :-(.
msg270080 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-07-10 10:37
At the time the code was written, libc and glibc were in wide spread use, so it's not surprising that it doesn't work well for other C libs.

Note that the routine returns the highest libc version number used and required by the executable (usually the Python interpreter). This does not necessarily correspond to the version installed on the system. The purpose of the function was to determine the minimum libc compatibility requirements of the executable.

The routine you quote uses ctypes and only works for glibc, so parsing needs to be kept around as fallback solution. It also returns the libc version that is currently used on the system; not necessarily the minimum version required, so semantics are different.
msg270089 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2016-07-10 15:14
> The purpose of the function was to determine the minimum libc compatibility requirements of the executable.

For what it's worth, I didn't know this, the pip authors obviously didn't know this, and after rereading the docs just now I still can't quite tell that this was intended. I'm not sure why one would want these semantics either, but at a minimum it would help to document the intended semantics much more clearly.

> parsing needs to be kept around as fallback solution

The point of the data I attached was that AFAICT there don't exist any currently-in-use, non-glibc systems where "parsing" returns a meaningful result.
msg319456 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-13 12:34
I need the glibc version for skipping a test if run with glibc containing a bug (see issue31630 and issue33630). platform.libc_ver() is not usable, it always returns '2.9' (the version that has bugs), while the actual version on my computer is '2.25' (the version that doesn't have these bugs).
msg319479 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-13 17:50
I agree that the strategy used for platform.libc_ver() is not perfect. But the implementation has bugs that make it useless. The following PR fixes two bugs in the implementation:

1) Version numbers compared as strings.
2) Versions that are located on the border of 16 KiB blocks were not recognized or were recognized incorrectly.
msg321301 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-09 08:47
New changeset 2a9b8babf0d09946ebebfdb2931cc0d3db5a1d3d by Serhiy Storchaka in branch 'master':
bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684)
https://github.com/python/cpython/commit/2a9b8babf0d09946ebebfdb2931cc0d3db5a1d3d
msg321308 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-09 09:55
New changeset 7c43b801503c802ed6ea4b811f5bc73791249d94 by Serhiy Storchaka in branch '3.7':
[3.7] bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684). (GH-8193)
https://github.com/python/cpython/commit/7c43b801503c802ed6ea4b811f5bc73791249d94
msg321310 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-09 11:38
New changeset d73497ba52171bc8f786a70ecf50d3104b596221 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684). (GH-8193) (GH-8195)
https://github.com/python/cpython/commit/d73497ba52171bc8f786a70ecf50d3104b596221
msg321311 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-09 11:39
New changeset b1e6e5615a8e82fcf569368fac5c5b0385929855 by Serhiy Storchaka in branch '2.7':
bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684). (GH-8193) (GH-8196)
https://github.com/python/cpython/commit/b1e6e5615a8e82fcf569368fac5c5b0385929855
msg321462 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-11 14:39
Can we close this issue? It seems like the bug has been fixed, no?
msg322021 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2018-07-20 15:46
please don't close this yet.  the patch introduces a dependency on the distutils package.  I don't see any reason to compare the libc version as a python egg version, so please avoid that.

if we need to have a module to compare arbitrary version strings, then we should think about adding one in the standard library, but I think that's what the packaging module is for.
msg322023 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2018-07-20 15:55
a tuple comparison should be good enough
msg323795 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2018-08-20 15:33
no, it's not fixed at all. Setting as a release blocker for 3.6 and 3.7.
msg323796 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2018-08-20 16:00
proposed patch attached
msg323802 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-08-20 17:22
platform-no-distutils.diff restores the original bug: "2.9" > "2.10".

PR 8356 removes the dependency from distutils and use a sophisticated function for parsing versions.
msg323805 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2018-08-20 17:57
fine! what prevents merging and backporting this issue?
msg323816 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-08-21 03:38
I'll merge it if it looks good to you.
msg323823 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2018-08-21 07:26
I added a comment to the PR, but other than that I think it's good to go.
msg323825 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2018-08-21 07:48
+1
msg324164 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-08-27 10:29
New changeset 7d81e8f5995df6980a1a02923e224a481375f130 by Serhiy Storchaka in branch 'master':
bpo-26544: Get rid of dependence from distutils in platform. (GH-8356)
https://github.com/python/cpython/commit/7d81e8f5995df6980a1a02923e224a481375f130
msg324231 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 09:51
Would it be possible to add a few tests on platform._comparable_version()? It would make me more confortable for backports.
msg324581 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-04 12:04
New changeset 7917aadb3edb7616d6164c5eaba24df6ac0a5fc6 by Serhiy Storchaka in branch 'master':
bpo-26544: Add test for platform._comparable_version(). (GH-8973)
https://github.com/python/cpython/commit/7917aadb3edb7616d6164c5eaba24df6ac0a5fc6
msg324582 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-04 12:32
Thanks for adding tests ;-) It now looks better to me ;-)
msg324588 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-04 14:31
New changeset 20a8392cec2967f15ae81633c1775645b3ca40da by Serhiy Storchaka in branch '3.7':
[3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356). (GH-8970)
https://github.com/python/cpython/commit/20a8392cec2967f15ae81633c1775645b3ca40da
msg324637 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-05 14:46
New changeset 1a3eb125dc07a28a5af62446778ed7cca95ed3da by Victor Stinner (Miss Islington (bot)) in branch '3.6':
[3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) (GH-8970) (GH-9061)
https://github.com/python/cpython/commit/1a3eb125dc07a28a5af62446778ed7cca95ed3da
msg324638 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-05 14:46
New changeset 9734024ec65311e33936faa83fb1cb249ef0de9d by Victor Stinner (Miss Islington (bot)) in branch '2.7':
bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) (GH-8952)
https://github.com/python/cpython/commit/9734024ec65311e33936faa83fb1cb249ef0de9d
msg324639 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-05 14:46
Thanks everybody! The issue should now be fixed in all supported branches ;-)
msg324784 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-07 18:46
Thank you for merging PRs Victor, but seems you have merged wrong PR for 2.7.
msg324794 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-07 21:12
> Thank you for merging PRs Victor, but seems you have merged wrong PR for 2.7.

Oh. I didn't notice that there were two PRs for 2.7. Do you want to rebase your PR 9062 on 2.7, or revert the commit that I merged? It's up to you ;-)

I reopen the issue.

It seems like PR 8971 is also open.
msg325158 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-09-12 18:32
This is now only open and a release blocker for 2.7, correct?
msg325520 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-17 10:25
Seems the only difference between PR 8952 and PR 9062 is additional tests. This is not important.
msg330951 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-03 15:49
New changeset 8687bd86e6f138ef0699a1e9f3f9555765949b51 by Victor Stinner in branch '2.7':
bpo-26544: Make platform.libc_ver() less slow (GH-10868)
https://github.com/python/cpython/commit/8687bd86e6f138ef0699a1e9f3f9555765949b51
msg331116 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-05 14:23
> bpo-26544: Make platform.libc_ver() less slow (GH-10868)
> https://github.com/python/cpython/commit/8687bd86e6f138ef0699a1e9f3f9555765949b51

"Coarse benchmark on Fedora 29: 1.6 sec => 0.1 sec."

Oops, my benchmark in the commit message was wrong, it included the startup time. Correct benchmark says 44x faster, it's way better!

Python 2.7: [old_py2] 1.51 sec +- 0.03 sec -> [if_in] 34.6 ms +- 0.4 ms: 43.61x faster (-98%)
History
Date User Action Args
2018-12-05 14:23:05vstinnersetmessages: + msg331116
2018-12-03 15:49:34vstinnersetmessages: + msg330951
2018-12-03 12:57:53vstinnersetpull_requests: + pull_request10104
2018-09-17 10:25:38serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg325520
2018-09-12 18:32:03ned.deilysetmessages: + msg325158
2018-09-07 21:12:48vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg324794
2018-09-07 18:46:44serhiy.storchakasetmessages: + msg324784
2018-09-05 14:46:55vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg324639

stage: patch review -> resolved
2018-09-05 14:46:29vstinnersetmessages: + msg324638
2018-09-05 14:46:01vstinnersetmessages: + msg324637
2018-09-04 14:43:17serhiy.storchakasetpull_requests: + pull_request8522
2018-09-04 14:31:37miss-islingtonsetpull_requests: + pull_request8521
2018-09-04 14:31:26serhiy.storchakasetmessages: + msg324588
2018-09-04 12:32:50vstinnersetmessages: + msg324582
2018-09-04 12:04:36serhiy.storchakasetmessages: + msg324581
2018-08-28 10:20:11serhiy.storchakasetpull_requests: + pull_request8446
2018-08-28 09:51:01vstinnersetmessages: + msg324231
2018-08-28 09:37:46serhiy.storchakasetpull_requests: + pull_request8444
2018-08-28 09:36:15serhiy.storchakasetpull_requests: + pull_request8443
2018-08-27 10:30:13miss-islingtonsetpull_requests: + pull_request8428
2018-08-27 10:29:56serhiy.storchakasetmessages: + msg324164
2018-08-21 07:48:33dokosetmessages: + msg323825
2018-08-21 07:26:53lemburgsetmessages: + msg323823
2018-08-21 03:38:31serhiy.storchakasetmessages: + msg323816
2018-08-20 17:57:12dokosetmessages: + msg323805
2018-08-20 17:22:27serhiy.storchakasetmessages: + msg323802
2018-08-20 16:00:56dokosetmessages: + msg323796
2018-08-20 16:00:30dokosetfiles: + platform-no-distutils.diff
2018-08-20 15:33:06dokosetpriority: normal -> release blocker

nosy: + ned.deily, benjamin.peterson
messages: + msg323795

keywords: + 3.6regression, 3.7regression
2018-07-20 17:42:05serhiy.storchakasetpull_requests: + pull_request7891
2018-07-20 15:55:14dokosetmessages: + msg322023
2018-07-20 15:46:14dokosetnosy: + doko
messages: + msg322021
2018-07-11 14:39:39vstinnersetnosy: + vstinner
messages: + msg321462
2018-07-09 11:39:09serhiy.storchakasetmessages: + msg321311
2018-07-09 11:38:30serhiy.storchakasetmessages: + msg321310
2018-07-09 10:17:15serhiy.storchakasetpull_requests: + pull_request7748
2018-07-09 09:56:47miss-islingtonsetpull_requests: + pull_request7747
2018-07-09 09:55:38serhiy.storchakasetmessages: + msg321308
2018-07-09 08:55:29serhiy.storchakasetpull_requests: + pull_request7745
2018-07-09 08:47:49serhiy.storchakasetmessages: + msg321301
2018-06-13 17:51:27serhiy.storchakasetversions: + Python 3.7, Python 3.8, - Python 3.4, Python 3.5
2018-06-13 17:51:06serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request7297
2018-06-13 17:50:38serhiy.storchakasetmessages: + msg319479
2018-06-13 12:34:53serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg319456
2016-07-10 15:14:08njssetmessages: + msg270089
2016-07-10 10:37:00lemburgsetmessages: + msg270080
2016-07-10 06:00:48njssetfiles: + results-20160709-224820.csv
nosy: + njs
messages: + msg270073

2016-03-12 13:35:18lemburgsetmessages: + msg261649
versions: + Python 2.7, Python 3.4, Python 3.6
2016-03-12 13:34:39lemburgsetmessages: + msg261648
2016-03-12 00:39:49ned.deilysetnosy: + lemburg
2016-03-12 00:36:03Thomas.Waldmanncreate