platform.libc_ver() returns incorrect version number #70731

thomaswaldmann · 2016-03-12T00:36:03Z

BPO	26544
Nosy	@malemburg, @doko42, @vstinner, @benjaminp, @ned-deily, @njsmith, @serhiy-storchaka, @ThomasWaldmann
PRs	bpo-26544: Fixed implementation of platform.libc_ver(). #7684 [3.7] bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684). #8193 [3.6] bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684) #8195 [2.7] bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684) #8196 bpo-26544: Get rid of dependence from distutils in platform. #8356 [2.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) #8952 [3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356). #8970 [3.6] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356). #8971 bpo-26544: Add test for platform._comparable_version(). #8973 [3.6] [3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356). (GH-8970) #9061 [2.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356). (GH-8970) #9062 [2.7] bpo-26544: Make platform.libc_ver() less slow #10868
Files	results-20160709-224820.csv platform-no-distutils.diff: platform-not-using-distutils

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-17.10:25:38.814>
created_at = <Date 2016-03-12.00:36:03.132>
labels = ['3.7', '3.8', 'type-bug', 'library', 'release-blocker']
title = 'platform.libc_ver() returns incorrect version number'
updated_at = <Date 2018-12-05.14:23:05.563>
user = 'https://github.com/ThomasWaldmann'

bugs.python.org fields:

activity = <Date 2018-12-05.14:23:05.563>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-17.10:25:38.814>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2016-03-12.00:36:03.132>
creator = 'Thomas.Waldmann'
dependencies = []
files = ['43672', '47755']
hgrepos = []
issue_num = 26544
keywords = ['patch', '3.6regression', '3.7regression']
message_count = 36.0
messages = ['261624', '261648', '261649', '270073', '270080', '270089', '319456', '319479', '321301', '321308', '321310', '321311', '321462', '322021', '322023', '323795', '323796', '323802', '323805', '323816', '323823', '323825', '324164', '324231', '324581', '324582', '324588', '324637', '324638', '324639', '324784', '324794', '325158', '325520', '330951', '331116']
nosy_count = 8.0
nosy_names = ['lemburg', 'doko', 'vstinner', 'benjamin.peterson', 'ned.deily', 'njs', 'serhiy.storchaka', 'Thomas.Waldmann']
pr_nums = ['7684', '8193', '8195', '8196', '8356', '8952', '8970', '8971', '8973', '9061', '9062', '10868']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26544'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

thomaswaldmann · 2016-03-12T00:36:03Z

platform.libc_ver() is trivially broken as it uses string comparison internally to determine the maximum libc version number (which is obviously broken as "2.9" > "2.10").

malemburg · 2016-03-12T13:34:40Z

True. At the time the code was written, this was not an issue :-)

Is the libc version information documented somewhere ? If so, we could probably add a better parser for it.

malemburg · 2016-03-12T13:35:18Z

Adding other Python versions as well, since this is a bug.

njsmith · 2016-07-10T06:00:48Z

We just ran into this in pip -- pypa/pip#3836

I'd really recommend dropping the current "grovel through the binary doing a regex search" strategy -- it's incredibly error prone, and AFAICT doesn't really give any value. This code OTOH reliably lets you detect glibc and gives the exact version number with no fuss:
https://github.com/pypa/pip/blob/master/pip/utils/glibc.py#L9

What about non-glibc systems? Unfortunately the current libc_ver() turns out not to work well for those either. Attached is a CSV file showing the return value of ~1.2 billion calls to platform.libc_ver() by the last 6 months of pip users. You can see that the current code basically never returns anything useful for non-glibc platforms. (The one exception is that it seems to be able to detect uclibc 0.9.32 and label it as "libc 0.9.32".)

Don't get me wrong: it'd be really really useful if there were some way to detect and distinguish between the common non-glibc libcs like musl/bionic/uclibc/..., but I'm not sure how to do that -- and unfortunately the current code definitely doesn't do the job :-(.

malemburg · 2016-07-10T10:37:01Z

At the time the code was written, libc and glibc were in wide spread use, so it's not surprising that it doesn't work well for other C libs.

Note that the routine returns the highest libc version number used and required by the executable (usually the Python interpreter). This does not necessarily correspond to the version installed on the system. The purpose of the function was to determine the minimum libc compatibility requirements of the executable.

The routine you quote uses ctypes and only works for glibc, so parsing needs to be kept around as fallback solution. It also returns the libc version that is currently used on the system; not necessarily the minimum version required, so semantics are different.

njsmith · 2016-07-10T15:14:09Z

The purpose of the function was to determine the minimum libc compatibility requirements of the executable.

For what it's worth, I didn't know this, the pip authors obviously didn't know this, and after rereading the docs just now I still can't quite tell that this was intended. I'm not sure why one would want these semantics either, but at a minimum it would help to document the intended semantics much more clearly.

parsing needs to be kept around as fallback solution

The point of the data I attached was that AFAICT there don't exist any currently-in-use, non-glibc systems where "parsing" returns a meaningful result.

serhiy-storchaka · 2018-06-13T12:34:54Z

I need the glibc version for skipping a test if run with glibc containing a bug (see bpo-31630 and bpo-33630). platform.libc_ver() is not usable, it always returns '2.9' (the version that has bugs), while the actual version on my computer is '2.25' (the version that doesn't have these bugs).

serhiy-storchaka · 2018-06-13T17:50:38Z

I agree that the strategy used for platform.libc_ver() is not perfect. But the implementation has bugs that make it useless. The following PR fixes two bugs in the implementation:

Version numbers compared as strings.
Versions that are located on the border of 16 KiB blocks were not recognized or were recognized incorrectly.

serhiy-storchaka · 2018-07-09T08:47:49Z

New changeset 2a9b8ba by Serhiy Storchaka in branch 'master':
bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684)
2a9b8ba

serhiy-storchaka · 2018-07-09T09:55:38Z

New changeset 7c43b80 by Serhiy Storchaka in branch '3.7':
[3.7] bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684). (GH-8193)
7c43b80

serhiy-storchaka · 2018-07-09T11:38:31Z

New changeset d73497b by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684). (GH-8193) (GH-8195)
d73497b

serhiy-storchaka · 2018-07-09T11:39:10Z

New changeset b1e6e56 by Serhiy Storchaka in branch '2.7':
bpo-26544: Fixed implementation of platform.libc_ver(). (GH-7684). (GH-8193) (GH-8196)
b1e6e56

vstinner · 2018-07-11T14:39:39Z

Can we close this issue? It seems like the bug has been fixed, no?

doko42 · 2018-07-20T15:46:15Z

please don't close this yet. the patch introduces a dependency on the distutils package. I don't see any reason to compare the libc version as a python egg version, so please avoid that.

if we need to have a module to compare arbitrary version strings, then we should think about adding one in the standard library, but I think that's what the packaging module is for.

doko42 · 2018-07-20T15:55:14Z

a tuple comparison should be good enough

doko42 · 2018-08-20T15:33:07Z

no, it's not fixed at all. Setting as a release blocker for 3.6 and 3.7.

doko42 · 2018-08-20T16:00:57Z

proposed patch attached

serhiy-storchaka · 2018-08-20T17:22:28Z

platform-no-distutils.diff restores the original bug: "2.9" > "2.10".

PR 8356 removes the dependency from distutils and use a sophisticated function for parsing versions.

doko42 · 2018-08-20T17:57:12Z

fine! what prevents merging and backporting this issue?

serhiy-storchaka · 2018-08-21T03:38:31Z

I'll merge it if it looks good to you.

malemburg · 2018-08-21T07:26:54Z

I added a comment to the PR, but other than that I think it's good to go.

doko42 · 2018-08-21T07:48:34Z

+1

serhiy-storchaka · 2018-08-27T10:29:56Z

New changeset 7d81e8f by Serhiy Storchaka in branch 'master':
bpo-26544: Get rid of dependence from distutils in platform. (GH-8356)
7d81e8f

vstinner · 2018-08-28T09:51:02Z

Would it be possible to add a few tests on platform._comparable_version()? It would make me more confortable for backports.

serhiy-storchaka · 2018-09-04T12:04:36Z

New changeset 7917aad by Serhiy Storchaka in branch 'master':
bpo-26544: Add test for platform._comparable_version(). (GH-8973)
7917aad

vstinner · 2018-09-04T12:32:51Z

Thanks for adding tests ;-) It now looks better to me ;-)

serhiy-storchaka · 2018-09-04T14:31:26Z

New changeset 20a8392 by Serhiy Storchaka in branch '3.7':
[3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356). (GH-8970)
20a8392

vstinner · 2018-09-05T14:46:01Z

New changeset 1a3eb12 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
[3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) (GH-8970) (GH-9061)
1a3eb12

vstinner · 2018-09-05T14:46:29Z

New changeset 9734024 by Victor Stinner (Miss Islington (bot)) in branch '2.7':
bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) (GH-8952)
9734024

vstinner · 2018-09-05T14:46:56Z

Thanks everybody! The issue should now be fixed in all supported branches ;-)

serhiy-storchaka · 2018-09-07T18:46:44Z

Thank you for merging PRs Victor, but seems you have merged wrong PR for 2.7.

vstinner · 2018-09-07T21:12:49Z

Thank you for merging PRs Victor, but seems you have merged wrong PR for 2.7.

Oh. I didn't notice that there were two PRs for 2.7. Do you want to rebase your PR 9062 on 2.7, or revert the commit that I merged? It's up to you ;-)

I reopen the issue.

It seems like PR 8971 is also open.

ned-deily · 2018-09-12T18:32:03Z

This is now only open and a release blocker for 2.7, correct?

serhiy-storchaka · 2018-09-17T10:25:39Z

Seems the only difference between PR 8952 and PR 9062 is additional tests. This is not important.

vstinner · 2018-12-03T15:49:34Z

New changeset 8687bd8 by Victor Stinner in branch '2.7':
bpo-26544: Make platform.libc_ver() less slow (GH-10868)
8687bd8

vstinner · 2018-12-05T14:23:05Z

bpo-26544: Make platform.libc_ver() less slow (GH-10868)
8687bd8

"Coarse benchmark on Fedora 29: 1.6 sec => 0.1 sec."

Oops, my benchmark in the commit message was wrong, it included the startup time. Correct benchmark says 44x faster, it's way better!

Python 2.7: [old_py2] 1.51 sec +- 0.03 sec -> [if_in] 34.6 ms +- 0.4 ms: 43.61x faster (-98%)

thomaswaldmann mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Mar 12, 2016

serhiy-storchaka added 3.7 (EOL) end of life 3.8 only security fixes labels Jun 13, 2018

doko42 added the release-blocker label Aug 20, 2018

vstinner closed this as completed Sep 5, 2018

vstinner reopened this Sep 7, 2018

serhiy-storchaka closed this as completed Sep 17, 2018

ezio-melotti transferred this issue from another repository Apr 10, 2022

platform.libc_ver() returns incorrect version number #70731

platform.libc_ver() returns incorrect version number #70731

Comments

thomaswaldmann mannequin commented Mar 12, 2016

thomaswaldmann mannequin commented Mar 12, 2016

malemburg commented Mar 12, 2016

malemburg commented Mar 12, 2016

njsmith commented Jul 10, 2016

malemburg commented Jul 10, 2016

njsmith commented Jul 10, 2016

serhiy-storchaka commented Jun 13, 2018

serhiy-storchaka commented Jun 13, 2018

serhiy-storchaka commented Jul 9, 2018

serhiy-storchaka commented Jul 9, 2018

serhiy-storchaka commented Jul 9, 2018

serhiy-storchaka commented Jul 9, 2018

vstinner commented Jul 11, 2018

doko42 commented Jul 20, 2018

doko42 commented Jul 20, 2018

doko42 commented Aug 20, 2018

doko42 commented Aug 20, 2018

serhiy-storchaka commented Aug 20, 2018

doko42 commented Aug 20, 2018

serhiy-storchaka commented Aug 21, 2018

malemburg commented Aug 21, 2018

doko42 commented Aug 21, 2018

serhiy-storchaka commented Aug 27, 2018

vstinner commented Aug 28, 2018

serhiy-storchaka commented Sep 4, 2018

vstinner commented Sep 4, 2018

serhiy-storchaka commented Sep 4, 2018

vstinner commented Sep 5, 2018

vstinner commented Sep 5, 2018

vstinner commented Sep 5, 2018

serhiy-storchaka commented Sep 7, 2018

vstinner commented Sep 7, 2018

ned-deily commented Sep 12, 2018

serhiy-storchaka commented Sep 17, 2018

vstinner commented Dec 3, 2018

vstinner commented Dec 5, 2018