The distribution name returned by platform.dist() depends on the order
of os.path.listdir( '/etc' ). It selects the first file matching the
regex r'(\w+)[-_](release|version)' and takes part of the file name
(i.e. matchResult.groups()[0]) as distribution name. But there are often
several files matching this pattern (at least on Fedora and Mandriva).
For instance, on a Mandriva 2007.1 official, I can see the following files:

[login@localhost ~]$ ls -l /etc/*-release
-rw-r--r-- 1 root root 137 jan 18  2007 /etc/lsb-release
lrwxrwxrwx 1 root root  16 oct  7 17:32 /etc/mandrakelinux-release ->
mandriva-r                                                             
                              elease
lrwxrwxrwx 1 root root  16 oct  7 17:32 /etc/mandrake-release ->
mandriva-releas                                                        
                                   e
-rw-r--r-- 1 root root  50 avr  2  2007 /etc/mandriva-release
lrwxrwxrwx 1 root root  16 oct  7 17:32 /etc/redhat-release ->
mandriva-release

Therefore, the result for platform.distrib()[0] could be mandriva,
mandrake, redhat or even lsb. The first match wins !

Ignoring symlinks could remove part of the problem. On Mandriva, it
would leave only lsb-release and mandriva-release. It is possible to
select the good one by ignoring lsb-release or by verifying the file's
content (mandriva-release has one line and lsb-release has several
lines). I do not know if the second method is portable.

Do you have a patch? That would help tremendously.

I think it is safe to ignore lsb-release. In fact, there seems to be a
command "lsb_release" that gives information about distribution. On my
SuSE box, this is what I get:

marvin:~# lsb_release -i
Distributor ID: SUSE LINUX
marvin:~# lsb_release -d
Description:    SUSE LINUX 10.1 (i586)
marvin:~# lsb_release -r
Release:        10.1

sapetnioc, can you check if this command exists on your system and if
so, it's output? platform.dist() can check for this command's existence
and if present, can perhaps use it to glean distro information. Please
let me know if you want to write the patch. I will do it otherwise.

I can easily do the patch to ignore symlinks and /etc/lsb-release but I am
not sure of the appropriate way to look for lsb_update command, is
distutils.spawn.find_executable( 'lsb_release' ) ok ?
If you need the patch earlier than begining of next week, you should do it.
Otherwise I can make it.

On Mandriva 2007.1, the command exists if the package "lsb-release" is
installed. I do not know if it is always installed (I selected a checkbox
"LSB" during install, this checkbox is unchecked by default).
The output is:

[yann@localhost ~]$ lsb_release -a
LSB Version:
lsb-3.1-ia32:lsb-3.1-noarch:core-3.0-ia32:core-3.0-noarch:core-3.1-ia32:core-3.1-noarch:cxx-3.0-ia32:cxx-3.0-noarch:graphics-3.0-ia32:graphics-3.0-noarch:lsb-3.0-ia32:lsb-3.0-noarch:qt4-3.1-ia32:qt4-3.1-noarch
Distributor ID: MandrivaLinux
Description:    Mandriva Linux
Release:        2007.1
Codename:       Official

[yann@localhost ~]$ lsb_release -i
Distributor ID: MandrivaLinux

[yann@localhost ~]$ lsb_release -d
Description:    Mandriva Linux

[yann@localhost ~]$ lsb_release -r
Release:        2007.1

On Fedora 4 (rather old) , the command is in the package "redhat-lsb".

yc176684:src$ lsb_release -a
LSB Version:    1.3
Distributor ID: FedoraCore
Description:    Fedora Core release 4 (Stentz)
Release:        4
Codename:       Stentz

yc176684:src$ lsb_release -i
Distributor ID: FedoraCore

yc176684:src$ lsb_release -d
Description:    Fedora Core release 4 (Stentz)

yc176684:src$ lsb_release -r
Release:        4

On Fedora 7, the command is in the package "redhat-lsb" and seem to be
installed by default.

gargamel:riviere% lsb_release -a
LSB
Version:    :core-3.1-ia32:core-3.1-noarch:
graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: Fedora
Description:    Fedora release 7 (Moonshine)
Release:        7
Codename:       Moonshine

gargamel:riviere% lsb_release -i
Distributor ID: Fedora

gargamel:riviere% lsb_release -d
Description:    Fedora release 7 (Moonshine)

gargamel:riviere% lsb_release -r
Release:        7

On 10/25/07, Raghuram Devarakonda < report@bugs.python.org> wrote:
>
>
> Raghuram Devarakonda added the comment:
>
> I think it is safe to ignore lsb-release. In fact, there seems to be a
> command "lsb_release" that gives information about distribution. On my
> SuSE box, this is what I get:
>
> marvin:~# lsb_release -i
> Distributor ID: SUSE LINUX
> marvin:~# lsb_release -d
> Description:    SUSE LINUX 10.1 (i586)
> marvin:~# lsb_release -r
> Release:        10.1
>
> sapetnioc, can you check if this command exists on your system and if
> so, it's output? platform.dist() can check for this command's existence
> and if present, can perhaps use it to glean distro information. Please
> let me know if you want to write the patch. I will do it otherwise.
>
> ----------
> nosy: +draghuram
>
> __________________________________
> Tracker <report@bugs.python.org >
> <http://bugs.python.org/issue1322 >
> __________________________________
>

On 10/25/07, Yann Cointepas <report@bugs.python.org> wrote:
> I can easily do the patch to ignore symlinks and /etc/lsb-release but I am
> not sure of the appropriate way to look for lsb_update command, is
> distutils.spawn.find_executable( 'lsb_release' ) ok ?

You can just execute the command and if there is any error, default to
the current code. There is no need to explicitly check for the
binary's existence.

> If you need the patch earlier than begining of next week, you should do it.
> Otherwise I can make it.

I don't "need" the patch :-). If you can't get to it some time next
week, I will try to come up with the patch.

>
> On Mandriva 2007.1, the command exists if the package "lsb-release" is
> installed. I do not know if it is always installed (I selected a checkbox
> "LSB" during install, this checkbox is unchecked by default).
> The output is:
>
> [yann@localhost ~]$ lsb_release -a
> LSB Version:
> lsb-3.1-ia32:lsb-3.1-noarch:core-3.0-ia32:core-3.0-noarch:core-3.1-ia32:core-3.1-noarch:cxx-3.0-ia32:cxx-3.0-noarch:graphics-3.0-ia32:graphics-3.0-noarch:lsb-3.0-ia32:lsb-3.0-noarch:qt4-3.1-ia32:qt4-3.1-noarch
> Distributor ID: MandrivaLinux
> Description:    Mandriva Linux
> Release:        2007.1
> Codename:       Official
>
> [yann@localhost ~]$ lsb_release -i
> Distributor ID: MandrivaLinux
>
> [yann@localhost ~]$ lsb_release -d
> Description:    Mandriva Linux
>
> [yann@localhost ~]$ lsb_release -r
> Release:        2007.1
>
> On Fedora 4 (rather old) , the command is in the package "redhat-lsb".
>
> yc176684:src$ lsb_release -a
> LSB Version:    1.3
> Distributor ID: FedoraCore
> Description:    Fedora Core release 4 (Stentz)
> Release:        4
> Codename:       Stentz
>
> yc176684:src$ lsb_release -i
> Distributor ID: FedoraCore
>
> yc176684:src$ lsb_release -d
> Description:    Fedora Core release 4 (Stentz)
>
> yc176684:src$ lsb_release -r
> Release:        4
>
> On Fedora 7, the command is in the package "redhat-lsb" and seem to be
> installed by default.
>
> gargamel:riviere% lsb_release -a
> LSB
> Version:    :core-3.1-ia32:core-3.1-noarch:
> graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: Fedora
> Description:    Fedora release 7 (Moonshine)
> Release:        7
> Codename:       Moonshine
>
> gargamel:riviere% lsb_release -i
> Distributor ID: Fedora
>
> gargamel:riviere% lsb_release -d
> Description:    Fedora release 7 (Moonshine)
>
> gargamel:riviere% lsb_release -r
> Release:        7
>
> On 10/25/07, Raghuram Devarakonda < report@bugs.python.org> wrote:
> >
> >
> > Raghuram Devarakonda added the comment:
> >
> > I think it is safe to ignore lsb-release. In fact, there seems to be a
> > command "lsb_release" that gives information about distribution. On my
> > SuSE box, this is what I get:
> >
> > marvin:~# lsb_release -i
> > Distributor ID: SUSE LINUX
> > marvin:~# lsb_release -d
> > Description:    SUSE LINUX 10.1 (i586)
> > marvin:~# lsb_release -r
> > Release:        10.1
> >
> > sapetnioc, can you check if this command exists on your system and if
> > so, it's output? platform.dist() can check for this command's existence
> > and if present, can perhaps use it to glean distro information. Please
> > let me know if you want to write the patch. I will do it otherwise.
> >
> > ----------
> > nosy: +draghuram
> >
> > __________________________________
> > Tracker <report@bugs.python.org >
> > <http://bugs.python.org/issue1322 >
> > __________________________________
> >
>
> Added file: http://bugs.python.org/file8609/unnamed
>
> __________________________________
> Tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1322>
> __________________________________
>

Ony my Ubuntu box lsb_release is just a small Python script that parses
/etc/lsb-release. I suggest that your read the LSB standards about the
file and use the information to parse it instead of invoking a program.

> Ony my Ubuntu box lsb_release is just a small Python script that parses
> /etc/lsb-release. I suggest that your read the LSB standards about the
> file and use the information to parse it instead of invoking a program.

Can you please check if it supports all the options mentioned
previously in the bug report? I agree that reading from a file is
preferable to running a command, if issues OP mentioned can be
addressed.

heimes@seneca:~$ /usr/bin/lsb_release -a
LSB Version:   
core-2.0-noarch:core-3.0-noarch:core-3.1-noarch:core-2.0-ia32:core-3.0-ia32:core-3.1-ia32:cxx-2.0-noarch:cxx-3.0-noarch:cxx-3.1-noarch:cxx-2.0-ia32:cxx-3.0-ia32:cxx-3.1-ia32:graphics-2.0-noarch:graphics-3.0-noarch:graphics-3.1-noarch:graphics-2.0-ia32:graphics-3.0-ia32:graphics-3.1-ia32:desktop-3.1-noarch:desktop-3.1-ia32
Distributor ID: Ubuntu
Description:    Ubuntu 7.10
Release:        7.10
Codename:       gutsy
heimes@seneca:~$ /usr/bin/lsb_release -i
Distributor ID: Ubuntu
heimes@seneca:~$ /usr/bin/lsb_release -d
Description:    Ubuntu 7.10
heimes@seneca:~$ /usr/bin/lsb_release -c
Codename:       gutsy
heimes@seneca:~$ /usr/bin/lsb_release -r
Release:        7.10

Please read http://linux.die.net/man/1/lsb_release. It explains the
content of /etc/*-release in detail.

I am writing a patch but I have a few questions:

1) There are at most three places where the distribution name can be
found. What is the priority order to select only one name ?
The three places are:
  a) Inside the /etc/lsb-release file
  b) In the name of the /etc/<distrib>-release file
  c) In the content of the /etc/<distrib>-release file
For instance, on Mandriva 2007.1 the possible names are:
  a) 'MandrivaLinux'
  b) 'mandriva'
  c) 'Mandriva Linux'
I would suggest to put a) first to be compatible with LSB
but on most systems it would change the value returned by
platform.dist after the patch (is it a problem ?).
I would have liked to use c) as second choice but this space
in the name set by Mandriva could be a problem (It's possible
to suppress spaces in the result though).

2) Can I remove supported_dists parameter of platform.dist ?
There could be a list of supported distributions but why
as a parameter of this function ?

> I am writing a patch but I have a few questions:
> 
> 1) There are at most three places where the distribution name can be
> found. What is the priority order to select only one name ?
> The three places are:
>   a) Inside the /etc/lsb-release file
>   b) In the name of the /etc/<distrib>-release file
>   c) In the content of the /etc/<distrib>-release file

As far as I remember the specs a /etc/*-release file has a higher
priority than /etc/lsb-release.

> 2) Can I remove supported_dists parameter of platform.dist ?
> There could be a list of supported distributions but why
> as a parameter of this function ?

I agree. A module global list is better than a list as a function argument.

Can you also use a global variable instead of "/etc"? Something like
ETC_DIR = "/etc" for example. It would allow you to ship samples from
several distribution and run unit tests against each.

Christian

> Can you also use a global variable instead of "/etc"? Something like
> ETC_DIR = "/etc" for example. It would allow you to ship samples from
> several distribution and run unit tests against each.

I've attached the two relevant files from Ubuntu 7.10 Gutsy.

I joined a modified version of platform.py. Here is a summary of the
modification:
- replaced '/etc' by global variable _etc_dir
- removed supported_dists parameter of dist()
- each element of dist() result is searched in the following places
  (the first value found is kept):
  1) First line of /etc/<distribname>-release
  2) Contents of /etc/lsb-release
  3) Extract <distribname> from /etc/<distribname>-release

I tested it on Mandriva 2007.1, Fedora 4 and on a faked /etc directory
containing the Ubuntu files send by tiran. Results are:
  Mandriva 2007.1: ('MandrivaLinux', '2007.1', 'Official')
  Fedora 4: ('FedoraCore', '4', 'Stentz')
  Faked Ubuntu: ('Ubuntu', '7.10', 'gutsy')

Yann Cointepas wrote:
> I joined a modified version of platform.py. Here is a summary of the
> modification:
> - replaced '/etc' by global variable _etc_dir
> - removed supported_dists parameter of dist()
> - each element of dist() result is searched in the following places
>   (the first value found is kept):
>   1) First line of /etc/<distribname>-release
>   2) Contents of /etc/lsb-release
>   3) Extract <distribname> from /etc/<distribname>-release

Here is an updated patch against Python 2.5 (your file was based on
Python 2.5's platform.py, wasn't it?).

CHANGES:

 * Added unit tests
 * Add Lib/test/platform directory for unit tests
 * removed usage of string
 * minor cleanups

Now for the funny part. You have to port the patch to the svn trunk. The
file can't be altered in the 2.5 release cycle. The changes are too
great. Unfortunately the patch doesn't apply cleanly.

Please add more test data to the platform/ directory and test_platform.py

In order to build the trunk:
svn co http://svn.python.org/projects/python/trunk
cd trunk
./configure
make
patch -p0 < patchfile
./python Lib/test/test_platform.py
svn diff > patchfile

Christian

I'm mentoring a task for GHOP which is going to fix the problem.

I'm work on this issue in
GHOP(http://code.google.com/p/google-highly-open-participation-psf/issues/detail?id=216&can=1&colspec=ID%20Status%20ClaimedBy%20Due%20NeedsReview%20Summary)
I'm attach updated patch for python trunk. This patch fixes issue and
add additional test for some other Linux distributions.

Thanks Pavel!
First I need confirmation from the GHOP project and Georg that you have
submitted the contributor form. Then I'm going to merge your patch.

You can see confirmation from Georg on thread in GHOP:
http://code.google.com/p/google-highly-open-participation-psf/issues/detail?id=216&can=1&colspec=ID%20Status%20ClaimedBy%20Due%20NeedsReview%20Summary#c20

I can update patch for 3.0 (it don't applies now) if you are ready to
commit them.

In this fix I removed the use of the file name in order to return the
distname. Now, only the file contents
is taken into account.

On Centos, the file name is the same as on Redhat, but its contents is
different:
$ cat /etc/redhat-release 
CentOS release 5 (Final)

Compare:

With test_platform_py26.diff:
Python 2.6a3+ (trunk:62996M, May 10 2008, 16:38:41) 
[GCC 4.1.2 20070626 (Red Hat 4.1.2-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.dist()
('redhat', '5', 'Final')
>>> 

With the new fix:
Python 2.6a3+ (trunk:62996M, May 10 2008, 16:38:41) 
[GCC 4.1.2 20070626 (Red Hat 4.1.2-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.dist()
('CentOS', '5', 'Final')
>>> 

I have tested this fix on Ubuntu, Centos5 and RHEL 5 and it works fine.
Ubuntu: ('Ubuntu', '8.04', 'hardy')
Fedora: ('fedora', '8', 'Werewolf')

Christian was reviewing this for GHOP.

Please see the top of platform.py:

#    This module is maintained by Marc-Andre Lemburg <mal@egenix.com>.
#    If you find problems, please submit bug reports/patches via the
#    Python SourceForge Project Page and assign them to "lemburg".
#
#    Note: Please keep this module compatible to Python 1.5.2.

I wonder why the ticket wasn't assigned to me.

Regarding the patch: 

 * the dist() function has been superseded by linux_distribution().

 * the supported_dists argument is a documented feature of
linux_distributions(), so should not be removed

 * the Windows name normalization should not be removed (it fixes a bug
in Vista)

 * the change from using string functions to using string methods breaks
1.5.2 compatibility (*)

 * the _ETC_DIR global just makes things harder to read and there's no
apparent need for it

(*) It's probably time to drop 1.5.2 compatibility and only keep the
module compatible to Python 2.1, so this is not much of an issue.

Overall, I think it's better to define a fixed search order for the
release files than to try to figure out and parse random release files
that happen to match the release file RE.

For that to work, it would help a lot if you could provide the file name
and contents of various platform release files.

Also note that linux_distribution() will use the parsed distro name from
the release file per default (full_distribution_name=1), so the problem
described in the original ticket description should no longer be
relevant: all release files point to the same lsb-release file in the end.

I don't have access to Mandriva. Could you please let me know whether
the current SVN version of platform.py still exhibits the mentioned
problem ?

Thanks.

Please see also #3937 for a patch which first tries to parse
"/etc/lsb-release", then tries to execute "lsb_release", then falls back
to the old behavior of platform.dist().  (Note that parsing the file
named /etc/lsb-release is not guaranteed to work by the Linux Standard
Base spec, but that executing "lsb_release" is.  On the other hand, the
former currently seems to work on more Debian/Ubuntu installations than
the latter does, and invoking lsb_release in a subprocess takes
significantly more time.)

Please see the links to the Linux Standard Base specifications which I
posted in #3937.  (The specification was originally published in 2001,
so most of the Linux systems that future versions of Python will be
deployed on were developed long after that specification was written.)

My patch, in #3937, is against the python-release25-maint branch.  I
tested the patch locally and it worked, and then I wrote a patch for my
project -- allmydata.org "Tahoe" -- which does something similar:

http://allmydata.org/trac/tahoe/browser/src/allmydata/__init__.py?rev=2976

I committed that change to Tahoe, and it was automatically tested on
eleven different systems by our buildbot:

http://allmydata.org/buildbot/waterfall

The resulting distribution-identification can be seen in the logs of
those tests.  For any machine on the waterfall, click on the "test.log"
hyperlink and search in text for "tahoe versions:".  For example, here
is the one for our Debian etch machine -- "debian", "4.0":

http://allmydata.org/buildbot/builders/etch/builds/1205/steps/test/logs/test.log

here is the one for the Ubuntu dapper machine -- "Ubuntu", "6.06":

http://allmydata.org/buildbot/builders/dapper/builds/1773/steps/test/logs/test.log

and here is the result for my Ubuntu hardy workstation -- "Ubuntu", "8.04":

http://allmydata.org/buildbot/builders/zooko%20yukyuk%20hardy/builds/216/steps/test/logs/test.log

Zooko, I think the main reason for the parser in platform.py to fail on
Ubuntu is that Ubuntu doesn't ship with a /etc/ubuntu-release file and
instead uses the optional override parameters in /etc/lsb-release to
setup the distribution values.

E.g. on my Ubuntu box it has these lines:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=7.10
DISTRIB_CODENAME=gutsy
DISTRIB_DESCRIPTION="Ubuntu 7.10"

I guess we'll need to add a special parser for lsb-release files which
then gets tried *after* having tried the /etc/<distro>-release files
(this is how the lsb_release command works as well) and overrides any
settings found in the <distro>-release file.

If that doesn't work either, the function will need to fallback to
_dist_try_harder().

There's no need to start a subprocess for running lsb_release. 

Please also note that the subprocess module is not available in Python
2.1, so it's not an option anyway (platform.py defines it's own popen()
helper which should be used instead).

Okay, per MAL's request over on #3937, I tried
platform.get_linux_distribution() on the current svn trunk (which I
assume is the version that is about to become python 2.6).  It gave the
same not-so-great answer as platform.dist() used to: ('debian',
'lenny/sid', '').  I then ported my patch to trunk (attached), and it
gave the answer I wanted: ('Ubuntu', '8.04', 'hardy').

Note that technique of parsing /etc/lsb-release or invoking lsb_release
is more standard and generic and easier to maintain than the current
technique of trying a series of specific file parses for specific
"supported" distributions.  For example, if some new Linux distribution
is invented, or if this is tried on a distribution that isn't in the
supported list (has anyone tried this on Foresight Linux yet?), then my
technique has a good chance of working on that distribution, where the
current technique has a higher chance of accidentally mis-identifying it
as some other distribution, for example the way that the current trunk
mis-identifies Ubuntu 8.04 as Debian lenny/sid.

MAL:  why do you say it is better to look for
/etc/$supportedplatform-release files first instead of looking for
/etc/lsb-release first?

I do not know if /etc/lsb-release is suitably generic -- I've tried it
only on a few platforms.  I do know that executing lsb_release is
suitably generic since it is standard, but I prefer not to try it first
since it imposes about half-a-second delay.

On 2008-09-23 21:49, Zooko O'Whielacronx wrote:
> Zooko O'Whielacronx <zooko@zooko.com> added the comment:
> 
> MAL:  why do you say it is better to look for
> /etc/$supportedplatform-release files first instead of looking for
> /etc/lsb-release first?

Because that's exactly what lsb_release does as well. The data
in /etc/lsb-release can only override data already parsed from
the /etc/<distro>-release file.

> I do not know if /etc/lsb-release is suitably generic -- I've tried it
> only on a few platforms.  I do know that executing lsb_release is
> suitably generic since it is standard, but I prefer not to try it first
> since it imposes about half-a-second delay.

lsb_release is standard on LSB compliant Linuxes, but the much
older /etc/<distro>-release file approach is still valid and in
wide use.

E.g. on SuSE, /etc/lsb-release doesn't contain any usable
distribution information. On Fedora, that file doesn't exist at
all.

It's better to follow the approach taken by lsb_release and then
add calling lsb_release as one of the methods taken by
_dist_try_harder() (using platform.popen()) should the parsers
fail.

This avoids spawning a process in most cases.

> Because that's exactly what lsb_release does as well.

You must know something about common lsb_release implementations that I
don't.  As far as I saw in the LSB documentation, it is required to
print out information in a certain format, but how it is implemented is
totally up to the distribution in question.

You give examples of SuSE and Fedora as not having /etc/lsb-release
files, and I'm sure you are right, but I happen to know that both of
them have compliant lsb_release executables (and that they have had for
many releases).  So, the patch that I've submitted will definitely work
correctly for those two distributions, although it will pay the price of
having to spawn a subprocess and then wait for the lsb_release
executable to do its work (however it does it).

However, presumably your SuSE- and Fedora- specific techniques will give
correct answers on those platforms faster than the generic lsb_release
would.

On 2008-09-23 22:19, Zooko O'Whielacronx wrote:
> Zooko O'Whielacronx <zooko@zooko.com> added the comment:
> 
>> Because that's exactly what lsb_release does as well.

I have to correct that: lsb_release will only look at the other
release files in case it doesn't already enough information from
the lsb-release file.

> You must know something about common lsb_release implementations that I
> don't.  As far as I saw in the LSB documentation, it is required to
> print out information in a certain format, but how it is implemented is
> totally up to the distribution in question.

Just do a "man lsb_release" or look at the lsb_release shell script.

> You give examples of SuSE and Fedora as not having /etc/lsb-release
> files,

Fedora doesn't have that file, so lsb_release has to read the results
from /etc/fedora-release. SuSE does, but doesn't override
the default set in /etc/SuSE-release.

> and I'm sure you are right, but I happen to know that both of
> them have compliant lsb_release executables (and that they have had for
> many releases).  So, the patch that I've submitted will definitely work
> correctly for those two distributions, although it will pay the price of
> having to spawn a subprocess and then wait for the lsb_release
> executable to do its work (however it does it).
> 
> However, presumably your SuSE- and Fedora- specific techniques will give
> correct answers on those platforms faster than the generic lsb_release
> would.

Yep and the same is true for all other _supported_dists. I always try
to avoid spawning external processes whenever I can.

Well, for what it is worth I've updated the custom "detect linux
distribution" code in Tahoe yet again.  The current version first tries
to parse /etc/lsb-version (fast, gives a good answer on Ubuntu, and
hopefully at least semi-standardized), then invokes platform.dist()
(fast, customized, but gives a bad answer on Ubuntu), and then if
platform.dist() returns an empty distribution or empty release (which
I'm not sure if that is even possible), it invokes lsb_release -a in a
subprocess (slow, totally standardized).

Here's the patch:

http://allmydata.org/trac/tahoe/changeset/2981

Running it on the Tahoe buildbot shows that we currently have the
followings kinds of buildslaves:

http://allmydata.org/buildbot/waterfall?show_events=true

Linux-Ubuntu_6.06-i686-32bit, Linux-Ubuntu_6.10-i686-32bit,
Linux-Ubuntu_7.10-i686-32bit, SunOS-5.11-i86pc-i386-32bit,
Linux-Ubuntu_7.04-i686-32bit, Linux-debian_4.0-i686-32bit,
Linux-Ubuntu_8.04-i686-32bit, Darwin-9.4.0-i386-32bit,
Linux-Ubuntu_8.04-x86_64-64bit, Windows-XP-5.1.2600-SP2,
CYGWIN_NT-5.1-1.5.25-0.156-4-2-i686-32bit-WindowsPE

/etc/lsb-release is not yet specified by the lsb.

See
https://lists.linux-foundation.org/pipermail/lsb-discuss/2008-March/004842.html
https://lists.linux-foundation.org/pipermail/lsb-discuss/2008-March/004843.html
https://lists.linux-foundation.org/pipermail/lsb-discuss/2008-March/004845.html

MAL, please can we add zooko's patch in some form? The current
implementation assumes an implementation, which doesn't exist on all
platforms, and just dividing linux distributions in "unsupported" and
"supported" seems to be odd.

You cite some URL's in platform.py:

 - http://linuxmafia.com/faq/Admin/release-files.html
 
   "Linux System Base-compliant systems should have a file called
   /etc/lsb_release, which may be in addition to a distribution-
   specific file."

   The standard doesn't say anything like this.

 - http://linux.die.net/man/1/lsb_release
   This is one implementation, not more. The current platform.py
   implementation seems to require this particular implementation.

The only relevant URL is missing in this file:
http://refspecs.freestandards.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/lsbrelease.html

doko: thanks for your interest encouraging more formal and generic
solutions to this.


For what it is worth, the current version of my patch (used in Tahoe) is
here:

http://allmydata.org/trac/tahoe/browser/src/allmydata/__init__.py?rev=20081125155118-92b7f-f74fc964ebd9d3c59afde68b6688c56ce20cca39#L31

I had to add a special case for Arch Linux, which gets triggered after
the three main cases.  The cases currently are, in order:

1.  Parse /etc/lsb-release (fast, semi-de-facto-standard, generic,
hopefully a future de-jure-standard).
2.  Invoke the Python Standard Library's platform.dist() (pros: fast,
has lots of customized special cases for different linux distros, cons:
has lots of customized special cases for different linux distros, gives
bogus answers for Ubuntu and Arch Linux)
3.  Subprocess execute "lsb_release" (pros: a real de-jure-standard!
cons: slow, and is not actually a de-facto-standard since many important
Linux installations don't come by default with the package that provides
the de-jure-standard "lsb_release" executable, even though they do come
by default with the de-facto-semi-standard "/etc/lsb-release" file).
4.  Arch Linux

On 2009-03-18 18:13, Matthias Klose wrote:
> Matthias Klose <doko@debian.org> added the comment:
> 
> MAL, please can we add zooko's patch in some form? The current
> implementation assumes an implementation, which doesn't exist on all
> platforms, and just dividing linux distributions in "unsupported" and
> "supported" seems to be odd.

I've already mentioned what needs to be done to make Zooko's patch
suitable for platform.py (see further up on this ticket discussion).

The patch still doesn't meet those requirement, so cannot be applied.

Furthermore, I would prefer that piping to lsb_release only be used
as fallback solution. This should not be the default mechanism.

Regarding the problems on Ubuntu: That's mainly due to the way Ubuntu
decided to use the LSB standard. Most other Linux platforms work
just fine with the existing platform.py parser.

In order to support Ubuntu as well, we'll have to parse the
DISTRIB_* "variables" in /etc/lsb-release.

> You cite some URL's in platform.py:
> 
>  - http://linuxmafia.com/faq/Admin/release-files.html
>  
>    "Linux System Base-compliant systems should have a file called
>    /etc/lsb_release, which may be in addition to a distribution-
>    specific file."
> 
>    The standard doesn't say anything like this.
> 
>  - http://linux.die.net/man/1/lsb_release
>    This is one implementation, not more. The current platform.py
>    implementation seems to require this particular implementation.
> 
> The only relevant URL is missing in this file:
> http://refspecs.freestandards.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/lsbrelease.html

Thanks, I'll add that. Note that the link doesn't mention anything
about the file format of the /etc/lsb_release file or how it should
be parsed. That's why I added the other URLs.

BTW: Here's what "man lsb_release" says on OpenSUSE:

"""
       If the installation is LSB  compliant,  the  "/etc/lsb-release"  file
should  contain  the
       LSB_VERSION  field.   The  value of the field should be a colon separated
list of supported
       module versions indicating the LSB specification modules to which the
installation is  com‐
       pliant. If the installation is not compliant, the above field should not
be present.

       Optional  fields are DISTRIB_ID, DISTRIB_RELEASE, DISTRIB_CODENAME,
DISTRIB_DESCRIPTION and
       can be used to override information which is parsed from the
"/etc/distrib-release" file.

       If the "/etc/lsb-release.d" directory exists, it is searched for
filenames which are  taken
       as additional module-version strings to add to LSB_VERSION.

       The "/etc/distrib-release" file contains a description line which is
parsed to get informa‐
       tion (especially on currently non-LSB compliant systems).
"""

It is interesting to note that Ubuntu's /etc/lsb_release file does not
contain the LSB_VERSION field. That would suggest, that Ubuntu is not
LSB compliant.

I just read back through this ticket, but I didn't understand exactly
what MAL wanted to have different from what this Python function
currently does:

http://allmydata.org/trac/tahoe/browser/src/allmydata/__init__.py?rev=20081125155118-92b7f-f74fc964ebd9d3c59afde68b6688c56ce20cca39#L31

MAL, could you please restate the changes you want?

By the way I think there is some confusion about what is standardized by
LSB.  As far as I know this one page is the complete spec:
http://refspecs.freestandards.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/lsbrelease.html
and it doesn't specify the existence of any file that we can parse.  So
the quote you mention: "Linux System Base-compliant systems should have
a file called /etc/lsb_release, which may be in addition to a
distribution-specific file." is just wrong.  More's the pity -- most
implementations use a file named /etc/lsb-release, and we can parse
that, and if we do it is much faster than executing the lsb_release
executable in a subprocess.  The slowness of invoking subprocess is why
I was forced to back off from my original patch of merely using only
what the LSB offers.

A second problem with relying on LSB is, as I've mentioned, that some
Linux distributions don't come (by default) with the de-facto-standard
"lsb_release" executable although they do come with an /etc/lsb-release
file.

That's why my current strategy is:

1.  Parse the de-facto-nearly-standard /etc/lsb-release file.

2.  Ad-hoc techniques encoded into the Python Standard Library's
platform.dist().

3.  Execute the de-jure-standard "lsb_release" in a subprocess.

4.  Arch Linux

Has there been any progress on incorporating the suggested here patch?  

A suggested patch is found here also:

http://code.google.com/p/google-highly-open-participation-psf/issues/detail?id=216


And a patch is applied to python in ubuntu packages:
see reference - https://bugs.launchpad.net/python/+bug/196526

But the original platform still doesn't support this and it can't be counted on. 
Will the patch given by zooko be applied? If not, what is missing in it?

For what it is worth, here is the current version of this code that we are using in Tahoe-LAFS:

http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/__init__.py?annotate=blame&rev=5033#L36

You can see the results on our buildbot:

http://tahoe-lafs.org/buildbot/waterfall?show_events=false

In the tahoe-version build step, e.g.:

http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/820/steps/tahoe-version/logs/stdio

Which says:

platform: Linux-debian_5.0.6-x86_64-64bit

[Zooko]
> I just read back through this ticket, but I didn't understand exactly
> what MAL wanted to have different from what this Python function
> currently does:

It may be this:

> It's better to follow the approach taken by lsb_release and then
> add calling lsb_release as one of the methods taken by
> _dist_try_harder() (using platform.popen()) should the parsers
> fail.
> This avoids spawning a process in most cases.

Éric Araujo wrote:
> 
> Éric Araujo <merwok@netwok.org> added the comment:
> 
> [Zooko]
>> I just read back through this ticket, but I didn't understand exactly
>> what MAL wanted to have different from what this Python function
>> currently does:
> 
> It may be this:
> 
>> It's better to follow the approach taken by lsb_release and then
>> add calling lsb_release as one of the methods taken by
>> _dist_try_harder() (using platform.popen()) should the parsers
>> fail.
>> This avoids spawning a process in most cases.

Indeed.

You also need to make sure that the function doesn't suddenly
return different output for setups that are supported and work
as advertised.

Note that the Debian mention in the string output is not really
incorrect, given that Ubuntu and other distros are based on the
work done by the Debian project.

Looking back, I shouldn't have added the function to begin
with and just use "Linux" in the platform string. It's causing
too much maintenance overhead.
Perhaps deprecating it altogether is more appropriate.

The hard part was supporting distro-specific release files; I think that now most of them provide the lsb_release info.  If it proves more complicated than that, then let’s deprecate the function.

There seems to be some mistake, re #msg134219 and #msg134255. The current version of may patch *does* avoid the cost of a subprocess in the common case. I described this new strategy in #msg73744 and as far as I know it satisfies all of MAL's earlier objection about that.

To recap, this code here: http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/__init__.py?annotate=blame&rev=5033#L36 does the following strategy:

1. Parse the /etc/lsb-release file. /etc/lsb-release is not part of the de jure standard, but it is a de facto standard that is available on many distributions. Parsing it is fast and gives the right answer on many distributions.

2. If that didn't work (which can happen on some distributions, including common ones when a certain optional "lsb base" package isn't installed), then invoke the current platform.dist() code. This is a lot of code, its code has to be changed before it can recognize any new distribution or a change in a distribution, and it gives answers on Ubuntu and Arch Linux which users say are the wrong answer, but it is fast and it gives the answer users want in most cases.

3. If that didn't work (which presumably only happens on distributions that the authors of platform.dist() didn't know about or didn't bother to support), then invoke the de jure standard executable "lsb_release". This works on any LSB-compliant system, but it costs a subprocess.

4. If that didn't work, check for /etc/arch-release to signal Arch Linux.

FTR, doko applied this patch for Debian and Ubuntu: http://patch-tracker.debian.org/patch/series/view/python3.2/3.2.1~rc1-1/platform-lsbrelease.diff

http://patch-tracker.debian.org/patch/series/view/python3.2/3.2.1~rc1-1/platform-lsbrelease.diff

This patch fails if "(?:DISTRIB_CODENAME\s*=)\s*(.*)" regex doesn't match (_u_id variable is not set).