Hello, guys.  I am looking for feedback on the topic which I am going to
lay out.  First, sys.path:

-------------------
# python2.4
>>> import sys
>>> sys.path
['', '/home/rudd-o', '/usr/lib64/python24.zip', '/usr/lib64/python2.4',
'/usr/lib64/python2.4/plat-linux2', '/usr/lib64/python2.4/lib-tk',
'/usr/lib64/python2.4/lib-dynload',
'/usr/lib64/python2.4/site-packages',
'/usr/lib64/python2.4/site-packages/Numeric',
'/usr/lib64/python2.4/site-packages/PIL',
'/usr/lib64/python2.4/site-packages/gtk-2.0',
'/usr/lib/python2.4/site-packages', '/usr/lib64/site-python',
'/usr/lib/site-python']
------------------

Now: Currently, bdist_rpm target builds RPMs that install to
/usr/lib(64)/python2.X/site-packages/.  This is wasteful and inefficient
in the case of pure python modules for the following reasons:

1. It necessitates a rebuild of the python modules for at least 2
architectures -- 64-bit and 32-bit, since they go to different
directories.  Worst part is, the packages themselves are
indistinguishable by name (they are built as Noarch in all
architectures) so a package built in 64bit cannot function in 32bit BUT
it is *installable* by RPM.
2. It necessitates a rebuild for every python minor version out there,
2.4, 2.5, 2.6.  This is also problematic because packages need to be
rebuilt for every distribution release out there, instead of being able
to build a single RPM that works across distro releases (or, the holy
grail, across distros altogether).

Now, I have familiarized myself with distutils enough that I can make a
quick patch that

- detects if the package is a pure python module
- ensures that the install-purelib directive in the [install] section of
setup.cfg is absent

and if both conditions are present, then the package is automatically
installed into /usr/lib/site-python.

The benefits are clear:

- only a single RPM needs to be built across distro versions and
architectures
- it enables cheese shop packages to be quickly built on a machine and
then installed in another
- it eliminates the need for the tedious process of writing custom spec
files
- packages built this way can be loaded and run by virtually all 2.x
python interpreters out there right now
- it would be possible to even build python RPMs that work across
distributions

This patch is no more than a two-liner, and I am looking for input, to
see if someone agrees, or not, to flesh out any pitfalls before going
in.  So do your best and provide sensible reasons why I should not do
this, or else I will do it and submit a patch.

Clock's ticking.

More info:

From my wicked noarch RPM compiled in an i386 machine, installed into my
64 bit centos (same OS):

[rudd-o@tobey ~]$ python
Python 2.4.5 (#1, Mar 13 2009, 12:13:36)
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import wicked
>>> wicked.__file__
'/usr/lib/site-python/wicked/__init__.pyc'

Evidence that the compiled file is being not recompiled, and is being
used directly on import, even across architectures.

-------------

However, I have discovered that I cannot run the same pyc file across
interpreters, even though the py file would run, so I am stumped at
this.  The reason i am stumped is that I can find a
platform-independent, interpreter-independent dir (site-python), and a
platform-dependent, interpreter-dependent dir (site-packages), but NOT a
platform-independent, interpreter-dependent dir, which is the BIG issue
here if we want to at least avoid having to rebuild python packages for
different architectures with the SAME interpreter.

Ideas?  I am thirsty for more knowledge here and losing hope.

apparently, /usr/lib/python2.4/site-packages IS in the sys.path, which
would seem to indicate that python would actually load pure python
modules from there.

Which means that the only fixes that need to go within distutils would be:

1. a fix so in 64bit arches, pure python modules are STILL INSTALLED
within the /usr/lib/python2.X dir instead of lib64
2. making every RPM built out of a pure python module depend on the
specific Python interpreter it was built under, to a precision of
major.minor version.

Right?

This sort of busts my plan of providing universal RPM packages for all
distributions, BUT... BUT... at least means that a single package can
work in all platforms of all distributions that have a particular
version of the interpreter it was built in.  Which is a space / busywork
savings of at least platforms*distributions.

last comment, first line, I should amend it to say that even in 64bit
arches, the /usr/lib (NOT ilb64) sitepackages dir is available.

I would have to actually try this on the packages that I have, see how i
fare.

FYI: In RPM, the correct dependency to require a particulr python minor
version is:

python(abi) = 2.X

where X is the minor version.

FYI: http://rudd-o.com/new-projects/python-improvements

Apropos this bug, the issues are fleshed out here:

http://rudd-o.com/new-projects/python-improvements/how-to-slash-man-years-from-the-process-of-building-rpms-out-of-python-modules

ahaha! I have the patch that implements this functionality.

1) it renames the RPM package name to python-%packagename-py2.4 and and
its requirements as well.
2) it autooptimizes (-O1) any RPM built if no optimization has been
configured in setup.cfg, so it works with any SELinux-enabled or any
other MAC-enabled platforms.

patch against distutils of python 2.4.  works with latest setuptools
just fine.

Improved version of the autonaming patch, now makes rpmbuild accept
files that were installed with spaces on their names.

Hi Rudd-O. Sorry i didn't look at your patches and idea yet, I am
preparing Pycon.

Make sur you write your patches against Python truk, not Python 2.4.
For example the current has already -O enabled for build_rpm

patch does what others did, plus this time it lets the specfile
autodiscover the python abi so the name is correct regardless of against
whichever python interpreter the package is built.

about python trunk...

gimme some time to port them incrementally to all popular stable pythons
first, then to trunk.  will be glad to do this.

Now, by trunk, do you mean python 3.x?

Rudd-O, I am trying to see if we can build a team of people that could
work on python <-> RPM tasks to adress all related issues. I'll let you
know when it's set.

Excellent.

El Domingo 05 Abril 2009, 
=?utf-8?q?Tarek_Ziad=C3=A9_=3Creport=40bugs=2Epython=2Eorg=3E?=@psf.upfronthosting.co.za 
escribió:
> Tarek Ziadé <ziade.tarek@gmail.com> added the comment:
>
> Rudd-O, I am trying to see if we can build a team of people that could
> work on python <-> RPM tasks to adress all related issues. I'll let you
> know when it's set.
>
> ----------
> priority:  -> high
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue5482>
> _______________________________________

-- 

	Manuel Amador (Rudd-O) <rudd-o@rudd-o.com>
	Rudd-O.com - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: UB40 - I would do for you
We are not loved by our friends for what we are; rather, we are loved in
spite of what we are.
		-- Victor Hugo

I saw your talk.  Excellent.  Let's make sysadmins and release managers
happy while keeping the excellent buildout infrastructure for
developers.  I KNOW we can do it, and I'm going to help you get there. 
 And maybe next year we both can do a talk about it all.  And kick
everybody's ass.

see msg108752

To complement Terry’s closing message: bdist_rpm is frozen in distutils, removed in distutils2/packaging, but has a new life at https://bitbucket.org/tarek/pypi2rpm/