classification
Title: at runtime, distutils uses buildtime files
Type: behavior Stage:
Components: Distutils2, Library (Lib) Versions: Python 3.2
process
Status: closed Resolution:
Dependencies: Superseder: Avoid parsing pyconfig.h and Makefile by autogenerating extension module
View: 9878
Assigned To: tarek Nosy List: a.badger, amcnabb, dmalcolm, eric.araujo, fdrake, tarek
Priority: normal Keywords:

Created on 2008-11-19 23:52 by a.badger, last changed 2010-10-21 22:49 by eric.araujo. This issue is now closed.

Messages (11)
msg76083 - (view) Author: Toshio Kuratomi (a.badger) * Date: 2008-11-19 23:52
When using some distutils functions, distutils attempts to use buildtime
files like Makefile and pyconfig*.h as data sources.  For instance, this
snippet::

  from distutils.command.install import install
  from distutils.core import Distribution
  dist = Distribution({"name": "foopkg"})
  cmd = install(dist)
  cmd.ensure_finalized()

There's two reasons this should change.

1) Some Linux distributions separate the python runtime and buildtime
files and put the buildtime files in a -devel package.  Depending on
these buildtime files means that the -devel package can be needed for
running python scripts.  For instance, here's the traceback that occurs
when the previous commands are run without python-devel on Fedora Linux::

  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python2.5/distutils/cmd.py", line 117, in
ensure_finalized
      self.finalize_options()
    File "/usr/lib/python2.5/distutils/command/install.py", line 273, in
finalize_options
      (prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
    File "/usr/lib/python2.5/distutils/sysconfig.py", line 493, in
get_config_vars
      func()
    File "/usr/lib/python2.5/distutils/sysconfig.py", line 352, in
_init_posix
      raise DistutilsPlatformError(my_msg)
  distutils.errors.DistutilsPlatformError: invalid Python installation:
unable to open /usr/lib/python2.5/config/Makefile (No such file or
directory)

2) keeping the information in a Makefile and *.h files and then having
regular expressions pull the information out is fragile and not what the
tools were meant for.  Using a defined data format is much better.

The variables necessary for building extensions should be placed in a
data file of some sort.  This can be built by the configure script at
the same time as it's substituting variables into the Makefile and
pyconfig files.

xml is good for interoperability and we have good modules in the std
library for that now.  .ini is less verbose and we have modules to deal
with that as well.
msg94707 - (view) Author: Andrew McNabb (amcnabb) Date: 2009-10-30 15:21
I've noticed this, too, and I agree with Toshio's observations.  Tarek,
do you have any opinions?
msg94711 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2009-10-30 15:54
For Fedora, I've fixed this (I hope) by special-casing those two files:
 %{_libdir}/python%{pybasever}/config/Makefile
  /usr/include/python2.6/pyconfig-{32|64}.h
making them part of the core python package.

See downstream bug here:
https://bugzilla.redhat.com/show_bug.cgi?id=531901

(BTW, downstream, we rename pyconfig.h and generate a stub that #includes either pyconfig-
32/64 so that both 32 and 64-bit devel packages can be installed on 32-bit hosts; we patch 
distutils so that it parses the wordlength-specific file)
msg94712 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2009-10-30 15:56
> For Fedora, I've fixed this (I hope) by special-casing those two files:
Sorry; for "fixed", read "addressed"; this covers part 1 of the issue, 
but not part 2.
msg95269 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-11-15 01:11
This is a problem indeed. 

One solution would be to generate a module in the stdlib that contains
all these info, when configure is called.

as a matter of fact, I am currently working in a branch to add a module
called "sysconfig" to the stdlib, that contains installation paths
extracted from distutils/sysconfig and site.py, so the stdlib has only
one place to handle those.

This module will basically be the last spot to look for data in makefile
and pyconfig.h, so maybe we could inject in it a condensed version of
these files, in the form of a dict.

(I'll send a mail on python-dev about this)
msg95271 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-11-15 02:16
see http://mail.python.org/pipermail/python-dev/2009-November/094232.html

(notice that the dependency in install can be removed easily because it
just reads variables from sys and does not require to import sysconfig)
msg96578 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-12-18 19:21
I am closing this since Fedora has fixed the issue on their side.

On a side note: I am working on this new sysconfig module, in a branch
called tarek_sysconfig
msg96594 - (view) Author: Toshio Kuratomi (a.badger) * Date: 2009-12-19 01:42
Hey tarek, the main thrust of this bug for me was storing the data in an
inappropriate format and not having an API to get at it; things that I
think the sysconfig branch will address.  Does it make sense to have a
bug to track that progress?  Does it make sense for that bug to be this
one or a new one?
msg115842 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-08 02:11
I think the future sysconfig module (http://bitbucket.org/tarek/distutils2/src/tip/docs/design/wiki.rst) addresses part of the issue: It will use a configparser file to store installation directories. There is nothing about other build-time variables, though; this could be added to finally remove the brittle Makefile parsing in sysconfig.

There is no bug to track this work that I know of.
msg116145 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-12 01:08
#9807 has been recently opened.
msg119348 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-10-21 22:49
... and #9807 spawned #9878.
History
Date User Action Args
2010-10-21 22:49:06eric.araujosetsuperseder: deriving configuration information for different builds with the same prefix -> Avoid parsing pyconfig.h and Makefile by autogenerating extension module
messages: + msg119348
2010-09-13 01:07:47eric.araujosetsuperseder: deriving configuration information for different builds with the same prefix
2010-09-12 01:08:58eric.araujosetmessages: + msg116145
2010-09-08 02:11:35eric.araujosetversions: + Python 3.2, - Python 2.6, Python 3.0, Python 3.1, Python 2.7
nosy: + fdrake, eric.araujo

messages: + msg115842

components: + Library (Lib), Distutils2, - Distutils
2009-12-19 01:42:09a.badgersetmessages: + msg96594
2009-12-18 19:21:44tareksetstatus: open -> closed

messages: + msg96578
2009-11-15 02:16:44tareksetmessages: + msg95271
2009-11-15 01:12:01tareksetmessages: + msg95269
2009-10-30 15:56:33dmalcolmsetmessages: + msg94712
2009-10-30 15:54:42dmalcolmsetnosy: + dmalcolm
messages: + msg94711
2009-10-30 15:21:30amcnabbsetnosy: + amcnabb
messages: + msg94707
2009-02-06 09:17:53tareksetassignee: tarek
nosy: + tarek
versions: + Python 2.6, Python 3.0, Python 3.1, Python 2.7, - Python 2.5
2008-11-19 23:52:17a.badgercreate