Issue142

If you're reporting an issue for setuptools 0.7 or higher, please use BitBucket

Title easy_install ignores 'headers' passed to setup()
Priority bug Status chatting
Superseder Nosy List pje, tseaver
Assigned To Keywords

Created on 2012-07-09.15:13:45 by tseaver, last changed 2012-07-10.12:32:34 by tseaver.

Files
File name Uploaded Type Edit Remove
setuptools-issue_142-part_one.patch tseaver, 2012-07-09.18:21:22 text/plain
Messages
msg685 (view) Author: tseaver Date: 2012-07-10.12:32:33
That code does work for the case I'm currently working on (breaking out the 'persistent' package into its own project, and making 'ZODB3' depend on it).

I would have preferred a pkg_resources API which gave me direct access to the "root" of an installed distribution:  include paths are normally going to be relative to there, rather than to a given Python package.  E.g., the BTrees
code in ZODB3 uses:

  #include <persistent/cPersistence.h>

I think the same will apply to packages like SciPy -> NumPy.
msg684 (view) Author: pje Date: 2012-07-10.04:19:04
Yeah, like that, except I'm not sure I'd make '..' the default; I'd probably want to require an explicit path.  Did that code work for you?
msg683 (view) Author: tseaver Date: 2012-07-10.00:46:13
PJE wrote:


> Actually, it looks like it only requires that it be a list, but the
> things in it only need to have __str__ in order to be usable, as they
> are formatted with "-I %s".  AFAICT, that's the only place the actual
> include paths are used.  So instead of a generator function, you need
> a delayed string type.

> If that works, then it should be possible to provide some sort of
> ModuleHeader() object in setuptools that does the same thing.

Maybe something like::

    import os

    from pkg_resources import require
    from pkg_resources import resource_filename

    class ModuleHeaderDir(object):

        def __init__(self, require_spec, where='..'):
            # By default, assume top-level pkg has the same name as the
            # upstream distribution.
            # Also assume that headers are located in the package dir, and
            # are meant to be included as follows:
            #    #include "upstream/header_name.h"
            self._require_spec = require_spec
            self._where = where

        def __str__(self):
            require(self._require_spec)
            return os.path.abspath(
                        resource_filename(self._require_spec, self._where))

And in the downstream package's setup.py::

    from setuptools import ModuleHeaderDir
    from setuptools import Extension
    from setuptools import setup

    setup(name='downstream',
          ext_modules = [Extension('_someExtension',
                            ['src/downstream/_someExtension.c'],
                            iinclude_dirs=[ModuleHeaderDir('upstream')])],
          #...
         )
msg682 (view) Author: pje Date: 2012-07-09.22:51:32
On Mon, Jul 9, 2012 at 5:31 PM, Tres Seaver wrote:
> I tried making a generator function for the 'include_dirs' argument to
> the Extension in the downstream setup.py, but distutils barfs if it is not a
> bare list of strings.

Actually, it looks like it only requires that it be a list, but the
things in it only need to have __str__ in order to be usable, as they
are formatted with "-I %s".  AFAICT, that's the only place the actual
include paths are used.  So instead of a generator function, you need
a delayed string type.

If that works, then it should be possible to provide some sort of
ModuleHeader() object in setuptools that does the same thing.
msg681 (view) Author: tseaver Date: 2012-07-09.21:31:05
Your proposed solution isn't workable, either:  the 'pkg_resources' code
can't find the upstream distribution at module scope (before calling setup),
which means that 'setup_requires' can't install it before it is needed.

I tried making a generator function for the 'include_dirs' argument to
the Extension in the downstream setup.py, but distutils barfs if it is not a
bare list of strings.

The only workaround I have found to date is to include the headers in the
downstream package (e.g., via 'svn:externals' hackery).
msg676 (view) Author: pje Date: 2012-07-09.20:25:04
Currently, easy_install only modifies files in the directories specified via the command line or configuration files; it is not a backwards-compatible change for it to install things someplace else.

More specifically: you can't properly *uninstall* an egg if its header files are installed elsewhere.  See issue41 for more discussion on the general problem of adding post-install operations like this to easy_install.  The critical issue is that there's no uninstall operation implemented, as compared to say pip or packaging.

If you need to access headers from another module, the only way to do this right now is to store the headers in a package directory, and then have the depending setup.py use pkg_resources.resource_filenmae() to fetch the header filename(s) for inclusion.  It's not a great approach, but it's the only thing that will work consistently w/the current architecture.

Notice, by the way that easy_install is an embedded command in many systems, which would break if it suddenly had system-wide side-effects.  Not the least of such embeddings is the implementation of the setup_requires and tests_require keywords, which invoke easy_install to fetch the needed eggs, and put them in a temporary location.  It simply won't work to have those things install system header files.

This is why easy_install doesn't do post-install scripts, and CAN'T do them without some explicit request for it to run said post-install scripts.
msg675 (view) Author: tseaver Date: 2012-07-09.18:21:22
First part:  get the 'headers' value mapped onto an 'EGG-INFO/headers.txt' file.
msg674 (view) Author: tseaver Date: 2012-07-09.18:04:48
I'm not sure why you believe it is not fixable.  For instance, I think
we could approach this in two steps:

- First, update 'egg_info' command such that it creates an
  'EGG-INFO/headers.txt' file whose contents are the values passed as
  'headers' (one per line).  While existing distributions would be missing
  this file, newly-created ones would have it.

- Then, when installing a distribution, check for a 'headers.txt' info file.
  If present, use its contents to drive the distutils 'install_headers'
  machinery.
msg672 (view) Author: pje Date: 2012-07-09.17:03:00
This isn't fixable within the current easy_install architecture, unfortunately, but is fixable in "packaging", at least in principle.
msg670 (view) Author: tseaver Date: 2012-07-09.15:16:33
Ref:  https://bitbucket.org/tarek/distribute/issue/295
msg669 (view) Author: tseaver Date: 2012-07-09.15:13:45
A bare distutils install of a package with 'headers' passed to 'setup()'
dispatches to 'install_headers', which copies the exported header files
into the system include path.

'easy_install' bypasses the distutils 'install()' when doing a normal
egg install, and does not re-implement the 'install_headers' dance.

Furthermore, the 'bdist_egg' it creates, even though it contains the
header files, has no metadata which would allow the installer to
discover that they should be copied.
History
Date User Action Args
2012-07-10 12:32:34tseaversetmessages: + msg685
2012-07-10 04:19:05pjesetmessages: + msg684
2012-07-10 00:46:14tseaversetmessages: + msg683
2012-07-09 22:51:32pjesetmessages: + msg682
2012-07-09 21:31:05tseaversetmessages: + msg681
2012-07-09 20:25:05pjesetmessages: + msg676
2012-07-09 18:21:23tseaversetfiles: + setuptools-issue_142-part_one.patch
messages: + msg675
2012-07-09 18:04:48tseaversetmessages: + msg674
2012-07-09 17:03:01pjesetnosy: + pje
messages: + msg672
2012-07-09 15:16:33tseaversetstatus: unread -> chatting
messages: + msg670
2012-07-09 15:13:45tseavercreate