That code does work for the case I'm currently working on (breaking out the 'persistent' package into its own project, and making 'ZODB3' depend on it).

I would have preferred a pkg_resources API which gave me direct access to the "root" of an installed distribution:  include paths are normally going to be relative to there, rather than to a given Python package.  E.g., the BTrees
code in ZODB3 uses:

  #include <persistent/cPersistence.h>

I think the same will apply to packages like SciPy -> NumPy.

Yeah, like that, except I'm not sure I'd make '..' the default; I'd probably want to require an explicit path.  Did that code work for you?

PJE wrote:


> Actually, it looks like it only requires that it be a list, but the
> things in it only need to have __str__ in order to be usable, as they
> are formatted with "-I %s".  AFAICT, that's the only place the actual
> include paths are used.  So instead of a generator function, you need
> a delayed string type.

> If that works, then it should be possible to provide some sort of
> ModuleHeader() object in setuptools that does the same thing.

Maybe something like::

    import os

    from pkg_resources import require
    from pkg_resources import resource_filename

    class ModuleHeaderDir(object):

        def __init__(self, require_spec, where='..'):
            # By default, assume top-level pkg has the same name as the
            # upstream distribution.
            # Also assume that headers are located in the package dir, and
            # are meant to be included as follows:
            #    #include "upstream/header_name.h"
            self._require_spec = require_spec
            self._where = where

        def __str__(self):
            require(self._require_spec)
            return os.path.abspath(
                        resource_filename(self._require_spec, self._where))

And in the downstream package's setup.py::

    from setuptools import ModuleHeaderDir
    from setuptools import Extension
    from setuptools import setup

    setup(name='downstream',
          ext_modules = [Extension('_someExtension',
                            ['src/downstream/_someExtension.c'],
                            iinclude_dirs=[ModuleHeaderDir('upstream')])],
          #...
         )

On Mon, Jul 9, 2012 at 5:31 PM, Tres Seaver wrote:
> I tried making a generator function for the 'include_dirs' argument to
> the Extension in the downstream setup.py, but distutils barfs if it is not a
> bare list of strings.

Actually, it looks like it only requires that it be a list, but the
things in it only need to have __str__ in order to be usable, as they
are formatted with "-I %s".  AFAICT, that's the only place the actual
include paths are used.  So instead of a generator function, you need
a delayed string type.

If that works, then it should be possible to provide some sort of
ModuleHeader() object in setuptools that does the same thing.

Your proposed solution isn't workable, either:  the 'pkg_resources' code
can't find the upstream distribution at module scope (before calling setup),
which means that 'setup_requires' can't install it before it is needed.

I tried making a generator function for the 'include_dirs' argument to
the Extension in the downstream setup.py, but distutils barfs if it is not a
bare list of strings.

The only workaround I have found to date is to include the headers in the
downstream package (e.g., via 'svn:externals' hackery).

Currently, easy_install only modifies files in the directories specified via the command line or configuration files; it is not a backwards-compatible change for it to install things someplace else.

More specifically: you can't properly *uninstall* an egg if its header files are installed elsewhere.  See issue41 for more discussion on the general problem of adding post-install operations like this to easy_install.  The critical issue is that there's no uninstall operation implemented, as compared to say pip or packaging.

If you need to access headers from another module, the only way to do this right now is to store the headers in a package directory, and then have the depending setup.py use pkg_resources.resource_filenmae() to fetch the header filename(s) for inclusion.  It's not a great approach, but it's the only thing that will work consistently w/the current architecture.

Notice, by the way that easy_install is an embedded command in many systems, which would break if it suddenly had system-wide side-effects.  Not the least of such embeddings is the implementation of the setup_requires and tests_require keywords, which invoke easy_install to fetch the needed eggs, and put them in a temporary location.  It simply won't work to have those things install system header files.

This is why easy_install doesn't do post-install scripts, and CAN'T do them without some explicit request for it to run said post-install scripts.

First part:  get the 'headers' value mapped onto an 'EGG-INFO/headers.txt' file.

I'm not sure why you believe it is not fixable.  For instance, I think
we could approach this in two steps:

- First, update 'egg_info' command such that it creates an
  'EGG-INFO/headers.txt' file whose contents are the values passed as
  'headers' (one per line).  While existing distributions would be missing
  this file, newly-created ones would have it.

- Then, when installing a distribution, check for a 'headers.txt' info file.
  If present, use its contents to drive the distutils 'install_headers'
  machinery.

This isn't fixable within the current easy_install architecture, unfortunately, but is fixable in "packaging", at least in principle.

Ref:  https://bitbucket.org/tarek/distribute/issue/295

A bare distutils install of a package with 'headers' passed to 'setup()'
dispatches to 'install_headers', which copies the exported header files
into the system include path.

'easy_install' bypasses the distutils 'install()' when doing a normal
egg install, and does not re-implement the 'install_headers' dance.

Furthermore, the 'bdist_egg' it creates, even though it contains the
header files, has no metadata which would allow the installer to
discover that they should be copied.