distutils.sdist.add_defaults adds the Python modules and scripts and C
extensions found in setup.py to the MANIFEST.  It does *not* add
data_files mentioned in setup.py to the MANIFEST.

This is non-orthogonal and confusing, because it means that a
MANIFEST.in is required if you have data_files, optional if you do not.
 If you have data_files and do not have MANIFEST.in, you get a broken
package but no error.

Agreed; package_data are also ignored by sdist. Unfortunately, neither
setuptools seems to work as expected on sdist but at least bdist_egg does.

MANIFEST.in is an ugly hack and should be deprecated; you shouldn't have
to repeat yourself in two files.

I'll change add_defaults() so it includes package_data.

I don't think though, that MANIFEST.in is an ugly hack. Imho the best
practice is an explicit MANIFEST.in (and not use the .svn browsing
setuptools do)

I didn't mean to imply that automagic discovery based on external
version control software is better than MANIFEST.in; I favor
explicitness here as well. It's just that this information can (and
often has to) be duplicated in setup.py as 'package_data' or
'data_files'. For cases that package_data/data_files are not enough,
setup.py should be extended to handle them, instead of requiring to
write and keep in sync a separate file with its own mini syntax.

Right, but if MANIFEST.in is removed, I can see cases where you would
need the same kind of mini-syntax in your setup.py.

For instance, how would you tell sdist to recursively add files located
in a directory (like the current recursive-include feature of MANIFEST.in) ?

By an equivalent option in setup() of course. I'm not against the
*functionality* of MANIFEST.in but on that (a) it's a second file you
have to remember to write and maintain in addition to setup.py (b) has
its own ad-hoc syntax instead of python and (c) overlaps in scope with
the package_data and data_files of setup.py.

FWIW I wrote a module that overrides the default build_py and sdist
commands with versions that allow specifying package_data recursively
(while preserving file permissions, unlike the - buggy IMO - behavior of
distutils) so that I can get rid of the MANIFEST.in.

Ok I have finished the patch, I'll commit it during the week

> FWIW I wrote a module that overrides the default build_py and sdist
> commands with versions that allow specifying package_data recursively

Maybe that could be a new feature ? 


> (while preserving file permissions, unlike the - buggy IMO - behavior of
> distutils) so that I can get rid of the MANIFEST.in.

Sounds like a bug to me, could you fill an issue on that ?

done in r69692 and r69696.

> > FWIW I wrote a module that overrides the default build_py and sdist
> > commands with versions that allow specifying package_data recursively
> 
> Maybe that could be a new feature ? 

That would be nice, especially if we want to reimplement MANIFEST.in as
setup() option at some point. My current implementation doesn't extend
the API, so there's no way to specify a subset of files under a
directory like recursive-include; every directory matched by a glob is
copied in whole (recursively):

import os
from distutils.command.build_py import build_py as _build_py

class build_py(_build_py):
    def find_data_files(self, package, src_dir):
        files = []
        for p in _build_py.find_data_files(self, package, src_dir):
            if os.path.isdir(p):
                files.extend(os.path.join(par,f)
                             for par,dirs,files in os.walk(p) 
                             for f in files)
            else:
                files.append(p)
        return files


> > (while preserving file permissions, unlike the - buggy IMO - behavior of
> > distutils) so that I can get rid of the MANIFEST.in.
> 
> Sounds like a bug to me, could you fill an issue on that ?

If it's a bug, it's certainly not accidental; there's a big XXX comment
justifying this choice but I'm not convinced. I posted about it at
http://mail.python.org/pipermail/python-list/2009-January/524263.html;
if you think it's a bug I'll fill an issue.

> done in r69692 and r69696.

Great, thanks. The data_files part though seems incorrect; for one thing
each item in data_files can be either a (dir,files) tuple or a plain
string, and for two 'dir' is the output (installation) directory, not
the root of 'files'. Here's the relevant part from my module:

if self.distribution.has_data_files():
    for item in self.distribution.data_files:
        if isinstance(item, basestring): # plain file
            self.filelist.append(convert_path(item))
        else: # an (outdir, files) tuple
            outdir,data_files = item
            self.filelist.extend(convert_path(f) for f in data_files)

added in r69710. Thx

2009/2/17 George Sakkis <report@bugs.python.org>:
>> Maybe that could be a new feature ?
>
> That would be nice, especially if we want to reimplement MANIFEST.in as
> setup() option at some point. My current implementation doesn't extend
> the API, so there's no way to specify a subset of files under a
> directory like recursive-include; every directory matched by a glob is
> copied in whole (recursively):

Please could you add a feature request ?

We will need to discuss it there.

>
> import os
> from distutils.command.build_py import build_py as _build_py
>
> class build_py(_build_py):
>    def find_data_files(self, package, src_dir):
>        files = []
>        for p in _build_py.find_data_files(self, package, src_dir):
>            if os.path.isdir(p):
>                files.extend(os.path.join(par,f)
>                             for par,dirs,files in os.walk(p)
>                             for f in files)
>            else:
>                files.append(p)
>        return files

> If it's a bug, it's certainly not accidental; there's a big XXX comment
> justifying this choice but I'm not convinced. I posted about it at
> http://mail.python.org/pipermail/python-list/2009-January/524263.html;
> if you think it's a bug I'll fill an issue.

Please do so, I am focusing on the Distutils-SIG ML , so I missed it

I don't know yet what is a proper way to adress this, but the bug tracker seem
apprioriate for this.

>
> ----------
> versions: +Python 2.6, Python 3.0
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue2279>
> _______________________________________
> _______________________________________________
> Python-bugs-list mailing list
> Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/ziade.tarek%40gmail.com
>
>

Opened #5300 and #5302 for the mentioned issues. 

Btw in your patch, I believe `os.path.join(dirname, f)` should be
replaced by `f` alone; `dirname` refers to the dir under the
installation directory, not the source.

> Opened #5300 and #5302 for the mentioned issues. 

ok thanks

> Btw in your patch, I believe `os.path.join(dirname, f)` should be
> replaced by `f` alone; `dirname` refers to the dir under the
> installation directory, not the source.

Right, thanks George. I'v also added you in the ACKS for your contribution
on this (r69724)