With a simple setup.py:

from distutils.core import setup
setup(name='packagename')

And a MANIFEST.in containing:

include buildout.cfg 

The result of a debug run of python setup.py sdist on Windows is:

Distribution.parse_config_files():
options (after parsing config files):
no commands known yet
options (after parsing command line):
option dict for 'sdist' command:
  {}
running sdist
Distribution.get_command_obj(): creating 'sdist' command object
warning: sdist: missing required meta-data: version, url
warning: sdist: missing meta-data: either (author and author_email) or 
(maintainer and maintainer_email) mus
t be supplied
checking if setup.py newer than MANIFEST
warning: sdist: standard file not found: should have one of README, 
README.txt
reading manifest template 'MANIFEST.in'
include buildout.cfg
include_pattern: applying regex r'^buildout\.cfg$'
 adding buildout.cfg
Distribution.get_command_obj(): creating 'build' command object
exclude_pattern: applying regex r'^build\.*'
 removing buildout.cfg

...

The regex built is incorrect, although it is apparently correctly built 
on other platforms...

This seems to be a major flaw, noone caring about it?

I guess not... Maybe try catching Tarek on irc or drop him a mail?

Sorry for the lag. There are a lot of bugs and Tarek is very busy. The steps you describe need to be turned in a new test in py3k’s test_sdist.

I’ll write a unit test shortly.

Attaching a unit test translation of Chris’ report. The test pass on POSIX (linux2), can someone test on win32?

Tracing the flow from sdist to filelist, it seems to be not a regex error but a logic error. A file explicitly added should not be removed by the exclude filter afterward.

Test by haypo on win32:
- ['README', 'setup.py', 'somecode\\__init__.py']
?                                 ^^

+ ['README', 'buildout.cfg', 'setup.py', 'somecode/__init__.py']
?           ++++++++++++++++                      ^

The bug is confirmed. I’ll use os.sep to silence the second error, and look into test_filelist and filelist itself to fix it.

This is a regex bug, and it just bit me again :-(

Because of this bug, you cannot currently build a bdist_egg (and therefore cannot install with easy_install) http://pypi.python.org/pypi/buildout-versions on windows.

The only choice I have is to rename the 'buildout_versions' package inside the distribution to something that doesn't start with 'build'.

So, the workaround I gave doesn't work, because this stupid regex still knocks out all the egg-info. The only solution is to use that accidentally employed by http://svn.plone.org/svn/collective/buildout/buildout.dumppickedversions/trunk/ and put any files starting with 'build' and that need to be used on Windows in a sub-folder.

Here's a debug sdist build dump from buildout.dumppickedversions:

Running setup script 'setup.py'.
Distribution.parse_config_files():
options (after parsing config files):
no commands known yet
options (after parsing command line):
option dict for 'aliases' command:
  {}
option dict for 'sdist' command:
  {}
running sdist
Distribution.get_command_obj(): creating 'sdist' command object
running egg_info
Distribution.get_command_obj(): creating 'egg_info' command object
writing requirements to src\buildout.dumppickedversions.egg-info\requires.txt
writing src\buildout.dumppickedversions.egg-info\PKG-INFO
writing namespace_packages to src\buildout.dumppickedversions.egg-info\namespace_packages.txt
writing top-level names to src\buildout.dumppickedversions.egg-info\top_level.txt
writing dependency_links to src\buildout.dumppickedversions.egg-info\dependency_links.txt
writing entry points to src\buildout.dumppickedversions.egg-info\entry_points.txt
Distribution.get_command_obj(): creating 'build_py' command object
Distribution.get_command_obj(): creating 'build' command object
include_pattern: applying regex r'^src\\buildout\.dumppickedversions\.egg\-info\.*[^/]*\Z(?ms)'
 adding src\buildout.dumppickedversions.egg-info\dependency_links.txt
 adding src\buildout.dumppickedversions.egg-info\entry_points.txt
 adding src\buildout.dumppickedversions.egg-info\namespace_packages.txt
 adding src\buildout.dumppickedversions.egg-info\not-zip-safe
 adding src\buildout.dumppickedversions.egg-info\PKG-INFO
 adding src\buildout.dumppickedversions.egg-info\requires.txt
 adding src\buildout.dumppickedversions.egg-info\SOURCES.txt
 adding src\buildout.dumppickedversions.egg-info\top_level.txt
exclude_pattern: applying regex r'^build\.*'
 removing buildout.cfg
exclude_pattern: applying regex r'^buildout\.dumppickedversions\-0\.5\.*'
...

Now contrast with buildout_versions:

Running setup script 'setup.py'.
Distribution.parse_config_files():
options (after parsing config files):
no commands known yet
options (after parsing command line):
option dict for 'aliases' command:
  {}
option dict for 'sdist' command:
  {}
running sdist
Distribution.get_command_obj(): creating 'sdist' command object
running egg_info
Distribution.get_command_obj(): creating 'egg_info' command object
writing requirements to buildout_versions.egg-info\requires.txt
writing buildout_versions.egg-info\PKG-INFO
writing top-level names to buildout_versions.egg-info\top_level.txt
writing dependency_links to buildout_versions.egg-info\dependency_links.txt
writing entry points to buildout_versions.egg-info\entry_points.txt
Distribution.get_command_obj(): creating 'build_py' command object
Distribution.get_command_obj(): creating 'build' command object
include_pattern: applying regex r'^buildout\_versions\.egg\-info\.*[^/]*\Z(?ms)'
 adding buildout_versions.egg-info\dependency_links.txt
 adding buildout_versions.egg-info\entry_points.txt
 adding buildout_versions.egg-info\not-zip-safe
 adding buildout_versions.egg-info\PKG-INFO
 adding buildout_versions.egg-info\requires.txt
 adding buildout_versions.egg-info\SOURCES.txt
 adding buildout_versions.egg-info\top_level.txt
exclude_pattern: applying regex r'^build\.*'
 removing buildout_versions.egg-info\top_level.txt
 removing buildout_versions.egg-info\SOURCES.txt
 removing buildout_versions.egg-info\requires.txt
 removing buildout_versions.egg-info\PKG-INFO
 removing buildout_versions.egg-info\not-zip-safe
 removing buildout_versions.egg-info\entry_points.txt
 removing buildout_versions.egg-info\dependency_links.txt
 removing buildout.cfg
 removing buildout_versions.egg-info\SOURCES.txt
exclude_pattern: applying regex r'^buildout\-versions\-1\.4\.*'

Okay, so here's the patch:

Change line 352 on distutils/filelist.py from:

pattern_re = "^" + os.path.join(prefix_re, ".*" + pattern_re)

To:

pattern_re = "^"+prefix_re+re.escape(os.sep)+".*"+pattern_re

os.path.join is is not always the saviour ;-)

My feeling was that there was a logic bug in filelist (something explicitly added should not be filtered out by a default exclude pattern), so I’m glad to learn it’s just a regex bug.  I’ll add a test for the add/filter logic just in case.  Thanks a lot for the diagnosis and patch, I will probably commit it on Monday.

I’m sorry I couldn’t fix this one in 3.1.  I didn’t take the time to download and install a Windows to test this year, and right now I don’t have the bandwidth.  I’ll get to it as soon as possible.