classification
Title: bdist_msi fails on files with long names
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.2, Python 3.1, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: cgohlke, cmcqueen1975, fberger, janssen, markm, termim
Priority: normal Keywords: needs review, patch

Created on 2010-01-05 19:03 by termim, last changed 2011-03-27 10:37 by markm. This issue is now closed.

Files
File name Uploaded Description Edit
dist_msi_bug.zip termim, 2010-01-05 19:03 test project that creates the above error
msilib_make_short.diff cgohlke, 2010-01-15 20:51 review
msilib.diff cgohlke, 2010-10-22 08:45 review
msilib2.diff cgohlke, 2011-03-07 18:18 Revised patch
msilib2.diff loewis, 2011-03-27 07:55 Regenerating patch for review review
msilib.make_id_fix_and_tests.patch markm, 2011-03-27 09:58 Patch to fix msilib.make_id() and test it review
Messages (8)
msg97271 - (view) Author: Mikhail Terekhov (termim) Date: 2010-01-05 19:03
When I try to build msi installer for my module that contains files with long names it crashes with the following traceback

C:\home\python\dist_msi_bug>python setup.py bdist_msi
running bdist_msi
running build
running build_scripts
installing to build\bdist.win32\msi
running install_scripts
running install_data
running install_egg_info
Removing build\bdist.win32\msi\Lib\site-packages\hello-0.1-py2.6.egg-info
Writing build\bdist.win32\msi\Lib\site-packages\hello-0.1-py2.6.egg-info
Traceback (most recent call last):
  File "setup.py", line 8, in <module>
    data_files = [(".", ["data.seq.one.dat", "data.seq.two.dat"])],
  File "c:\python26\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "c:\python26\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "c:\python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "c:\python26\lib\distutils\command\bdist_msi.py", line 242, in run
    self.add_files()
  File "c:\python26\lib\distutils\command\bdist_msi.py", line 270, in add_files
    key = dir.add_file(file)
  File "c:\python26\lib\msilib\__init__.py", line 340, in add_file
    short = self.make_short(file)
  File "c:\python26\lib\msilib\__init__.py", line 299, in make_short
    assert file not in self.short_names
AssertionError


It seems that Directory.make_short fails to create unique short names in some cases. The test files are attached.
msg97844 - (view) Author: Christoph Gohlke (cgohlke) Date: 2010-01-15 20:51
I can confirm this issue. It prevents building a IPython msi installer on Python 2.6 for Windows. 

A patch to the Directory.make_short function in msilib\__init__.py is attached. It falls back to generating prefix~pos filenames when a short name is already taken and also generates short names more similar to Windows/NTFS.
msg104034 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2010-04-23 17:57
This looks a lot like bug 1128, too.  I think the patch there would also fix this one.
msg119367 - (view) Author: Christoph Gohlke (cgohlke) Date: 2010-10-22 08:45
Creating bdist_msi installers with files such as 'aixc++' (containing '+'), '.buildinfo' (starting with '.'), and 'py.~1.5.~' (containing '~') currently also fails, even with the proposed patch. 

The revised patch should fix these cases and further improves the make_short function to generate short names compatible with NTFS.
msg124015 - (view) Author: Florian Berger (fberger) Date: 2010-12-15 12:14
I can confirm that this issue persists in Python 3.1.2 on win32. The patch by cgohlke from 2010-10-22 fixes the problem here as well.
msg132292 - (view) Author: Roundup Robot (python-dev) Date: 2011-03-27 08:16
New changeset e51366a05b84 by Martin v. Löwis in branch '2.7':
Fix short file name generation in bdist_msi.
http://hg.python.org/cpython/rev/e51366a05b84

New changeset fd8336947f90 by Martin v. Löwis in branch '3.1':
Fix short file name generation in bdist_msi.
http://hg.python.org/cpython/rev/fd8336947f90

New changeset c7d0fc181376 by Martin v. Löwis in branch '3.2':
merge #7639
http://hg.python.org/cpython/rev/c7d0fc181376

New changeset a7e0a1dbfbb6 by Martin v. Löwis in branch 'default':
merge #7639
http://hg.python.org/cpython/rev/a7e0a1dbfbb6
msg132301 - (view) Author: Mark Mc Mahon (markm) * Date: 2011-03-27 09:58
Following up from my comment in issue1128, please find the patch msilib.make_id_fix_and_tests.patch

This improves the make_id() function in the following way:
 - ensures that NO invalid identifier characters make it through (it does this by only including good characters rather than filtering out some bad characters). The list of bad characters is immense (most unicode characters!) while the list of acceptable characters is limited.

This also adds some tests.
msg132305 - (view) Author: Mark Mc Mahon (markm) * Date: 2011-03-27 10:37
For Directory.make_short() the only things which are left could be considered splitting hairs.

1. Do we need to remove '"\/:' from the file name - if these characters are there then the file name is not valid in the first place.

2. These characters '+,;=[]' should really be changed to '_' if we want to match more closely how Windows makes short names.

I can write a patch for the above if they should be fixed and either way I can add tests for this function. (the test I had for testing my own patch - almost completely pass - but the only failure was depending on point 2.)
History
Date User Action Args
2011-03-27 10:37:12markmsetmessages: + msg132305
2011-03-27 09:58:50markmsetfiles: + msilib.make_id_fix_and_tests.patch
nosy: + markm, - python-dev
messages: + msg132301
2011-03-27 08:16:33python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg132292

resolution: fixed
stage: test needed -> resolved
2011-03-27 07:55:19loewissetfiles: + msilib2.diff
2011-03-07 18:18:42cgohlkesetfiles: + msilib2.diff
nosy: janssen, termim, cmcqueen1975, cgohlke, fberger
versions: + Python 2.7, Python 3.2
2010-12-15 12:14:40fbergersetnosy: + fberger

messages: + msg124015
versions: + Python 3.1
2010-10-22 08:45:40cgohlkesetfiles: + msilib.diff

messages: + msg119367
2010-07-23 07:27:48cmcqueen1975setnosy: + cmcqueen1975
2010-04-23 17:57:23janssensetnosy: + janssen
messages: + msg104034
2010-01-15 21:08:30brian.curtinsetpriority: normal
keywords: + needs review
type: crash -> behavior
stage: test needed
2010-01-15 20:51:53cgohlkesetfiles: + msilib_make_short.diff

nosy: + cgohlke
messages: + msg97844

keywords: + patch
2010-01-05 19:03:26termimcreate