I just ran "setup.py bdist_msi" with NLTK which is a pure Python
package. You can see the setup.py here:
http://code.google.com/p/nltk/source/browse/trunk/nltk/setup.py. Despite
the fact that NLTK is pure Python, the generated .msi's look like
nltk-0.9.8.win32-py2.5.msi and nltk-0.9.8.win32-py2.6.msi (built with
Python 2.5 and 2.6 respectively).

So, two questions: (1) are the version numbers supposed to be there? and
(2) if so, does that mean a .msi for a pure Python package built by
Python 2.6 won't work on any other version?

> So, two questions: (1) are the version numbers supposed to be there?

Yes.

> (2) if so, does that mean a .msi for a pure Python package built by
> Python 2.6 won't work on any other version?

Yes.

Mostly out of curiosity, why is that? With bdist_wininst, a pure Python
package would generate a version-less installer that could then be used
with any Python version.

> Mostly out of curiosity, why is that?

Primarily because it's not implemented. To implement it, you
would need to collect all Python installations on the system
from the registry, then create a UI to let the user select
a specific installation, then use that. Collecting all Python
versions is fairly difficult to do with standard MSI actions.

In addition, a common use case is that MSI installation works
unattended (no UI), in which case you would also have to make
a choice of default version to install to (e.g. with highest
version number).

Contributions are welcome.

I'm certainly no Windows API expert, but if no one takes a stab at it
sooner, maybe I can spend some time looking at this during PyCon.

I'm switching the ticket type to a feature request.

Ok, so here's what needs to happen to make this work. Note that all of
the following needs to happen at *runtime*, not at the time at which the
.msi is created:

(1) Find all sub-keys of SOFTWARE\Python\PythonCore in the registry.
These are the versions, e.g. 2.5, 2.6, 3.0, etc.

(2) For each version, get the Python installation dir from
SOFTWARE\Python\PythonCore\%(version)s\InstallPath

(3) Populate the ListView table with entries containing these
installation paths, e.g. something like:

    TARGETDIR 1 C:\Python24
    TARGETDIR 2 C:\Python25
    ...

(4) Modify Control/SelectDirectoryDlg so that it uses a ListView (filled
with the above values) instead of a DirectoryCombo.

(5) Make a couple minor edits to bdist_msi.py to stop it from inserting
the version into the .msi name for Python-only modules.

I looked into a couple ways of doing this. Ideally, we should avoid
using a CustomAction, which would require maintaining some additional C
or VBScript code, and instead do everything through the database tables
that are built into all .msi files. Some problems I've run into with
this approach:

* The only way to read the registry AFAICT is through the RegLocator
table: http://msdn.microsoft.com/en-us/library/aa371171(VS.85).aspx. But
RegLocator can only look up single key values, and cannot look up the
sub-keys under a key (which is what we need to get the Python versions).

* We could hard code all the possible versions, and stick all the
corresponding SOFTWARE\...\%(version)s\InstallPath keys into RegLocator.
Then these can be read into properties using the AppSearch table
(http://msdn.microsoft.com/en-us/library/aa371559(VS.85).aspx), and we
could then fill in the ListView table with the install paths. But
AFAICT, there is no way to keep from putting one row into the ListView
table for each version of Python we statically define. Which means a
bunch of inappropriate rows at runtime (e.g. there'd be a row for Python
2.3 even if there was no Python 2.3 on your system).

Basically, the problem is that we'd like to determine what goes into the
ListView table at runtime, but I can only figure out how to put things
into it at the time at which the .msi is built.

I'm going to continue to look into this, and I'd welcome suggestions
anyone has.

The last resort would be to create a CustomAction using a DLL or
VBScript, but I'm really trying to avoid that, both because maintaining
and debugging such code is painful, and because Michael Foord suggested
that some virus checkers will complain about .msi files with embedded
VBScript (probably forcing me to maintain a DLL - ugh!).

In

http://www.installsite.org/pages/en/msi/articles/MultiListBox/index.htm

there is a demo how to modify the listbox contents dynamically using 
VBScript.

Thanks for the link. I did play around with that code for quite a while,
and while I'm convinced there's a way to get it to work, I'm still
struggling with it. I believe the attached .vbs file does about what we
need to do, but I think I'm attaching it at the wrong location in the
.msi file because things seem to die at the "Session.Database.OpenView"
line with an error saying the object doesn't have that method (or no
error message but still at that line, depending on how I've attached the
action.)

Anyway, I'll probably keep playing around with this, but some food for
thought on why we may not want to implement this as a .vbs:
  http://blogs.msdn.com/robmen/archive/2004/05/20/136530.aspx
In particular, some anti-virus products will silently keep them from
working. That said, I probably need to figure out how to write the .vbs
code in order to (eventually) write the C or whatever else code.

My OpenView bug was just a missing "Set". The CustomAction does seem to
be correctly gathering the Python paths from the registry and filling
the ListView now. I've still got a couple of errors showing up later in
the process, but I expect I'll probably have a patch within a week or two.

Ok, I've made some progress on this. The attached patch now generates
MSIs which are version agnostic and look up the appropriate Python
version in the registry. Some things still remaining to do:

* The ProductName needs to be modified at runtime to prefix the "Python
X.Y" for Add/Remove Programs (ARP). I have a custom action making the
appropriate modifications, but this doesn't seem to affect the name in
ARP. Not entirely sure what the problem is, but it seems I may only be
setting the property on the client, and not on the server. I'll continue
looking into this.

* I need to thoroughly test that it does the right thing when a
particular version of Python is required. The code should basically be
there now to do that, but I haven't tested it at all. Probably I need to
play around both with target_version and actually building an extension
module.

Speaking of tests, I have no idea how I would write a unittest for any
of these things. To check some of them, it seems like you'd actually
have to install the .msi onto a Windows machine. So at the moment, all
my tests are by hand. Any better suggestions for testing these kinds of
things would be greatly appreciated.

I'm still stuck on getting the right name to show up in ARP.

Another problem: it seems like we have to update the ProductCode at
runtime as well - otherwise, you can only have one module installed for
all the versions of Python you have on your machine. But if we generate
a new GUID at runtime, how do we make sure that the GUID for moduleXXX +
Python 2.6 is the same every time (as it should be for the MSI to
recognize when a product is being removed, reinstalled, etc.)

Thoughts? Is there a way to deterministically generate a GUID for each
Python version? (Note that it doesn't make sense to do this ahead of
time - an installer created for Python 3.1 should work with Python 3.2
as well.)

Updated the patch to make sure ProductName is set before ValidateProductID.

> I'm still stuck on getting the right name to show up in ARP.

It may that indeed Installer blocks the property from being passed onto
the server side. Three things to try:
- inspect the log file, to see whether it is passed, and then whether it
  gets set. You do logging with "msiexec /i foo.msi /l*v foo.log".
- add the property to SecureCustomProperties, to have it passed to
  server mode; by default, only properties in UPPER_CASE get passed.
- alternatively, add an upper-case property, and make ProductName
  computed.

> Another problem: it seems like we have to update the ProductCode at
> runtime as well

I knew that would cause problems some day :-) I expected you to desire
this only for a single installation. Installing multiple copies is much
more difficult.

IIUC, a common approach is to use transforms, although I'm not sure how
precisely that would work.

A hacky approach might be to use computed uuids (if that can work at
all: I'm skeptical that you can change the productcode at runtime):
have a fixed ProductCode in the MSI, and then add the minor Python
version to the last digit of uuid. See Tools/msi/msi.py for how the
uuid of the Win64 installer changes from the one for the 32-bit
installer.

I think this is really is a question to ask on some MSI channels;
most likely, the answer is that this cannot possibly work.

Ok, I've been chatting with folks on microsoft.public.platformsdk.msi,
and I think the right approach here is to define a Feature for each
version of Python. Each Feature would install the exact same files, but
to a different Python directory.

One of the nice things about this approach would be that you could
install a module for multiple Python versions all at the same time. It
also should avoid the need for a VBScript or C-based CustomAction - we
should be able to do everything with the database tables.

One of the downsides is that we'll have to hard-code in all possible
versions of Python. My current plan is just to declare all versions as
2.0, 2.1, ..., 3.9. That should be good enough for a while, but at some
point this will have to get updated to work with Python 4.0. ;-)

I'll post back here when I've made some progress on this.

Ok, that was actually easier than I thought it would be. The new patch
introduces properties for each Python version (e.g. TARGETDIR2.4,
PYTHON.MACHINE.2.4, etc.), and disables and hides the features for any
Python versions that aren't found in the registry.

The one remaining issue: What should we do about Python installations
that are missing the appropriate keys in the registry? I imagine this
could happen if, say, you build Python from source. My first thought was
to add an "Other Python Installation" Feature that is disabled (but
visible) by default, and allow the path for that Feature to be filled in
by hand. Does that make sense?

Ok, I added one final Feature that allows the user to specify an
alternate Python directory. (The PathEdit for specifying the directory
will only display if this Feature is set to be installed.)

I think this patch is pretty much ready to go in now. It could use a
review and some testing from folks on other kinds of machines, but I
think all the functionality is there now. (And no need for CustomAction
scripts! Yay!)

If at all possible, I'd like to get this into the Python 3.1 beta so
that we can have as many people as possible test this out on other
Windows machines.

A slightly improved patch, using DuplicateFile instead of storing a copy
of each file for each Python version. Should keep the size of the
resulting MSI similar to the size of the currently generated MSIs.

Ok, one last tiny update that makes sure TARGETDIR is always set to one
of the TARGETDIRX.Ys from a Feature that is actually selected.

I swear I'm done with this now. ;-)

Can you kindly attach a demo MSI, to simplify review?

Here's an MSI generated for the argparse module.

The patch looks fine so far, please apply to trunk and 3k. As this is a
new feature, I think backporting it is not appropriate.

I believe that the support for (pre)install scripts is incorrect; I
would expect that each such script should be executed once per version
for which the package is installed. However, this can be fixed later;
for the moment, it would be good enough to warn at packaging time that
scripts may not work for multi-version packages.

You can also consider providing "Change" installation, allowing the user
to selectively add a package to a python version after that python
version is installed (or to remove it before the python version gets
removed). To do so, just add an option to the MaintenanceTypeDlg that
forks into the feature selection dialog, with the standard INSTALL action.

One additional suggestion: allow the packager to specify what the
minimum Python version is. Otherwise, you might have a package that
enables installation for Python 2.3 and 2.4 when the maintainer has
already stated that Python 2.5 is the minimum version supported.
Otherwise, looks great.

@martin: 
Thanks! Do I need to do something special to make the merging work
right? Or do I just apply the patch separately to the trunk and the py3k
branche?

Good point about the install script - I think the condition needs to be
("&Python%s=3" % ver) instead of "NOT Installed". I'll use this approach
when I apply the patch, but clearly this needs some testing by folks
that use install scripts either way. I didn't worry much about the
pre-install script, since finalize_options() already throws an exception
saying that isn't supported

I'll open a new issue for support for a "Change" installation. This is
one of the nice things that treating Python versions as features allows,
I just haven't had time to look into it yet.

@anthony:
Allowing a minimum version to be set sounds like a nice feature. Could
you open a new feature request?

I've created another feature request as requested for supplying a
minimum Python version when creating pure Python packages.

http://bugs.python.org/issue5926

> Thanks! Do I need to do something special to make the merging work
> right? Or do I just apply the patch separately to the trunk and the py3k
> branche?

You commit to the trunk, then you do "svnmerge merge -r<rev>" in the 3k
branch, then "svn commit -F svnmerge-something.txt" (in case of
conflicts, you fix them first, of course).

> Good point about the install script - I think the condition needs to be
> ("&Python%s=3" % ver) instead of "NOT Installed".

I'm not sure - I think the install script must run several times
actually, so there must be several custom actions, each with its own
condition.

> I'll open a new issue for support for a "Change" installation. This is
> one of the nice things that treating Python versions as features allows,
> I just haven't had time to look into it yet.

Well, if there had been separate packages per version, you could install
them independently, anyway.

> You commit to the trunk, then you do "svnmerge merge -r<rev>" in the 
> 3k branch, then "svn commit -F svnmerge-something.txt" (in case of
> conflicts, you fix them first, of course).

Done in r72306 for trunk and r72309 for 3k. Thanks.

> I'm not sure - I think the install script must run several times
> actually, so there must be several custom actions, each with its own
> condition.

Yep. That's what the patch does - adds a CustomAction and an entry in
InstallExecuteSequence for each version.

> Well, if there had been separate packages per version, you could 
> install them independently, anyway.

For what it's worth, you still have this option if anyone wants to
distribute multiple MSIs. Simply pass --target-version to bdist_msi, and
you'll get an MSI for only the selected version of Python that you can
install independently of other MSIs generated with --target-version.