I've now been able to build my installer.

I applied Travis Oliphant's patch from http://bugs.python.org/issue2399 to Lib/msilib/__init__.py, then added a __del__ method to the Directory class:

    def __del__(self):
        if self._numfiles_wo_commit > 0:
            self.db.Commit()
        self.db = None
        self.keyfiles = None
        self.cab = None
        self.ids = None

This seems to release enough memory that I can deal with a larger number of files.

I don't think there's a good way to write a unit test for this, though.  Success or failure will depend on the machine you're running it on, I think.