This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: msilib can't create large CAB files
Type: resource usage Stage: resolved
Components: Library (Lib), Windows Versions: Python 2.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Patches for Tools/msi
View: 2399
Assigned To: Nosy List: janssen, loewis, r.david.murray
Priority: normal Keywords:

Created on 2010-04-27 22:55 by janssen, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (10)
msg104379 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2010-04-27 22:55
I'm trying to create a CAB file containing about 69MB of data, in 4555 files.  msilib fails in CAB.commit():

$ python build-msi-installer.py /c/UpLib/1.7.9 ~/uplib 1.7.9 ./uplib-1.7.9.msi
c:\Documents and Settings\wjanssen\uplib\win32\uplib-1.7.9.msi
install_location c:/UpLib/1.7.9
c:\docume~1\wjanssen\locals~1\temp\tmpu5dwz6 68943045
gc: collecting generation 0...
gc: objects in each generation: 670 702 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 707 1364 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 699 2031 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 794 2680 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 730 3373 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 765 4018 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 741 4697 8255
gc: done, 0.0000s elapsed.
(1, 4555, None, '#prereqs', None, None)
Traceback (most recent call last):
  File "build-msi-installer.py", line 780, in <module>
    p.run()
  File "build-msi-installer.py", line 243, in run
    self.add_files()
  File "build-msi-installer.py", line 312, in add_files
    cab.commit(self.db)
  File "c:\Python26\lib\msilib\__init__.py", line 223, in commit
    [(1, self.index, None, "#"+self.name, None, None)])
  File "c:\Python26\lib\msilib\__init__.py", line 97, in add_data
    v = db.OpenView("SELECT * FROM `%s`" % table)
_msi.MSIError: 1: 2229 2: c:\Documents and Settings\wjanssen\uplib\win32\uplib-1.7.9.msi 3: Media 4: SELECT * FROM `Media` 
gc: collecting generation 2...
gc: objects in each generation: 416 5351 8254
gc: done, 0.0000s elapsed.
$

(I added a bit of code to print out the Media table record, and the size of the file created with FCICreate.)

This works fine if I choose a slightly smaller subset, such as /c/UpLib/1.7.9/lib, for instance.  So I'm guessing this is a memory problem; at some point there's just not enough memory to open the MSI file and load the Media table.

I opened the MSI file with orca and sure enough, there is no Media table in it.  Not sure how to take that, though;  perhaps the table only appears after a record has been written to it.

Another reason to allow multiple CAB files in a single installer?
msg104381 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-27 22:59
> Another reason to allow multiple CAB files in a single installer?

I'm still curious as to what the first reason is.
msg104390 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2010-04-28 01:12
Ummm, just in case the packager wanted to.

In my case, I was putting the files which were registered as part of the installation in one CAB file, and another set of temporary files which were used by some of the installation scripts, but not part of the installation file set, in a second CAB file.  I've since changed that second package to a zip file.
msg104391 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2010-04-28 01:14
So, I subclassed msilib.CAB, and re-wrote commit() so that the CAB file is created in a different process, a la Tools/msi/msilib.py.  Still have the same problem, though.  So now I'm thinking it's not a memory problem, but I'm wondering if there's something about the length of the path names in the CAB file, or some such.
msg104414 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-28 06:37
You could also try to commit the MSI file in-between, which may release memory.
msg104438 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2010-04-28 16:29
Yes, I've tried that.  No joy.  Right now I'm trying an approach which
packages each top-level directory as a separate cab.

What I'm finding is that if I get up around 4200 files, it breaks,
regardless of the file sizes.  Out of curiosity, how many files are in the
Python MSI file?

Bill

On Tue, Apr 27, 2010 at 11:37 PM, Martin v. Löwis <report@bugs.python.org>wrote:

>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
> You could also try to commit the MSI file in-between, which may release
> memory.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue8552>
> _______________________________________
>
msg104445 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2010-04-28 18:03
I've now been able to build my installer.

I applied Travis Oliphant's patch from http://bugs.python.org/issue2399 to Lib/msilib/__init__.py, then added a __del__ method to the Directory class:

    def __del__(self):
        if self._numfiles_wo_commit > 0:
            self.db.Commit()
        self.db = None
        self.keyfiles = None
        self.cab = None
        self.ids = None

This seems to release enough memory that I can deal with a larger number of files.

I don't think there's a good way to write a unit test for this, though.  Success or failure will depend on the machine you're running it on, I think.
msg104516 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2010-04-29 14:47
Another bit of info.  It's the frequent commits that seem to fix the problem; when I comment those out of the __del__ method, it fails as before.  I also notice that the finished installer is about twice the size of the two data files in it (the CAB file and my zip file).  Seems like a lot of overhead.
msg104540 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2010-04-29 17:04
Ah, found the size problem -- I was measuring something in 512 blocks not 1KB blocks.  Never mind.
msg185132 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-24 12:20
It looks like it turned out that there is nothing specific in this issue that isn't covered by issue 2399.
History
Date User Action Args
2022-04-11 14:57:00adminsetgithub: 52798
2013-03-24 12:20:38r.david.murraysetstatus: open -> closed

superseder: Patches for Tools/msi

nosy: + r.david.murray
messages: + msg185132
resolution: duplicate
stage: resolved
2010-04-29 17:04:07janssensetmessages: + msg104540
2010-04-29 14:47:59janssensetmessages: + msg104516
2010-04-28 18:03:20janssensetmessages: + msg104445
2010-04-28 17:57:01janssensetfiles: - unnamed
2010-04-28 16:29:15janssensetfiles: + unnamed

messages: + msg104438
2010-04-28 06:37:52loewissetmessages: + msg104414
2010-04-28 01:14:49janssensetmessages: + msg104391
2010-04-28 01:12:54janssensetmessages: + msg104390
2010-04-27 22:59:05loewissetmessages: + msg104381
2010-04-27 22:55:50janssencreate