The BZ2File class only supports one stream per file. It possible to have
multiple streams concatenated in one file, it the resulting data should
be the concatenation of all the streams. It's what the bunzip2 program
produces, for example. It's also supported by the gzip module.

Once this done, this would add the ability to open a file for appending,
by adding another stream to the file.

I'll probably try to do this, but the fact it's done in C (unlike gzip)
makes it harder, so if someone beats me to it, etc.

If you're referring to an 'append' mode for bz2file objects, it may be a
limitation of the underlying library: my version of bzlib.h only
provides BZ2_bzWriteOpen and BZ2_bzReadOpen - it's not immediately clear
how you would open a BZ2File in append mode looking at this API.

It may be possible to implement r/w/a using the lower-level
bzCompress/bzDecompress functions, but I doubt that's going to happen
unless somebody (such as yourself? :)) cares deeply about this.

Like gzip, you can concatenate two bzip2 files:

bzip2 -c /etc/passwd >/tmp/pass.bz2

bzip2 -c /etc/passwd >>/tmp/pass.bz2

bunzip2 will output both parts, generating two copies of the file.

So nothing needs to be done on compression, but uncompression needs to
look for another chunk of compressed data after finishing one chunk.

The gzip module supports reopening an existing file to add another
stream. I think the bz2 module should not the same.

I've got a patch that fixes this.  It allows BZ2File to read
multi-stream files as generated by pbzip2, allows BZ2File to open files
in append mode, and also updates bz2.decompress to allow it to handle
multi-stream chunks of data.

We originally wrote it against 2.5, but I've updated the patch to py3k
trunk, and attached it here.  If there's interest in a patch against 2.7
trunk, please let me know.

sorry, the previous patch was from an old version.  attaching the
correct version now.  apologies for the noise.

Some notes about posting patches:
- you should post the patch alone, not in an archive
- generally you should post patches against the 2.7 trunk, we take care
of merging them to py3k ourselves (but in this case the difference
should be minimal anyway)
- I'm not sure it's ok to add legal boilerplate at the top of files, we
never do that usually (and if everyone did it would become unreadable).
Does your company require you to do so?

I'll look at the patch itself another day, I don't have the time right
now. But thanks for posting it!

Thanks for the reply.

My company's legal dept. told me that we needed to put the boilerplate
into the files as part of releasing it under the apache license.  I used
a tarball because they also recommended including a full copy of the
license with the patch.

I'm reattaching just the patch to the bug now.  I'll check with legal
and see if they'd have a problem with removing the boilerplate.

If the patch is substantial enough that legal boilerplate is even an
issue, then I'm pretty sure a contributor agreement will be required for
patch acceptance, at which point I think the boilerplate won't be
needed.  The Apache license is certainly acceptable.  I'm obviously not
the authority on this, though.  That would be van Lindburg.

I can remove the boilerplate from the code as long as I add the
following to the submittal:

VMware, Inc. is providing this bz2 module patch to you under the terms
of the Apache License 2.0 with the understanding that you plan to
re-license this under the terms and conditions of the Python License.
This patch is provided as is, with no warranties or support. VMware
disclaims all liability in connection with the use/inability to use this
patch. Any use of the attached is considered acceptance of the above.

As far as I can tell, the patch looks mostly good.
I just wonder, in Util_HandleBZStreamEnd(), why you don't set self->mode
to MODE_CLOSED if BZ2_bzReadOpen() fails.

As a sidenote, the bz2 module implementation seems to have changed quite
a bit between trunk and py3k, so if you want it to be backported to
trunk (2.7), you'll have to provide a separate patch.

Hrm...yeah, I should probably be setting it to closed as soon as
BZ2_bzReadClose() returns, and then back to open once BZ2_bzReadOpen
succeeds.  Wasn't intentional...thanks for the catch.  You guys need a
new patch with that change in it?

I'll try and get a 2.7 patch done and uploaded in a day or two.

A new patch will make it more likely that it will actually get applied :)

Thanks for your work on this.

Understandable.  New patch attached.

I'm not comfortable with the following change (which appears twice in
the patch):

-			BZ2_bzReadClose(&bzerror, self->fp);
+			if (self->fp)
+				BZ2_bzReadClose(&bzerror, self->fp);
 			break;
 		case MODE_WRITE:
-			BZ2_bzWriteClose(&bzerror, self->fp,
-					 0, NULL, NULL);
+			if (self->fp)
+				BZ2_bzWriteClose(&bzerror, self->fp,
+						 0, NULL, NULL);


If you need to test for the file pointer, perhaps there's a logic flaw
in your patch. Also, it might be dangerous in write mode: could it occur
that the file isn't closed but the problem isn't reported?

That was mostly just out of paranoia, since the comments mentioned 
multiple calls to close being legal.  Looking at it again, that particular 
case isn't an issue, since we don't hit that call when the mode is 
MODE_CLOSED.  The testsuite runs happily with those changes reverted.  
Should I upload a new patch?

> That was mostly just out of paranoia, since the comments mentioned 
> multiple calls to close being legal.  Looking at it again, that particular 
> case isn't an issue, since we don't hit that call when the mode is 
> MODE_CLOSED.  The testsuite runs happily with those changes reverted.  
> Should I upload a new patch?

You don't need to, but on the other hand I forgot to ask you to update
the documentation :-) (see Doc/library/bz2.rst)