Python currently supports zlib, gzip and bzip2 compressors. What is missing is support
for xz (http://tukaani.org/xz/). It comes with a C library.

Is zc really a C library? I could find a standalone program, but no
shared object.
Actually, it seems that zc is a file format based on the lzma algorithm.
The plan could be to first implement the lzma module (bpo-5689), then a
xzfile module in pure python.

What is xz compression and why is it important?

Skip

Yes, xz-utils contains a C library, though it still caries the name
"liblzma.so", probably for historic reasons.
You are right that xz is a file format based around the lzma algorithm.
It just uses a more advanced container format. (lzma-utils had no
container at all.)

xz is the successor of lzma, which provides a better compression than
bzip2, while decompression speed is comparable with gzip. It is used by
the GNU project for source tarball compression (replacing bzip2) and
supported by GNU tar. See http://en.wikipedia.org/wiki/Xz,
http://tukaani.org/xz/ and http://tukaani.org/lzma/ for reference.

Are xz and lzma formats compatible with each other? If not, which one is
the most popular?

As I understand it from the http://tukaani.org/xz/ page, .lzma and .xz
are two file formats bases on the lzma compression method.

• .lzma simply stores the compressed stream.
• .xz is more complex

.lzma is actually not a format. It is just the raw output of the LZMA1
coder. XZ instead is a container format for the LZMA2 coder, which
probably means LZMA+some metadata.
XZ is the official successor to .lzma, and GNU is using it already
(look at coreutils), and GNU tar officially supports it since 1.22.

Once Python gets native support for lzma/xz like it does for zlib, bzip2 it could switch to using it for bundles and remote transfers. See:

http://mercurial.selenic.com/bts/issue1463

With lzma/xz being able to compress so much better, it'd be really appreciated by users on especially slow links(!!).

Ugh, can't edit previous message. Meant to say:

"Once Python gets native support for lzma/xz like it does for zlib and bzip2, Mercurial could switch to using it for bundles and remote transfers."

For platforms with native support in-kernel (e.g. Linux) that could be used instead of the bundled version.

(Since Python is officially switching to Mercurial, arguably this issue even more important.)

About why xz is important.

gnu.org, tug.org started publishing sources in xz format, quick grep:

autoconf/autoconf.spec:Source0: http://ftp.gnu.org/gnu/autoconf/%{name}-%{version}.tar.xz
coreutils/coreutils.spec:Source0: http://ftp.gnu.org/gnu/coreutils/%{name}-%{version}.tar.xz
libpng12/libpng12.spec:Source0: http://downloads.sourceforge.net/libpng/libpng-%{version}.tar.xz
libpng/libpng.spec:Source0: http://downloads.sourceforge.net/libpng/%{name}-%{version}.tar.xz
parted/parted.spec:Source0: http://ftp.gnu.org/gnu/parted/%{name}-%{version}.tar.xz
texlive-texmf/texlive-texmf.spec:Source0: ftp://tug.org/texlive/historic/%{year}/texlive-%{version}-texmf.tar.xz

xz is also supported by automake as dist target.

We all agree that lzma/xz is important, what is needed is a patch.

There's a Python binding for some lzma lib here: https://launchpad.net/pyliblzma

Once xz is implemented, xz compatibility should be added to the tarfile library.

Ooops, I kinda should've commented on this issue here in stead, rather than in bpo-5689, so I'll just copy-paste it here as well:

I'm the author of the pyliblzma module, and if desired, I'd be happy to help out adapting pyliblzma for inclusion with python.
Most of it's code is based on bz2module.c, so it shouldn't be very far away from being good 'nuff.
What I see as required is:

• clean out use of C99 types etc.
• clean up the LZMAOptions class (this is the biggest difference from the bz2 module, as the filter supports a wide range of various options, everything related such as parsing, api documentation etc. was placed in it's own class, I've yet to receive any feedback on this decission or find any remote equivalents out there to draw inspiration from;)
• While most of the liblzma API has been implemented, support for multiple/alternate filters still remains to be implemented. When done it will also cause some breakage with the current pyliblzma API.

I plan on doing these things sooner or later anyways, it's pretty much just a matter of motivation and priorities standing in the way, actual interest from others would certainly have a positive effect on this. ;)

For other alternatives to the LGPL liblzma, you really don't have any, keep in mind that LZMA is "merely" the algorithm, while xz (and LZMA_alone, used for '.lzma', now obsolete, but still supported) are the actual format you want support for. The LZMA SDK does not provide any compatibility for this.

I will happily review any implementation, and I can help with inclusion into python trunk.

...the LGPL liblzma...

Can you check which licences cover the different parts of the module? I think that you will have to contribute your code under the Python Contributor Agreement; and I just grabbed some copy of the "xz-utils" source package, and it states that "liblzma is in the public domain".

ah, you're right, I forgot that the license for the library had changed as well (motivated by attempt of pleasing BSD people IIRC;), in the past the library was LGPL while only the 'xz' util was public domain..

For my code, feel free to use your own/any other license you'd like or even public domain (if the license of bz2module.c that much of it's derived from permits of course)!

I guess everyone should be happy now then. :)

Btw. for review, I think the code already available should be pretty much good 'nuff for an initial review. Some feedback on things not derived from bz2module.c would be nice, especially the LZMAOptions class would be nice as it's where most of the remaining work required for adding additional filters support. Would kinda blow if I did the work using an approach that would be dismissed as utterly rubbish. ;)

Oh well, it's out there available for anyone already, I probably won't(/shouldn't;) have time for it in a month at least, do as you please meanwhile. :)

Hello,

I'm the author of the pyliblzma module, and if desired, I'd be happy
to help out adapting pyliblzma for inclusion with python.
Most of it's code is based on bz2module.c, so it shouldn't be very far
away from being good 'nuff.

Well, I wouldn't say bz2module is the best module out there, but as you
say it's probably good enough :) And we can help you fix things if
needed.

Is pyliblzma compatible with Python 3.x? It's too late to incorporate
any new feature in Python 2.x now.

• While most of the liblzma API has been implemented, support for
  multiple/alternate filters still remains to be implemented. When done
  it will also cause some breakage with the current pyliblzma API.

Hmm, then perhaps you should first fix the current API so that adding
new features doesn't force you to break the API again. There are strict
rules for API breakage in the standard library.

By the way, adding a new module to the stdlib probably requires writing
a PEP (Python Enhancement Proposal). I wouldn't expect this very
proposal to be controversial, but someone has to do it.

Finally, when a module is in the stdlib, it is expected that maintenance
primarily happens in the Python SVN (or Mercurial) tree. We have a
couple of externally-maintained modules, but they're a source of
problems for us.

Yeah, I guess I anyways can just break the current API right away to make it compatible with future changes, I've already figured since long ago how it should look like. It's not like I have to implement the actual functionality to ensure compatibility, no-op works like charm. ;)

[Replying to msg106566]

if you're already looking at bpo-6715, then I don't get why you're
asking.. ;)

Can you please submit a contributor form?

Martin: For LGPL (or even GPL for that matter, disregarding linking
restrictions) libraries you don't have to distribute the sources of
those libraries at all (they're already made available by others, so
that would be quite overly redundant, uh?;). LGPL actually doesn't
even care at all about the license of your software as long as you
only dynamically link against it.

Of course you do. Quoting from the LGPL

"You may convey a Combined Work ... if you also do each of the following:
...
d) Do one of the following:
0) Convey the Minimal Corresponding Source under the terms of this
License, and the Corresponding Application Code in a form
suitable for, and under terms that permit, the user to recombine
or relink the Application with a modified version of the Linked
Version to produce a modified Combined Work, in the manner
specified by section 6 of the GNU GPL for conveying
Corresponding Source.
1) [not applicable to Windows]
"

I don't really get what the issue would be even if liblzma were still
LGPL, it doesn't prohibit you from distributing a dynamically linked
library along with python either if necessary (which of course would
be of convenience on win32..)..

Of course I can distribute a copy of an lzma DLL. However, I would have to provide ("convey") a copy of the source code of that DLL as well.

Of course I can distribute a copy of an lzma DLL. However, I would
have to provide ("convey") a copy of the source code of that DLL as
well.

Can you tell me where you are currently providing the source code for
the readline library, or the gdbm library?

Oh, and by the way, you should probably shut down PyPI, since there are
certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed
one), and you aren't offering a link to download those libraries' source
code either.

You seem to have no problem "conveying" copies for the source code of
non-LGPL libraries such as OpenSSL. Why is that?

Can you tell me where you are currently providing the source code for
the readline library, or the gdbm library?

We don't, as they aren't included in the Windows distribution. The
readline library doesn't work on Windows, anyway, and instead of gdbm,
we had traditionally been distributing bsddb instead on Windows.

Oh, and by the way, you should probably shut down PyPI, since there are
certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed
one), and you aren't offering a link to download those libraries' source
code either.

This is off-topic for this bug tracker. Please remain objective and
professional if you can manage to.

You seem to have no problem "conveying" copies for the source code of
non-LGPL libraries such as OpenSSL. Why is that?

Not sure what you are referring to. We don't provide the sources for the
OpenSSL libraries along with the Windows installer, because the license
of OpenSSL doesn't require us to. This is very convenient for our users.

> Oh, and by the way, you should probably shut down PyPI, since there are
> certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed
> one), and you aren't offering a link to download those libraries' source
> code either.

This is off-topic for this bug tracker. Please remain objective and
professional if you can manage to.

This whole subthread was already off-topic (since it was pointed out,
before your previous message, that the underlying lib is in the public
domain).

Actually, I would argue that the whole idea of promoting a rigorous
interpretation of a license has no place on the bug tracker. It makes no
sense to do this on an ad hoc fashion, especially if you want lawyers to
be involved (they are certainly not reading this).

(of course, you will also have understood that I disagree with such a
rigorous interpretation)

Not sure what you are referring to. We don't provide the sources for the
OpenSSL libraries along with the Windows installer, because the license
of OpenSSL doesn't require us to. This is very convenient for our users.

This was not about providing the sources together with the installer
(which even the GPL or the LGPL don't require to do), but providing them
as a separate bundle on the download site.
We do have a copy of the OpenSSL source tree somewhere, it is used by
the Windows build process.

I've ported pyliblzma to py3k now and also implemented the missing functionality I mentioned earlier, for anyone interested in my progress the branch is found at:
https://code.launchpad.net/~proyvind/pyliblzma/py3k

I need to fix some memory leakages (side effect of the new PyUnicode/Pybytes change I'm not 100% with yet;) and some various memory errors reported by valgrind etc. though, but things are starting to look quite nice already. :)

I've (finally) finalized the api and prepared pyliblzma to be ready for inclusion now.

The code can be found in the 'py3k' branch referred to earlier.

Someone else (don't remember who:p) volunteered for writing the PEP earlier, so I leave it up to that person to write the PEP, I won't be able to get around to do so myself in the near future..

As promised, here is a quick review of the module.
https://code.launchpad.net/~proyvind/pyliblzma/py3k looks ready for a new entry in the PyPI, but for inclusion in core python it needs some cleanup:

• I suppose that src/pyliblzma.c is the only interesting file. In CPython it will be named Modules/lzmamodule.c
• There are some calls to malloc(), calloc() and free() that are incorrect (there is even a call to free() on a PyObject*!); in any case, functions like PyMem_Malloc() are preferred.
• please don't use alloca (and it has another name on Windows)
• snprintf is not available everywhere; use PyOS_snprintf instead.
• The module does not compile on Windows: types like uint8_t don't exist, __attribute__((unused)) is GCC-specific.
• Don't use PyBytesObject*; PyObject* is enough and avoids many casts.
• PyLong_FromLong(LZMA_FILTER_LZMA1) truncates the value on 32bit platforms
• You could use LZMA_VERSION_STRING to fill the version attribute.

Also, I did not find any documentation.
Otherwise the code follows CPython conventions and is easy to read. Good job!

3.2beta1 is scheduled for November 13, 2010, and no new feature will be accepted after. Do you think you can update it before the limit? otherwise the package could live in PyPI before it is shipped with Python.

All fixed now. :)

All fixed now. :)

Does that refer to msg119743? Please provide this module
as a patch for branches/py3k, otherwise it's difficult to see
what exactly you are contributing.

Also, please provide Doc/library/lzma.rst.

Not meaning to sound petty, but wouldn't it be common etiquette to
retain some original copyright notice from original code intact..?

It seemed to me that Nadeem had rewritten everything from scratch. Is
there any code of yours in the current module?

I've opened bpo-13507 to track adding liblzma to the OS X installer builds.

Ah, I thought that he had reused most of the original C code in _lzmamodule.c not replaced by python code, but I see that not being the case now (only slight fragments;).

Oh well, I thought that I'd still earned a note with some slight credit at least, but I guess I won't go postal or anything in lack of either.. :p

Oh well, I thought that I'd still earned a note with some slight credit
at least

I completely agree. Sometimes people get credit for simple bug fixes (count me among them) so the author of the first working implementation deserves some recognition IMO.

Nadeem: Instead of duplicating the list of archiving/compression modules in each doc, what about only linking to the shutil doc for archives and the archiving.rst file? (I can make the patch, just wanted feedback first)

> Not meaning to sound petty, but wouldn't it be common etiquette to
> retain some original copyright notice from original code intact..?

It seemed to me that Nadeem had rewritten everything from scratch. Is
there any code of yours in the current module?

That is correct. Based on my experience with the bz2 module, rewriting
from scratch seemed to make more sense.

> Oh well, I thought that I'd still earned a note with some slight
credit at least

I completely agree. Sometimes people get credit for simple bug fixes
(count me among them) so the author of the first working implementation
deserves some recognition IMO.

Of course. How does this look?

diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -502,6 +502,7 @@
 Peter van Kampen
 Rafe Kaplan
 Jacob Kaplan-Moss
+Per Øyvind Karlsen
 Lou Kates
 Hiroaki Kawai
 Sebastien Keim
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -400,6 +400,7 @@
 \-------

 - Issue bpo-6715: Add a module 'lzma' for compression using the LZMA algorithm.
\+  Thanks to Per Øyvind Karlsen for the initial implementation.

 - Issue bpo-13487: Make inspect.getmodule robust against changes done to
   sys.modules while it is iterating over it.

Instead of duplicating the list of archiving/compression modules in
each doc, what about only linking to the shutil doc for archives and
the archiving.rst file?

Sure, go ahead. I actually hadn't realized that each section of the
library docs had a "sub-index" page like that until now...

--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -400,6 +400,7 @@
-------

 - Issue bpo-6715: Add a module 'lzma' for compression using the LZMA algorithm.
\+  Thanks to Per Øyvind Karlsen for the initial implementation.

The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message should be enough. Lately I’ve noticed some attributions in NEWS, but it’s usually not done (as redundant).

what about a mention in lzmamodule.c?

I actually hadn't realized that each section of the library docs had a "sub-index" page like that
That’s multiple-entry navigation :) You can jump to a module page, or use the index, or use the search, or browse and see the sub-indexes.

As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the archiving subsection, there’s already a link to the subsection index, so I just removed the “See also zlib, etc.” lines (except for the link from zlib to gzip). I added a link from archiving.rst to shutil and more links and reST targets in shutil.

The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message
should be enough. Lately I’ve noticed some attributions in NEWS, but
it’s usually not done (as redundant).

I generally add attributions in NEWS since that's where most people
learn about changes (for changes which aren't in whatsnew).

New changeset 6cde416ef03b by Nadeem Vawda in branch 'default':
Credit Per Øyvind Karlsen for the initial implementation of the lzma module (issue bpo-6715).
http://hg.python.org/cpython/rev/6cde416ef03b

As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the
archiving subsection, there’s already a link to the subsection index,
so I just removed the “See also zlib, etc.” lines (except for the link
from zlib to gzip). I added a link from archiving.rst to shutil and
more links and reST targets in shutil.

Looks good to me.

The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message
should be enough. Lately I’ve noticed some attributions in NEWS, but
it’s usually not done (as redundant).

According to Doc/whatsnew/3.3.rst:

"The maintainer will go through Misc/NEWS periodically and add
changes; it's therefore more important to add your changes to
Misc/NEWS than to this file."

what about a mention in lzmamodule.c?

Done and committed.

I’ll commit my doc patch to all branches later.

I checked the 4 red 3.3 builbots and they can’t build _lzma.

I’ll commit my doc patch to all branches later.

OK, great.

I checked the 4 red 3.3 builbots and they can’t build _lzma.

For the record, the 3.x buildbots that currently aren't able to build the
_lzma module are:

• x86 Ubuntu Shared
• sparc solaris10 gcc
• AMD64 Snow Leopard 2
• x86 debian parallel
• ARM Ubuntu
• x86 Tiger
• AMD64 OpenIndiana
• x86 OpenIndiana
• x86 FreeBSD 7.2
• x86 FreeBSD 6.4

Additionally, the AMD64 debian bigmem, PPC Tiger and PPC Leopard builders
have been offline/broken since before changeset 74d182cf0187 was committed.

Le jeudi 08 décembre 2011 à 17:40 +0000, Nadeem Vawda a écrit :

Nadeem Vawda <nadeem.vawda@gmail.com> added the comment:

> I’ll commit my doc patch to all branches later.

OK, great.

> I checked the 4 red 3.3 builbots and they can’t build _lzma.

For the record, the 3.x buildbots that currently aren't able to build the
_lzma module are:

Ok, I've e-mailed the owners.

Will someone write an entry in the "what's new" file for 3.3?
(Doc/whatsnew/3.3.rst)

Regards

Antoine.

New changeset 66df5ace0eee by Nadeem Vawda in branch 'default':
What's New in Python 3.3: Add entry for lzma module (issue bpo-6715).
http://hg.python.org/cpython/rev/66df5ace0eee

Ok, I've e-mailed the owners.

Thanks. I was just thinking I should send a reminder, in case they
missed the note in my announcement on python-dev.

Will someone write an entry in the "what's new" file for 3.3?

Done

Is there any reason why this issue is still open?

Yes, almost half of the buildbots still don't have the xz-utils headers
installed, and thus are not building/testing the lzma module. I've kept
the issue open as a reminder to myself to follow up on this.

New changeset 11bd2d32b4e8 by Éric Araujo in branch '3.2':
Improve interlinking of archiving/compression modules docs.
http://hg.python.org/cpython/rev/11bd2d32b4e8

New changeset 1cb9b8126534 by Éric Araujo in branch 'default':
Merge edits from 3.2 (bpo-13716, bpo-1040439, bpo-2945, bpo-13770, bpo-6715)
http://hg.python.org/cpython/rev/1cb9b8126534

BTW, any plans on a PyPI backport for fun and profit?

BTW, any plans on a PyPI backport for fun and profit?

At present, no. I'll look into it later in the year, but at the moment
I don't have the time to work on it - I suspect the parts written in C
will require substantial changes to work under 2.x.

New changeset 7c22281e967c by Éric Araujo in branch '2.7':
Improve interlinking of archiving/compression modules docs.
http://hg.python.org/cpython/rev/7c22281e967c

Why is this issue still open?

I had intended to wait until the code had been tested on all of the
buildbots before closing it. However, it's taking a while to get xz-utils
installed on the last few bots, so it doesn't make sense to keep the
issue open for this.

Apologies for noise, but since a backport was discussed, I'm mentioning this here.

I've started implementing a backport, currently working and tested on Mac OS X and Linux, back to Python 3.0 - supporting Python 2 would be nice but probably significantly more work (assistance welcome - please get in touch on github):
https://github.com/peterjc/backports.lzma

Assuming Nadeem has no objections, I intend to publish this on PyPI (I have tried to email directly but perhaps I'm using an old email address or he has been busy, another reason for commenting here). Thanks!

Apologies again for the noise, but I've just made the first public release of the lzma backport at http://pypi.python.org/pypi/backports.lzma/ with the repository as mentioned before at https://github.com/peterjc/backports.lzma

I have tested this on Python 2.6, 2.7 and 3.0 through 3.3 under Linux and Mac OS. I've not tried building this on Windows yet, but it should be possible and I will be reviewing Amaury's notes on this thread.

My thanks to Nadeem and Per for looking over this with helpful feedback, and agreeing to the use of the 3-clause BSD license for the backport.

If anyone has any further questions about the backport, please get in touch via GitHub or email me.