Issue6715
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2009-08-17 09:47 by devurandom, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
py3k-lzmamodule.patch | proyvind, 2010-10-28 22:09 | lzmamodule patch against py3k branch | ||
teststring.lzma | proyvind, 2010-10-28 22:10 | Lib/test/teststring.lzma | ||
teststring.xz | proyvind, 2010-10-28 22:11 | Lib/test/teststring.xz | ||
lzma-win32.diff | amaury.forgeotdarc, 2011-10-11 00:42 | |||
9276fc685c05.diff | nadeem.vawda, 2011-11-19 22:07 | review | ||
archiving-modules-links.diff | eric.araujo, 2011-11-30 16:11 | review |
Repositories containing patches | |||
---|---|---|---|
http://hg.python.org/sandbox/nvawda#lzma-module |
Messages (138) | |||
---|---|---|---|
msg91657 - (view) | Author: (devurandom) | Date: 2009-08-17 09:47 | |
Python currently supports zlib, gzip and bzip2 compressors. What is missing is support for xz (http://tukaani.org/xz/). It comes with a C library. |
|||
msg91658 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2009-08-17 11:37 | |
Is zc really a C library? I could find a standalone program, but no shared object. Actually, it seems that zc is a file format based on the lzma algorithm. The plan could be to first implement the lzma module (issue5689), then a xzfile module in pure python. |
|||
msg91660 - (view) | Author: Skip Montanaro (skip.montanaro) * | Date: 2009-08-17 11:51 | |
What is xz compression and why is it important? Skip |
|||
msg91661 - (view) | Author: (devurandom) | Date: 2009-08-17 12:13 | |
Yes, xz-utils contains a C library, though it still caries the name "liblzma.so", probably for historic reasons. You are right that xz is a file format based around the lzma algorithm. It just uses a more advanced container format. (lzma-utils had no container at all.) xz is the successor of lzma, which provides a better compression than bzip2, while decompression speed is comparable with gzip. It is used by the GNU project for source tarball compression (replacing bzip2) and supported by GNU tar. See http://en.wikipedia.org/wiki/Xz, http://tukaani.org/xz/ and http://tukaani.org/lzma/ for reference. |
|||
msg92163 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2009-09-02 11:02 | |
Are xz and lzma formats compatible with each other? If not, which one is the most popular? |
|||
msg92167 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2009-09-02 14:22 | |
As I understand it from the http://tukaani.org/xz/ page, .lzma and .xz are two file formats bases on the lzma compression method. - .lzma simply stores the compressed stream. - .xz is more complex |
|||
msg92174 - (view) | Author: (devurandom) | Date: 2009-09-02 18:10 | |
.lzma is actually not a format. It is just the raw output of the LZMA1 coder. XZ instead is a container format for the LZMA2 coder, which probably means LZMA+some metadata. XZ is the official successor to .lzma, and GNU is using it already (look at coreutils), and GNU tar officially supports it since 1.22. |
|||
msg98774 - (view) | Author: Garen (Garen) | Date: 2010-02-03 06:27 | |
Once Python gets native support for lzma/xz like it does for zlib, bzip2 it could switch to using it for bundles and remote transfers. See: http://mercurial.selenic.com/bts/issue1463 With lzma/xz being able to compress so much better, it'd be really appreciated by users on especially slow links(!!). |
|||
msg98776 - (view) | Author: Garen (Garen) | Date: 2010-02-03 06:34 | |
Ugh, can't edit previous message. Meant to say: "Once Python gets native support for lzma/xz like it does for zlib and bzip2, Mercurial could switch to using it for bundles and remote transfers." For platforms with native support in-kernel (e.g. Linux) that could be used instead of the bundled version. (Since Python is officially switching to Mercurial, arguably this issue even more important.) |
|||
msg98794 - (view) | Author: Arkadiusz Miśkiewicz (arekm) | Date: 2010-02-03 19:44 | |
About why xz is important. gnu.org, tug.org started publishing sources in xz format, quick grep: autoconf/autoconf.spec:Source0: http://ftp.gnu.org/gnu/autoconf/%{name}-%{version}.tar.xz coreutils/coreutils.spec:Source0: http://ftp.gnu.org/gnu/coreutils/%{name}-%{version}.tar.xz libpng12/libpng12.spec:Source0: http://downloads.sourceforge.net/libpng/libpng-%{version}.tar.xz libpng/libpng.spec:Source0: http://downloads.sourceforge.net/libpng/%{name}-%{version}.tar.xz parted/parted.spec:Source0: http://ftp.gnu.org/gnu/parted/%{name}-%{version}.tar.xz texlive-texmf/texlive-texmf.spec:Source0: ftp://tug.org/texlive/historic/%{year}/texlive-%{version}-texmf.tar.xz xz is also supported by automake as dist target. |
|||
msg98806 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-02-04 00:40 | |
We all agree that lzma/xz is important, what is needed is a patch. |
|||
msg98899 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-02-05 19:35 | |
There's a Python binding for some lzma lib here: https://launchpad.net/pyliblzma |
|||
msg101941 - (view) | Author: tdjacr (thedjatclubrock) | Date: 2010-03-30 14:41 | |
Once xz is implemented, xz compatibility should be added to the tarfile library. |
|||
msg106427 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-05-25 11:20 | |
Ooops, I kinda should've commented on this issue here in stead, rather than in issue5689, so I'll just copy-paste it here as well: I'm the author of the pyliblzma module, and if desired, I'd be happy to help out adapting pyliblzma for inclusion with python. Most of it's code is based on bz2module.c, so it shouldn't be very far away from being good 'nuff. What I see as required is: * clean out use of C99 types etc. * clean up the LZMAOptions class (this is the biggest difference from the bz2 module, as the filter supports a wide range of various options, everything related such as parsing, api documentation etc. was placed in it's own class, I've yet to receive any feedback on this decission or find any remote equivalents out there to draw inspiration from;) * While most of the liblzma API has been implemented, support for multiple/alternate filters still remains to be implemented. When done it will also cause some breakage with the current pyliblzma API. I plan on doing these things sooner or later anyways, it's pretty much just a matter of motivation and priorities standing in the way, actual interest from others would certainly have a positive effect on this. ;) For other alternatives to the LGPL liblzma, you really don't have any, keep in mind that LZMA is "merely" the algorithm, while xz (and LZMA_alone, used for '.lzma', now obsolete, but still supported) are the actual format you want support for. The LZMA SDK does not provide any compatibility for this. |
|||
msg106430 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2010-05-25 12:05 | |
I will happily review any implementation, and I can help with inclusion into python trunk. > ...the LGPL liblzma... Can you check which licences cover the different parts of the module? I think that you will have to contribute your code under the Python Contributor Agreement; and I just grabbed some copy of the "xz-utils" source package, and it states that "liblzma is in the public domain". |
|||
msg106433 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-05-25 13:06 | |
ah, you're right, I forgot that the license for the library had changed as well (motivated by attempt of pleasing BSD people IIRC;), in the past the library was LGPL while only the 'xz' util was public domain.. For my code, feel free to use your own/any other license you'd like or even public domain (if the license of bz2module.c that much of it's derived from permits of course)! I guess everyone should be happy now then. :) Btw. for review, I think the code already available should be pretty much good 'nuff for an initial review. Some feedback on things not derived from bz2module.c would be nice, especially the LZMAOptions class would be nice as it's where most of the remaining work required for adding additional filters support. Would kinda blow if I did the work using an approach that would be dismissed as utterly rubbish. ;) Oh well, it's out there available for anyone already, I probably won't(/shouldn't;) have time for it in a month at least, do as you please meanwhile. :) |
|||
msg106441 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-05-25 15:34 | |
Hello, > I'm the author of the pyliblzma module, and if desired, I'd be happy > to help out adapting pyliblzma for inclusion with python. > Most of it's code is based on bz2module.c, so it shouldn't be very far > away from being good 'nuff. Well, I wouldn't say bz2module is the best module out there, but as you say it's probably good enough :) And we can help you fix things if needed. Is pyliblzma compatible with Python 3.x? It's too late to incorporate any new feature in Python 2.x now. > * While most of the liblzma API has been implemented, support for > multiple/alternate filters still remains to be implemented. When done > it will also cause some breakage with the current pyliblzma API. Hmm, then perhaps you should first fix the current API so that adding new features doesn't force you to break the API again. There are strict rules for API breakage in the standard library. By the way, adding a new module to the stdlib probably requires writing a PEP (Python Enhancement Proposal). I wouldn't expect this very proposal to be controversial, but someone has to do it. Finally, when a module is in the stdlib, it is expected that maintenance primarily happens in the Python SVN (or Mercurial) tree. We have a couple of externally-maintained modules, but they're a source of problems for us. |
|||
msg106567 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-05-26 18:48 | |
Yeah, I guess I anyways can just break the current API right away to make it compatible with future changes, I've already figured since long ago how it should look like. It's not like I have to implement the actual functionality to ensure compatibility, no-op works like charm. ;) |
|||
msg106572 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2010-05-26 19:54 | |
[Replying to msg106566] > if you're already looking at issue6715, then I don't get why you're > asking.. ;) Can you please submit a contributor form? > Martin: For LGPL (or even GPL for that matter, disregarding linking > restrictions) libraries you don't have to distribute the sources of > those libraries at all (they're already made available by others, so > that would be quite overly redundant, uh?;). LGPL actually doesn't > even care at all about the license of your software as long as you > only dynamically link against it. Of course you do. Quoting from the LGPL "You may convey a Combined Work ... if you also do each of the following: ... d) Do one of the following: 0) Convey the Minimal Corresponding Source under the terms of this License, and the Corresponding Application Code in a form suitable for, and under terms that permit, the user to recombine or relink the Application with a modified version of the Linked Version to produce a modified Combined Work, in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source. 1) [not applicable to Windows] " > I don't really get what the issue would be even if liblzma were still > LGPL, it doesn't prohibit you from distributing a dynamically linked > library along with python either if necessary (which of course would > be of convenience on win32..).. Of course I can distribute a copy of an lzma DLL. However, I would have to provide ("convey") a copy of the source code of that DLL as well. |
|||
msg106578 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-05-26 20:47 | |
> Of course I can distribute a copy of an lzma DLL. However, I would > have to provide ("convey") a copy of the source code of that DLL as > well. Can you tell me where you are currently providing the source code for the readline library, or the gdbm library? Oh, and by the way, you should probably shut down PyPI, since there are certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed one), and you aren't offering a link to download those libraries' source code either. You seem to have no problem "conveying" copies for the source code of non-LGPL libraries such as OpenSSL. Why is that? |
|||
msg106580 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2010-05-26 21:01 | |
> Can you tell me where you are currently providing the source code for > the readline library, or the gdbm library? We don't, as they aren't included in the Windows distribution. The readline library doesn't work on Windows, anyway, and instead of gdbm, we had traditionally been distributing bsddb instead on Windows. > Oh, and by the way, you should probably shut down PyPI, since there are > certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed > one), and you aren't offering a link to download those libraries' source > code either. This is off-topic for this bug tracker. Please remain objective and professional if you can manage to. > You seem to have no problem "conveying" copies for the source code of > non-LGPL libraries such as OpenSSL. Why is that? Not sure what you are referring to. We don't provide the sources for the OpenSSL libraries along with the Windows installer, because the license of OpenSSL doesn't require us to. This is very convenient for our users. |
|||
msg106581 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-05-26 21:34 | |
> > Oh, and by the way, you should probably shut down PyPI, since there are > > certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed > > one), and you aren't offering a link to download those libraries' source > > code either. > > This is off-topic for this bug tracker. Please remain objective and > professional if you can manage to. This whole subthread was already off-topic (since it was pointed out, before your previous message, that the underlying lib is in the public domain). Actually, I would argue that the whole idea of promoting a rigorous interpretation of a license has no place on the bug tracker. It makes no sense to do this on an ad hoc fashion, especially if you want lawyers to be involved (they are certainly not reading this). (of course, you will also have understood that I disagree with such a rigorous interpretation) > Not sure what you are referring to. We don't provide the sources for the > OpenSSL libraries along with the Windows installer, because the license > of OpenSSL doesn't require us to. This is very convenient for our users. This was not about providing the sources together with the installer (which even the GPL or the LGPL don't require to do), but providing them as a separate bundle on the download site. We do have a copy of the OpenSSL source tree somewhere, it is used by the Windows build process. |
|||
msg106710 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-05-29 06:48 | |
I've ported pyliblzma to py3k now and also implemented the missing functionality I mentioned earlier, for anyone interested in my progress the branch is found at: https://code.launchpad.net/~proyvind/pyliblzma/py3k I need to fix some memory leakages (side effect of the new PyUnicode/Pybytes change I'm not 100% with yet;) and some various memory errors reported by valgrind etc. though, but things are starting to look quite nice already. :) |
|||
msg119742 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-10-27 22:30 | |
I've (finally) finalized the api and prepared pyliblzma to be ready for inclusion now. The code can be found in the 'py3k' branch referred to earlier. Someone else (don't remember who:p) volunteered for writing the PEP earlier, so I leave it up to that person to write the PEP, I won't be able to get around to do so myself in the near future.. |
|||
msg119743 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2010-10-27 23:40 | |
As promised, here is a quick review of the module. https://code.launchpad.net/~proyvind/pyliblzma/py3k looks ready for a new entry in the PyPI, but for inclusion in core python it needs some cleanup: - I suppose that src/pyliblzma.c is the only interesting file. In CPython it will be named Modules/lzmamodule.c - There are some calls to malloc(), calloc() and free() that are incorrect (there is even a call to free() on a PyObject*!); in any case, functions like PyMem_Malloc() are preferred. - please don't use alloca (and it has another name on Windows) - snprintf is not available everywhere; use PyOS_snprintf instead. - The module does not compile on Windows: types like uint8_t don't exist, __attribute__((unused)) is GCC-specific. - Don't use PyBytesObject*; PyObject* is enough and avoids many casts. - PyLong_FromLong(LZMA_FILTER_LZMA1) truncates the value on 32bit platforms - You could use LZMA_VERSION_STRING to fill the version attribute. Also, I did not find any documentation. Otherwise the code follows CPython conventions and is easy to read. Good job! 3.2beta1 is scheduled for November 13, 2010, and no new feature will be accepted after. Do you think you can update it before the limit? otherwise the package could live in PyPI before it is shipped with Python. |
|||
msg119777 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-10-28 13:25 | |
All fixed now. :) |
|||
msg119779 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2010-10-28 13:38 | |
> All fixed now. :) Does that refer to msg119743? Please provide this module as a patch for branches/py3k, otherwise it's difficult to see what exactly you are contributing. Also, please provide Doc/library/lzma.rst. |
|||
msg119815 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-10-28 22:09 | |
Here's a patch with the latest code generated against py3k branch, it comes with Doc/library/lzma.rst as well now. |
|||
msg119816 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-10-28 22:10 | |
here's Lib/test/teststring.lzma, required by the test suite. |
|||
msg119817 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-10-28 22:11 | |
here's Lib/test/teststring.xz, required by the test suite. |
|||
msg119826 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-10-28 23:37 | |
Could you upload the patch to http://codereview.appspot.com/? |
|||
msg119862 - (view) | Author: Georg Brandl (georg.brandl) * | Date: 2010-10-29 06:54 | |
After applying the patch, it builds fine here and the test suite passes. However, it seems to leak quite a bit -- if I run regrtest with -R::, my system starts swapping heavily after the second run. In lzmamodule, there are lots of API calls that aren't error-checked, e.g. PyDict_SetItemString(filter, "dict_size", PyLong_FromLong((long)lzma_options.dict_size)); General comments: please don't introduce tabs and trailing whitespace in C and Python files, and wrap your lines at 79 characters whenever possible. reST function/class directives need a blank line after the signature. (But don't worry about the markup too much, I'll review the new file anyway after commit.) @pitrou: Why is this marked "Python 3.3"? If the error handling in the C module is corrected, it's in a good enough shape to be committed before 3.2b1, and the remaining bugs ironed out until final. |
|||
msg119865 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-10-29 07:00 | |
> @pitrou: Why is this marked "Python 3.3"? If the error handling in > the C module is corrected, it's in a good enough shape to be committed > before 3.2b1, and the remaining bugs ironed out until final. I think it needs a real review before going in. Now if that review can get done and the issues fixed before the beta, it's ok. By a quick look at the code it seemed to need quite a bit of polish before being acceptable for commit, but I might be mistaken. (we should also avoid the multiprocessing syndrome where we rushed some alpha-quality code just before the feature deadline and then had to fix many issues in urgency. I do believe xz/lzma support is important in the long term, though) Oh, by the way, Per should agree to do maintenance directly against the CPython repository. Please, no more externally-maintained modules (you know what I'm talking about). |
|||
msg119866 - (view) | Author: Georg Brandl (georg.brandl) * | Date: 2010-10-29 07:04 | |
Yes, definitely no externally maintained modules. |
|||
msg120050 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-10-31 15:04 | |
I've uploaded a new version of the patch to http://codereview.appspot.com/2724043/ now. I'd be okay on doing maintenance directly against the CPython repository btw. :) |
|||
msg120068 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2010-10-31 16:43 | |
After starting to review the code, I'm becoming skeptical whether this is the right approach. This does way to much action in C, and thus becomes complicated but also limited. An alternative approach would be to just expose lzma_code to Python, and then integrate that into the io architecture, i.e. as a subclass of RawIOBase. I.e. LZMAFile would go; if you want to compress to a file, use FileIO instead, and wrap that with a LZMAIO (say). LZMACompressor and Decompressor could stay if desired, although it seems more liblzma-like to have a single object for both compression and decompression. In addition, I find the options object too complicated. It seems to serve documentation purposes only, so I wonder whether it could be reduced in code size. |
|||
msg120084 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-10-31 20:57 | |
LZMAFile, LZMACompressor & LZMADecompressor are all inspired by and written to be as similar to bz2's for easier use & maintenance. I must admit that I haven't really put much thought into alternate ways to implement them beyond monkey see, monkey do.. ;) LZMAOptions is a bit awkwardly written, but it doesn't serve documentation purposes only, it also exposes these values for max, min etc. to python (ie. as used by it's regression tests) and are also used when processing various compression options passed. IMO it does serve a useful purpose, but certainly wouldn't hurt from being rewritten in some better way... |
|||
msg120085 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2010-10-31 21:15 | |
I feel guilty of having said, some months ago, that "Well, I wouldn't say bz2module is the best module out there, but as you say it's probably good enough" (my words). It's true that bz2module is not awful in terms of coding style or quality; the main issue with it is that uses outdated ways of doing I/O compared to what Python 3 recommends (that is, compose various building blocks together to allow for better re-use). A point of reference is gzip, where the core compression routines (mostly wrappers) are in zlibmodule (in C), but the high-level file object is written in pure Python (and is composeable with other building blocks for fast buffered IO). |
|||
msg120086 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2010-10-31 21:24 | |
Hehe, don't feel guily on my part at least, I had already implemented it like this long before. :p I guess I could rewrite it following these suggestions, but I probably won't be able to finish it in time for 3.2 beta. |
|||
msg120089 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2010-10-31 21:35 | |
> LZMAFile, LZMACompressor & LZMADecompressor are all inspired by and > written to be as similar to bz2's for easier use & maintenance. I > must admit that I haven't really put much thought into alternate ways > to implement them beyond monkey see, monkey do.. ;) My concern really is with LZMAFile only. It uses stdio, and it really shouldn't - we are trying to drop stdio in Python, as it's causing too many problems. LZMACompressor/Decompressor may be fine. > LZMAOptions is a bit awkwardly written, but it doesn't serve > documentation purposes only, it also exposes these values for max, > min etc. to python (ie. as used by it's regression tests) and are > also used when processing various compression options passed. > > IMO it does serve a useful purpose, but certainly wouldn't hurt from > being rewritten in some better way... If you are willing to do that, here is what I propose: rename the module to _lzma, drop options and lzmafile from _lzma. Then write lzma.py, which re-introduces this Options class in pure Python, as well as reintroduces LZMAFile, wrapping io.FileIO. The various constants that go into options should be exposed from _lzma using PyModule_AddIntConstant (except perhaps for the ones that you define yourself, such as LZMA_BEST_SPEED) That assumes, of course, that there is a need to provide backwards compatibility. I would personally be fine with *just* having the LZMA symbolic constants exposed, leaving it to the user to determine which constant serves which purpose (e.g. guessing what LZMA_DICT_SIZE_MIN and LZMA_DICT_SIZE_MAX do when there is a dict_size keyword parameter isn't too hard). But then, having the various options explained in a consistent manner may be useful enough to preserve the Options class. |
|||
msg120091 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2010-10-31 21:46 | |
> I guess I could rewrite it following these suggestions, but I probably won't be able to > finish it in time for 3.2 beta. The more I think about it, the more I feel like -1 about this code as long as it uses stdio, and does buffering. We really don't need another implementation of splitting a stream of bytes into lines. |
|||
msg122424 - (view) | Author: Dan Stromberg (strombrg) | Date: 2010-11-26 00:51 | |
While I realize Python extension modules are pretty much the norm in CPython, it'd be pretty cool if xz support could be written overtop of ctypes. ctypes seems to be emerging as the way of doing FFI across different python implementations. I've been working on a backup engine in my spare time, and I'd very much like to use a Python xz module in it. However, the code I have so far runs about 4x faster on pypy than CPython, even if I turn on psyco in CPython. IOW, I'd be pretty pleased to see an xz module that works equally well on CPython, pypy (seems to have pretty good ctypes), jython (has the beginnings of ctypes in 2.5.2rc*), iron python (supposedly included recently)... |
|||
msg122426 - (view) | Author: Éric Araujo (eric.araujo) * | Date: 2010-11-26 01:20 | |
I’m afraid ctypes is too insecure for stdlib code. Even using it in some tests had to be discussed and decided at EuroPython. I’m told that PyPy has CPyExt now, so this may solved your compat concern :) |
|||
msg122435 - (view) | Author: Dan Stromberg (strombrg) | Date: 2010-11-26 04:51 | |
I agree that ctypes is a bit more brittle - both ctypes and c extension modules can yield segfaults, but at least the c extension module is likely to give an error or warning when you rebuild it. However, I'm getting the impression that: 1) In pypy, the C extension module layer isn't a complete emulation 2) In pypy, the C extension module is a bit disfavored compared to ctypes 3) In pypy, there's a performance expense for CPyExt compared to ctypes 3) (less important) In pypy, the C extension module layer is still too young to do much with at this time I've been wondering if maybe there should be a way to check ctypes use against .h's at build time. That way, the brittleness should be about the same. |
|||
msg122443 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2010-11-26 07:32 | |
Careful: the ctypes module is much slower with pypy than with cpython. In any case, a C extension module is preferred because the (platform-dependent) details of the liblzma library are defined in a .h header file, not as a ctypes description. Future versions of the library are likely to be source-compatible, but may change some struct member for example; this would break a ctypes-based interface. |
|||
msg122444 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2010-11-26 07:41 | |
> While I realize Python extension modules are pretty much the norm in > CPython, it'd be pretty cool if xz support could be written overtop > of ctypes. ctypes seems to be emerging as the way of doing FFI > across different python implementations. There is a clear policy in CPython that ctypes must not be used to implement library modules. |
|||
msg127384 - (view) | Author: shirish (shirish) | Date: 2011-01-29 07:52 | |
no programmer or anything. Just adding that the wikipedia link given is not right. The correct link is https://secure.wikimedia.org/wikipedia/en/wiki/XZ_Utils Just my 2 paise. |
|||
msg133214 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-04-07 13:02 | |
Having "Christophe Simonis" in the nosy list is precluding NOSY changes. I guess the space is causing problems. Should we forbid spaces in usernames (maybe provided by OpenID, who knows)??? What do we need to progress in this issue?. A rewrite to use Python 3 IO facilities?. |
|||
msg133216 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-04-07 13:10 | |
> Having "Christophe Simonis" in the nosy list is precluding NOSY > changes. I guess the space is causing problems. Should we forbid > spaces in usernames (maybe provided by OpenID, who knows)??? Please report this in the meta-tracker (link on the left). > What do we need to progress in this issue?. > > A rewrite to use Python 3 IO facilities?. See Nadeem's recent work on the bz2 module. |
|||
msg133218 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-04-07 13:30 | |
Reported to metatracker. Thanks for the kick :-). As a reference, I think you are refering to #5863 and #10791. Pretty fine job, I must say. Nadeem, what do you think?. Is xz in your list?. |
|||
msg133280 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-04-08 00:20 | |
> As a reference, I think you are refering to #5863 and #10791. Pretty fine job, I must say. Thank you :) > Nadeem, what do you think?. Is xz in your list?. Yes, it's the next substantial thing I was planning on working on. I don't have a lot of free time at the moment, though, so it might take a while. But I'll definitely have it done in time for the release of 3.3 ;) Looking at the pyliblzma code, I see that it provides the same thread-safety guarantees as bz2 does. In rewriting, I would like to remove the locking code, as it complicates the control flow logic of the C code, and doesn't seem like the sort of thing a compression library should be concerned with. Are there any objections to this? (Also, if anyone knows why bz2 was written to be thread-safe in the first place, please enlighten me) |
|||
msg133282 - (view) | Author: Dan Stromberg (strombrg) | Date: 2011-04-08 00:50 | |
I don't know that much about compression, but I wonder if a threadsafe compression module would enable parallel forms of compression? If yes, then multithreaded might be a big benefit, in light of multicore taking off. EG: http://www.compression.ca/pbzip2/ It might be worthwhile to check with the authors (of bzip2 and pyliblzma modules) about why they went with thread safety. |
|||
msg133285 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-04-08 01:36 | |
I think doesn't makes sense for two threads using at the same time the same compress/decompress object. But I could guess ony reason for the locking code is to avoid crashing the interpreter just having two threads racing each other on an inherently mutable object. Dan, parallel compression simply split the input data and compress each fragment independly. The actual compression object is not shared, each block is compressed with its own self-contained compression context. |
|||
msg136059 - (view) | Author: Dan Stromberg (strombrg) | Date: 2011-05-16 01:48 | |
Interesting thing to consider: maybe it'd be better to add support for libarchive, which includes xz support among other things. http://code.google.com/p/libarchive/ |
|||
msg136070 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-05-16 07:13 | |
> maybe it'd be better to add support for libarchive That wouldn't be better. We want the C dependency as small and low-level as possible, so that the dependency is small and the functionality flexible to implement stuff in Python on top of it. |
|||
msg136078 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-05-16 11:25 | |
Also, if I'm reading the documentation correctly, there isn't any support for incremental in-memory coding, so it wouldn't be possible to implement classes analogous to BZ2Compressor/BZ2Decompressor. |
|||
msg143173 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-08-29 18:53 | |
Attached is a patch (f3cf187208ea.diff) containing my work so far on reimplementing the lzma module. So far I've just done the LZMACompressor and LZMADecompressor classes, but I'm hoping to implement LZMAFile this weekend. |
|||
msg143756 - (view) | Author: Nam Nguyen (Nam.Nguyen) * | Date: 2011-09-09 06:52 | |
In uint32_converter, I'm not sure the if statement comparing val and UINT32_MAX makes sense. val was defined to be unsigned long, which is 32bit on 32-bit compiler. |
|||
msg143757 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-09-09 08:08 | |
> In uint32_converter, I'm not sure the if statement comparing val and > UINT32_MAX makes sense. val was defined to be unsigned long, which is > 32bit on 32-bit compiler. Yes, on a 32-bit system, this comparison will always be false. However, on a 64-bit system, an unsigned long will (usually) be 64 bits wide. In this case, we do need to check that the value fits into a uint32_t. |
|||
msg143876 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-09-11 21:01 | |
I've attached another patch (fe2c9998f329.diff) with a more complete implementation of the lzma module. All that's left now is to write the documentation, and make sure that the module can build on Windows. |
|||
msg144066 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-09-15 08:15 | |
I've attached another patch that includes proper ReST documentation, a few more tests, and some fixups following Ezio and Amaury's reviews. |
|||
msg144085 - (view) | Author: Lars Gustäbel (lars.gustaebel) * | Date: 2011-09-15 16:14 | |
Today I played around with lzma support for tarfile based on your last patch (see issue5689). There are a few minor issues that I just wanted to mention, as they break the tarfile testsuite: - LZMAFile does not expose a name attribute. BZ2File doesn't either (not in 3.x anyway), but GzipFile does. - LZMAFile does not allow a 'b' in the mode argument, unlike GzipFile and BZ2File. - The bz2 module exposes many error conditions as standard Python exceptions, e.g. IOError, EOFError. The lzma module uses LZMAError for all errors without distinction. |
|||
msg144107 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-09-15 22:03 | |
Thanks for getting the ball rolling with that. I'll want to try and familiarize myself with the code for tarfile module before I start changing stuff, but those points shouldn't be hard to fix, one way or the other. |
|||
msg144118 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-09-16 10:10 | |
Nadeem, I started reviewing this path, but now stopped since the patch I reviewed isn't available anymore. Please let us know when the patch has a state where you don't plan to make more changes. |
|||
msg144119 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-09-16 10:27 | |
Ah, sorry about that. It probably wasn't necessary to regenerate the patch after I addressed the comments in each review. The patch should be stable now; I don't plan to make any further changes for a while. (At some point I'll want to add a few more tests, and get the module to compile on Windows. But that can wait.) |
|||
msg144162 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-09-16 21:15 | |
I'm a bit worried by the Windows version: - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly variables declared in the middle of a block. It's doable for sure, but it's a lot of work. - liblzma is normally compiled with mingw, but we have to be sure that is uses the correct MSCRT C runtime, and what about debug builds? - All extension modules use static libraries: zlib, expat, sqlite. But a gcc static library can't be used by Visual Studio. - The way recommended by XZ is to use a precompiled liblzma.dll; Then it should be easy to build an extension module, but its would be the first time that we distribute an extension module which needs a non-system DLL. Is it enough to copy it next to _lzma.pyd? Is there some work to do in the installer? Too many "but"s :) |
|||
msg144788 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-10-02 22:49 | |
Thanks for investigating the Windows situation. > - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly > variables declared in the middle of a block. It's doable for sure, but it's a > lot of work. I don't think that creating our own MSVC-friendly fork of liblzma is really an option. Over and above the work of porting it in the first place (and all the opportunities for bugs to creep in along the way), we'd also have to worry about keeping up to date with upstream changes. I believe we currently do something similar with libffi (for ctypes), and the impression I've gotten is that it's caused a lot of trouble. > - The way recommended by XZ is to use a precompiled liblzma.dll; Then it > should be easy to build an extension module, but its would be the first time > that we distribute an extension module which needs a non-system DLL. Is it > enough to copy it next to _lzma.pyd? Is there some work to do in the > installer? I would guess that this is sufficient, but my knowledge of how Windows DLLs work is minimal. Could someone with more platform knowledge weigh in on whether this would work (and if there are any problems it might cause)? |
|||
msg144790 - (view) | Author: Dan Stromberg (strombrg) | Date: 2011-10-02 22:55 | |
On Sun, Oct 2, 2011 at 3:49 PM, Nadeem Vawda <report@bugs.python.org> wrote: > > Nadeem Vawda <nadeem.vawda@gmail.com> added the comment: > > Thanks for investigating the Windows situation. > > > - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly > > variables declared in the middle of a block. It's doable for sure, but > it's a > > lot of work. > > I don't think that creating our own MSVC-friendly fork of liblzma is really > an > option. Over and above the work of porting it in the first place (and all > the > opportunities for bugs to creep in along the way), we'd also have to worry > about > keeping up to date with upstream changes. I believe we currently do > something > similar with libffi (for ctypes), and the impression I've gotten is that > it's > caused a lot of trouble. It's much better to contribute patches upstream. > > - The way recommended by XZ is to use a precompiled liblzma.dll; Then it > > should be easy to build an extension module, but its would be the first > time > > that we distribute an extension module which needs a non-system DLL. Is > it > > enough to copy it next to _lzma.pyd? Is there some work to do in the > > installer? > > I would guess that this is sufficient, but my knowledge of how Windows DLLs > work > is minimal. Could someone with more platform knowledge weigh in on whether > this > would work (and if there are any problems it might cause)? I've not done much with windows dll's, but I've heard they're pretty similar to AIX shared libraries which I've done some work with. AIX shared libraries don't deal with versioning well - if you have two version of the same library on a system, you have to pop them into two different loader domains, or suffer unresolved externals at runtime. Then your applications are in of the two loader domains, and if they're in the wrong one, you again suffer unresolved externals at runtime. |
|||
msg144866 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-10-04 08:12 | |
Based on Amaury's report, I would suggest going forward integrating the xz module for configure-based systems, and letting someone else handle Windows integration later if a solution is found. |
|||
msg144901 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-04 16:56 | |
> - liblzma can't be compiled by Visual Studio: too many C99 isms, > mostly variables declared in the middle of a block. It's doable for > sure, but it's a lot of work. I'd be in favor of doing so, and then feeding patches upstream. Hopefully, eventually, the code would compile out of the box on VS. > - liblzma is normally compiled with mingw, but we have to be sure > that is uses the correct MSCRT C runtime, and what about debug > builds? In principle, it's not necessary to use the same CRT, as long as we aren't passing CRT objects across DLL boundaries (memory blocks managed by malloc/free would be candidates). I haven't reviewed the module to find out whether the liblzma interface involves CRT objects. > - The way recommended by XZ is to use a precompiled liblzma.dll; Then > it should be easy to build an extension module, but its would be the > first time that we distribute an extension module which needs a > non-system DLL. Is it enough to copy it next to _lzma.pyd? Is there > some work to do in the installer? It wouldn't actually be the first time. We also ship Tcl DLLs. But it's a pain, so it would be much better if the sources were actually referenced in the VS project - so we would not need a library at all. |
|||
msg144902 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-04 17:00 | |
> Based on Amaury's report, I would suggest going forward integrating > the xz module for configure-based systems, and letting someone else > handle Windows integration later if a solution is found. -1000. I feel quite strongly that this should not be added unless there is also support in the Windows build process for it. I'll see what I can do, but it may take some time - until then, I urge to stall this issue, i.e. not proceed with checking it in. |
|||
msg144904 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-10-04 17:07 | |
> > Based on Amaury's report, I would suggest going forward integrating > > the xz module for configure-based systems, and letting someone else > > handle Windows integration later if a solution is found. > > -1000. I feel quite strongly that this should not be added unless there > is also support in the Windows build process for it. Why? |
|||
msg144910 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-04 17:43 | |
Am 04.10.11 19:08, schrieb Antoine Pitrou: > > Antoine Pitrou<pitrou@free.fr> added the comment: > >>> Based on Amaury's report, I would suggest going forward integrating >>> the xz module for configure-based systems, and letting someone else >>> handle Windows integration later if a solution is found. >> >> -1000. I feel quite strongly that this should not be added unless there >> is also support in the Windows build process for it. > > Why? This module is only useful in the standard library if it is available on all systems. If it is not available on all systems, it may just as well be available from PyPI only. Now, as it needs to be available on Windows, I want to see the actual Windows support, else we would have to block the release until Windows support is available, possibly then reverting the module because no Windows support is forthcoming. So it's easier not to commit in the first place. |
|||
msg144912 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-10-04 17:46 | |
> This module is only useful in the standard library if it is available > on all systems. Not really. xz is becoming a defacto standard under Linux (and perhaps other free Unices) while I guess it is marginal under Windows. We have other system-specific functionality, and nobody sees it as a bad thing. |
|||
msg144915 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-04 18:05 | |
> Not really. xz is becoming a defacto standard under Linux (and perhaps > other free Unices) while I guess it is marginal under Windows. > We have other system-specific functionality, and nobody sees it as a bad > thing. That's because all system-specific functionality that we have really depends on system features which just can't be available elsewhere. For all functionality that in principle works on all systems, it also actually works on all systems for Python. In cases where stuff was only available on Linux even though it could work on other systems, people *did* see it as a bad thing. |
|||
msg144916 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-10-04 18:08 | |
> That's because all system-specific functionality that we have really > depends on system features which just can't be available elsewhere. > For all functionality that in principle works on all systems, it also > actually works on all systems for Python. In cases where stuff was > only available on Linux even though it could work on other systems, > people *did* see it as a bad thing. Agreed, but it's a probably with the external library. That's like saying we are responsible if libffi fails building with the AIX compiler. |
|||
msg144917 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-10-04 18:08 | |
> > That's because all system-specific functionality that we have really > > depends on system features which just can't be available elsewhere. > > For all functionality that in principle works on all systems, it also > > actually works on all systems for Python. In cases where stuff was > > only available on Linux even though it could work on other systems, > > people *did* see it as a bad thing. > > Agreed, but it's a probably with the external library. That's like > saying we are responsible if libffi fails building with the AIX > compiler. s/probably/problem/ ;) |
|||
msg144938 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-10-05 09:24 | |
I agree with Martin here. We should *NOT* have first and second class OS support, if we can avoid it. That said, I wonder what happens in Windows with the BZ2 module, for instance :-?. Do we include the BZ2 sourcecode to compile it under windows?. I know, for instance, that Windows 2.* builds include Berkeley DB. |
|||
msg144939 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-10-05 09:27 | |
> I agree with Martin here. We should *NOT* have first and second class > OS support, if we can avoid it. The key word being "if we can avoid it". Jesus, if you are a Windows expert, your contribution is welcome. |
|||
msg144940 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-10-05 09:31 | |
For bz2, Tools/buildbot/external-common.bat has code to download bz2 source, and PCbuild/_bz2.vcproj include and compile these files together with _bz2.pyd. The _ssl module does a similar thing, except that libeay32.lib and libssleay32.lib are built in a separate step. |
|||
msg144941 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-05 09:58 | |
> That said, I wonder what happens in Windows with the BZ2 module, for instance :-?. External code currently lives at http://svn.python.org/projects/external/. The build process gets it from there, and we may have local modifications to libraries where necessary. |
|||
msg144942 - (view) | Author: STINNER Victor (vstinner) * | Date: 2011-10-05 10:09 | |
> I agree with Martin here. We should *NOT* have first > and second class OS support, if we can avoid it. Ok but who will do the job? If nobody is motivated to fix compiler issues, it would be a pity to not add the module for that. |
|||
msg144946 - (view) | Author: Jesús Cea Avión (jcea) * | Date: 2011-10-05 10:50 | |
Antoine, I am a Linux/Solaris/Illumos guy. I only use Windows (virtualized) to sync my iPhone with iTunes :) |
|||
msg144992 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-10-06 11:01 | |
Wow, this discussion has gotten quite busy while I've been travelling... Martin, could you explain what the problems are with bundling a precompiled DLL for Windows? I am willing to do the work of getting liblzma to compile with VS if necessary, but I don't know how receptive the upstream maintainer will be to the changes. If I can explain how lack of VS support is a problem for us, the request should carry more weight. |
|||
msg145016 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-06 14:35 | |
Am 06.10.11 13:01, schrieb Nadeem Vawda: > > Nadeem Vawda<nadeem.vawda@gmail.com> added the comment: > > Wow, this discussion has gotten quite busy while I've been travelling... > > Martin, could you explain what the problems are with bundling a precompiled DLL > for Windows? Off-hand, it's only minor issues: e.g. when running Python out of its build directory, the DLL must be in the same directory. Now, since the output directory differs depending on build option, getting the DLL there might be tricky. Things I wonder about and couldn't quickly answer from the web: where exactly is the DLL that we would use? is there a AMD64 version of it? Does it come with import libraries usable by VS? |
|||
msg145020 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-10-06 15:30 | |
On http://tukaani.org/xz, I downloaded the file named xz-5.0.3-windows.zip. It contains precompiled dlls for both platforms: bin_i486/liblzma.dll and bin_x86_64/liblzma.dll Unfortunately, there is no import library for VS. It should not be too difficult to make one, though: the provided headers are C89, so it's enough to write stubs for the functions used by the extension module. |
|||
msg145022 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-10-06 15:43 | |
Hmm... according to http://git.tukaani.org/?p=xz.git;a=blob;f=windows/README-Windows.txt;hb=HEAD#l80, the MinGW-compiled static libs *can* be used with MSVC. Not sure how reliable the information is, but it's worth a try at least. |
|||
msg145023 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-10-06 15:51 | |
Ah indeed, the zip archive contains a doc/liblzma.def which can be used to build a liblzma.lib |
|||
msg145024 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-10-06 15:56 | |
Hey, today I learnt something about mingw! """Rename liblzma.a to e.g. liblzma_static.lib and tell MSVC to link against it.""" Apparently mingw can generate COFF libraries. This may simplify things *a lot*. |
|||
msg145317 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-10-11 00:42 | |
*Very* good news for lzma on windows: The precompiled static library liblzma.a works very well with MSVC (tested with VS2008 on Windows XP, 32bit). This was a surprise for me... Here is a patch for the win32 build files, to be applied after Nadeem's. I did not update the svn "external" repository, for my local copy I simply extracted xz-5.0.3-windows.zip. Tests pass in release and debug builds, and depends.exe shows no special requirements for _lzma.pyd. I could not test win64. Nadeem, in test.support precisionbigmemtest was recently renamed to bigmemtest, could you update your patch? |
|||
msg145338 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-10-11 10:03 | |
Awesome stuff! I'll post an updated patch during the course of the day. Martin: I've been having problems with Rietveld lately, so I'm posting my replies to your comments here instead. >> Modules/_lzmamodule.c:115: return _PyBytes_Resize(buf, size + BIGCHUNK); > This has quadratic performance. Correct. I copied the algorithm from _io.FileIO, under the assumption that there was a reason for not using a simpler O(n log n) doubling strategy. Do you know of any reason for this? Or is it safe to ignore it? >> Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS > It seems that the Windows version at least is not thread-safe. If so, you > would need an LZMA lock when releasing the GIL. Does the class need to be thread-safe, though? ISTM that there isn't any sensible use case for having two threads feeding data through the same compressor concurrently. (If we *do* want thread-safety, then it doesn't matter whether the underlying lib is internally thread-safe or not. We would still need to guard against the possibility of the _lzmamodule.c code in one thread modifying the lzma_stream's input or output pointer while lzma_code is operating on the stream's data in another thread.) |
|||
msg145340 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-10-11 11:39 | |
> >> Modules/_lzmamodule.c:115: return _PyBytes_Resize(buf, size + BIGCHUNK); > > This has quadratic performance. > > Correct. I copied the algorithm from _io.FileIO, under the assumption > that there was a reason for not using a simpler O(n log n) doubling > strategy. Do you know of any reason for this? Or is it safe to ignore it? I don't know, but I'd say it's safe to ignore it. Maybe open a separate bug about that. Generally we use a less-than-doubling strategy, to conserve memory (see e.g. bytearray.c). (note: it is O(n), not O(n log n)) > >> Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS > > It seems that the Windows version at least is not thread-safe. If so, you > > would need an LZMA lock when releasing the GIL. > > Does the class need to be thread-safe, though? ISTM that there isn't any > sensible use case for having two threads feeding data through the same > compressor concurrently. We certainly want to avoid crashes so, if that's what one risks from using lzma from several threads, we should protect against it. |
|||
msg145405 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-12 16:11 | |
>>> Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS >> It seems that the Windows version at least is not thread-safe. If so, you >> would need an LZMA lock when releasing the GIL. > > Does the class need to be thread-safe, though? As a matter of principle, Python code must not be able to crash the interpreter or corrupt memory. There are known bugs in this area, but if it's known in advance that an issue exists, we should avoid it. > ISTM that there isn't any > sensible use case for having two threads feeding data through the same > compressor concurrently. Right. So having a per-compressor mutex lock would be entirely reasonable. I could also accept a per-module lock. I could even accept the GIL, and if no other code is forth-coming, I would prefer to keep holding the GIL during comprssion over risking crashes. > (If we *do* want thread-safety, then it doesn't matter whether the > underlying lib is internally thread-safe or not. We would still need to > guard against the possibility of the _lzmamodule.c code in one thread > modifying the lzma_stream's input or output pointer while lzma_code is > operating on the stream's data in another thread.) I haven't reviewed the module in this respect. If you say that it wouldn't be thread-safe even if LZMA was compiled as thread-safe, then this definitely must be fixed. To elaborate on the policy: giving bogus data in cases of race conditions is ok; crashing the interpreter or corrupting memory is not. If bogus data is given, it would be useful if the bogosity can be specified (e.g. when multiple threads read from the same POSIX file concurrently, they also get bogus data, but in a manner where each input byte is given to exactly one thread). |
|||
msg145407 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-12 16:16 | |
>> Correct. I copied the algorithm from _io.FileIO, under the assumption >> that there was a reason for not using a simpler O(n log n) doubling >> strategy. Do you know of any reason for this? Or is it safe to ignore it? > > I don't know, but I'd say it's safe to ignore it. To elaborate: ISTM that it's actually a bug in FileIO. I can imagine where it's coming from (i.e. somebody feeling that overhead shouldn't grow unbounded), but I think that's ill-advised - *if* somebody really produces multi-gigabyte data (and some people eventually will), they still deserve good performance, and they will be able to afford the memory overhead (else they couldn't store the actual output, either). > Generally we use a less-than-doubling strategy, to conserve memory (see > e.g. bytearray.c). Most definitely. In case it isn't clear (but it probably is here): any constant factor > 1.0 will provide amortized linear complexity. |
|||
msg145409 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-10-12 16:21 | |
> To elaborate on the policy: giving bogus data in cases of race > conditions is ok; crashing the interpreter or corrupting memory > is not. If bogus data is given, it would be useful if the bogosity > can be specified (e.g. when multiple threads read from the same > POSIX file concurrently, they also get bogus data, but in a manner > where each input byte is given to exactly one thread). OK, that makes sense. My next patch will include per-compressor locks for LZMACompressor and LZMADecompressor. > To elaborate: ISTM that it's actually a bug in FileIO. I've filed issue 13159 proposing that FileIO be fixed. |
|||
msg145446 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-10-13 01:01 | |
Patch update time! Noteworthy changes: * Windows build support, thanks to Amaury * Thread safety for LZMACompressor and LZMADecompressor, by means of per-instance locks * O(n) buffer growth strategy I've tried running the tests on Windows, but I've been getting failures that don't look like they're related to the new code. I think they are due to issue 7443. I'll try and tweak my Windows setup a bit over the weekend to see if I can get things working smoothly. |
|||
msg145456 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2011-10-13 15:29 | |
I have imported xz-5.0.3 into the externals repository now. |
|||
msg145737 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-10-17 17:32 | |
After disabling a bunch of services and skipping the tests that are failing on the Windows buildbots, it seems that the failures I was getting last week have disappeared. It looks like they were due to the Windows search indexer preventing temp directories from being deleted... >_> > I have imported xz-5.0.3 into the externals repository now. I tried to test this out, but it looks like liblzma.a is missing from the SVN repo. There should be files with that name in the bin_i486 and bin_x86-64 directories. Apart from this, is there anything else that needs to be done before this change can be committed to the main repo? |
|||
msg145918 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-10-19 16:09 | |
> Apart from this, is there anything else that needs to be done before this > change can be committed to the main repo? Things look fine to me, apart from a couple of minor comments I've made on the review tool. |
|||
msg147769 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-11-16 18:02 | |
What is the status on this, Nadeem? It would be lovely to get the feature in the stdlib. |
|||
msg147770 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-11-16 18:43 | |
I just added the missing files liblzma.a to the "externals" repository. If someone can quickly check that it works on win32, I don't have anything else to add to this change. |
|||
msg147771 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-11-16 19:10 | |
Everything is ready to go, as far as I'm concerned. Once I've tested the Windows build using the binary from the externals repository, I'll be ready to commit it to the main repository. |
|||
msg147772 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-11-16 21:18 | |
I've tested the Windows build, and it works fine. If there are no objections, I'll commit on Saturday. Btw, what is the preferred channel for communicating with buildbot owners? I'll want to send out an email asking that all the *nix bots have liblzma-dev installed, so that they can actually run the tests... |
|||
msg147784 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-11-17 00:49 | |
I've tested under 64-bit Windows and it worked fine. |
|||
msg147788 - (view) | Author: STINNER Victor (vstinner) * | Date: 2011-11-17 01:48 | |
I wrote a short review on http://bugs.python.org/review/6715/show |
|||
msg148412 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-11-26 15:13 | |
Victor: Thanks for the review; I've replied to your comments and updated the patch. Let me know what you think of the changes. |
|||
msg148427 - (view) | Author: STINNER Victor (vstinner) * | Date: 2011-11-26 23:07 | |
The last patch (9276fc685c05.diff) looks good to me. Go ahead for the commit! |
|||
msg148622 - (view) | Author: Roundup Robot (python-dev) | Date: 2011-11-29 22:36 | |
New changeset 74d182cf0187 by Nadeem Vawda in branch 'default': Issue #6715: Add module for compression using the LZMA algorithm. http://hg.python.org/cpython/rev/74d182cf0187 |
|||
msg148624 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-11-29 23:08 | |
The patch has been committed. I'm leaving this issue open for a while longer, until more of the buildbots are building and testing the module. |
|||
msg148626 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2011-11-29 23:19 | |
Not meaning to sound petty, but wouldn't it be common etiquette to retain some original copyright notice from original code intact..? |
|||
msg148632 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-11-30 01:54 | |
> Not meaning to sound petty, but wouldn't it be common etiquette to > retain some original copyright notice from original code intact..? It seemed to me that Nadeem had rewritten everything from scratch. Is there any code of yours in the current module? |
|||
msg148647 - (view) | Author: Ned Deily (ned.deily) * | Date: 2011-11-30 10:11 | |
I've opened Issue13507 to track adding liblzma to the OS X installer builds. |
|||
msg148653 - (view) | Author: Per Øyvind Karlsen (proyvind) | Date: 2011-11-30 14:22 | |
Ah, I thought that he had reused most of the original C code in _lzmamodule.c not replaced by python code, but I see that not being the case now (only slight fragments;). Oh well, I thought that I'd still earned a note with some slight credit at least, but I guess I won't go postal or anything in lack of either.. :p |
|||
msg148654 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-11-30 14:46 | |
> Oh well, I thought that I'd still earned a note with some slight credit at least I completely agree. Sometimes people get credit for simple bug fixes (count me among them) so the author of the first working implementation deserves some recognition IMO. |
|||
msg148658 - (view) | Author: Éric Araujo (eric.araujo) * | Date: 2011-11-30 15:09 | |
Nadeem: Instead of duplicating the list of archiving/compression modules in each doc, what about only linking to the shutil doc for archives and the archiving.rst file? (I can make the patch, just wanted feedback first) |
|||
msg148659 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-11-30 15:11 | |
>> Not meaning to sound petty, but wouldn't it be common etiquette to >> retain some original copyright notice from original code intact..? > > It seemed to me that Nadeem had rewritten everything from scratch. Is > there any code of yours in the current module? That is correct. Based on my experience with the bz2 module, rewriting from scratch seemed to make more sense. >> Oh well, I thought that I'd still earned a note with some slight > credit at least > > I completely agree. Sometimes people get credit for simple bug fixes > (count me among them) so the author of the first working implementation > deserves some recognition IMO. Of course. How does this look? diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -502,6 +502,7 @@ Peter van Kampen Rafe Kaplan Jacob Kaplan-Moss +Per Øyvind Karlsen Lou Kates Hiroaki Kawai Sebastien Keim diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -400,6 +400,7 @@ ------- - Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm. + Thanks to Per Øyvind Karlsen for the initial implementation. - Issue #13487: Make inspect.getmodule robust against changes done to sys.modules while it is iterating over it. |
|||
msg148662 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-11-30 15:18 | |
> Instead of duplicating the list of archiving/compression modules in > each doc, what about only linking to the shutil doc for archives and > the archiving.rst file? Sure, go ahead. I actually hadn't realized that each section of the library docs had a "sub-index" page like that until now... |
|||
msg148668 - (view) | Author: Éric Araujo (eric.araujo) * | Date: 2011-11-30 15:57 | |
--- a/Misc/NEWS +++ b/Misc/NEWS @@ -400,6 +400,7 @@ ------- - Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm. + Thanks to Per Øyvind Karlsen for the initial implementation. The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message should be enough. Lately I’ve noticed some attributions in NEWS, but it’s usually not done (as redundant). |
|||
msg148670 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2011-11-30 16:08 | |
what about a mention in lzmamodule.c? |
|||
msg148671 - (view) | Author: Éric Araujo (eric.araujo) * | Date: 2011-11-30 16:11 | |
> I actually hadn't realized that each section of the library docs had a "sub-index" page like that That’s multiple-entry navigation :) You can jump to a module page, or use the index, or use the search, or browse and see the sub-indexes. As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the archiving subsection, there’s already a link to the subsection index, so I just removed the “See also zlib, etc.” lines (except for the link from zlib to gzip). I added a link from archiving.rst to shutil and more links and reST targets in shutil. |
|||
msg148675 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-11-30 17:04 | |
> The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message > should be enough. Lately I’ve noticed some attributions in NEWS, but > it’s usually not done (as redundant). I generally add attributions in NEWS since that's where most people learn about changes (for changes which aren't in whatsnew). |
|||
msg148695 - (view) | Author: Roundup Robot (python-dev) | Date: 2011-11-30 23:22 | |
New changeset 6cde416ef03b by Nadeem Vawda in branch 'default': Credit Per Øyvind Karlsen for the initial implementation of the lzma module (issue #6715). http://hg.python.org/cpython/rev/6cde416ef03b |
|||
msg148696 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-11-30 23:22 | |
> As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the > archiving subsection, there’s already a link to the subsection index, > so I just removed the “See also zlib, etc.” lines (except for the link > from zlib to gzip). I added a link from archiving.rst to shutil and > more links and reST targets in shutil. Looks good to me. > The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message > should be enough. Lately I’ve noticed some attributions in NEWS, but > it’s usually not done (as redundant). According to Doc/whatsnew/3.3.rst: "The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file." > what about a mention in lzmamodule.c? Done and committed. |
|||
msg149041 - (view) | Author: Éric Araujo (eric.araujo) * | Date: 2011-12-08 16:48 | |
I’ll commit my doc patch to all branches later. I checked the 4 red 3.3 builbots and they can’t build _lzma. |
|||
msg149047 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-12-08 17:40 | |
> I’ll commit my doc patch to all branches later. OK, great. > I checked the 4 red 3.3 builbots and they can’t build _lzma. For the record, the 3.x buildbots that currently aren't able to build the _lzma module are: * x86 Ubuntu Shared * sparc solaris10 gcc * AMD64 Snow Leopard 2 * x86 debian parallel * ARM Ubuntu * x86 Tiger * AMD64 OpenIndiana * x86 OpenIndiana * x86 FreeBSD 7.2 * x86 FreeBSD 6.4 Additionally, the AMD64 debian bigmem, PPC Tiger and PPC Leopard builders have been offline/broken since before changeset 74d182cf0187 was committed. |
|||
msg149048 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-12-08 17:58 | |
Le jeudi 08 décembre 2011 à 17:40 +0000, Nadeem Vawda a écrit : > Nadeem Vawda <nadeem.vawda@gmail.com> added the comment: > > > I’ll commit my doc patch to all branches later. > > OK, great. > > > I checked the 4 red 3.3 builbots and they can’t build _lzma. > > For the record, the 3.x buildbots that currently aren't able to build the > _lzma module are: Ok, I've e-mailed the owners. Will someone write an entry in the "what's new" file for 3.3? (Doc/whatsnew/3.3.rst) Regards Antoine. |
|||
msg149062 - (view) | Author: Roundup Robot (python-dev) | Date: 2011-12-08 23:34 | |
New changeset 66df5ace0eee by Nadeem Vawda in branch 'default': What's New in Python 3.3: Add entry for lzma module (issue #6715). http://hg.python.org/cpython/rev/66df5ace0eee |
|||
msg149063 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-12-08 23:58 | |
> Ok, I've e-mailed the owners. Thanks. I was just thinking I should send a reminder, in case they missed the note in my announcement on python-dev. > Will someone write an entry in the "what's new" file for 3.3? Done |
|||
msg149601 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-12-16 09:00 | |
Is there any reason why this issue is still open? |
|||
msg149605 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2011-12-16 09:37 | |
Yes, almost half of the buildbots still don't have the xz-utils headers installed, and thus are not building/testing the lzma module. I've kept the issue open as a reminder to myself to follow up on this. |
|||
msg152683 - (view) | Author: Roundup Robot (python-dev) | Date: 2012-02-05 12:50 | |
New changeset 11bd2d32b4e8 by Éric Araujo in branch '3.2': Improve interlinking of archiving/compression modules docs. http://hg.python.org/cpython/rev/11bd2d32b4e8 New changeset 1cb9b8126534 by Éric Araujo in branch 'default': Merge edits from 3.2 (#13716, #1040439, #2945, #13770, #6715) http://hg.python.org/cpython/rev/1cb9b8126534 |
|||
msg153349 - (view) | Author: Éric Araujo (eric.araujo) * | Date: 2012-02-14 16:18 | |
BTW, any plans on a PyPI backport for fun and profit? |
|||
msg153352 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2012-02-14 16:55 | |
> BTW, any plans on a PyPI backport for fun and profit? At present, no. I'll look into it later in the year, but at the moment I don't have the time to work on it - I suspect the parts written in C will require substantial changes to work under 2.x. |
|||
msg154298 - (view) | Author: Roundup Robot (python-dev) | Date: 2012-02-26 02:49 | |
New changeset 7c22281e967c by Éric Araujo in branch '2.7': Improve interlinking of archiving/compression modules docs. http://hg.python.org/cpython/rev/7c22281e967c |
|||
msg154985 - (view) | Author: STINNER Victor (vstinner) * | Date: 2012-03-05 22:43 | |
Why is this issue still open? |
|||
msg155010 - (view) | Author: Nadeem Vawda (nadeem.vawda) * | Date: 2012-03-06 11:30 | |
I had intended to wait until the code had been tested on all of the buildbots before closing it. However, it's taking a while to get xz-utils installed on the last few bots, so it doesn't make sense to keep the issue open for this. |
|||
msg174960 - (view) | Author: Peter (maubp) | Date: 2012-11-06 10:05 | |
Apologies for noise, but since a backport was discussed, I'm mentioning this here. I've started implementing a backport, currently working and tested on Mac OS X and Linux, back to Python 3.0 - supporting Python 2 would be nice but probably significantly more work (assistance welcome - please get in touch on github): https://github.com/peterjc/backports.lzma Assuming Nadeem has no objections, I intend to publish this on PyPI (I have tried to email directly but perhaps I'm using an old email address or he has been busy, another reason for commenting here). Thanks! |
|||
msg179095 - (view) | Author: Peter (maubp) | Date: 2013-01-05 00:13 | |
Apologies again for the noise, but I've just made the first public release of the lzma backport at http://pypi.python.org/pypi/backports.lzma/ with the repository as mentioned before at https://github.com/peterjc/backports.lzma I have tested this on Python 2.6, 2.7 and 3.0 through 3.3 under Linux and Mac OS. I've not tried building this on Windows yet, but it should be possible and I will be reviewing Amaury's notes on this thread. My thanks to Nadeem and Per for looking over this with helpful feedback, and agreeing to the use of the 3-clause BSD license for the backport. If anyone has any further questions about the backport, please get in touch via GitHub or email me. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:56:51 | admin | set | github: 50964 |
2013-01-05 00:13:47 | maubp | set | messages: + msg179095 |
2012-11-06 10:05:04 | maubp | set | nosy:
+ maubp messages: + msg174960 |
2012-03-06 11:30:58 | nadeem.vawda | set | status: open -> closed resolution: fixed messages: + msg155010 stage: patch review -> resolved |
2012-03-05 22:43:18 | vstinner | set | messages: + msg154985 |
2012-02-26 02:49:09 | python-dev | set | messages: + msg154298 |
2012-02-14 16:55:05 | nadeem.vawda | set | messages: + msg153352 |
2012-02-14 16:18:31 | eric.araujo | set | messages: + msg153349 |
2012-02-05 12:50:11 | python-dev | set | messages: + msg152683 |
2011-12-16 09:37:02 | nadeem.vawda | set | messages: + msg149605 |
2011-12-16 09:00:46 | pitrou | set | messages: + msg149601 |
2011-12-08 23:58:05 | nadeem.vawda | set | messages: + msg149063 |
2011-12-08 23:34:39 | python-dev | set | messages: + msg149062 |
2011-12-08 17:58:12 | pitrou | set | messages: + msg149048 |
2011-12-08 17:40:56 | nadeem.vawda | set | messages: + msg149047 |
2011-12-08 16:48:24 | eric.araujo | set | messages: + msg149041 |
2011-11-30 23:22:19 | nadeem.vawda | set | messages: + msg148696 |
2011-11-30 23:22:05 | python-dev | set | messages: + msg148695 |
2011-11-30 17:04:30 | pitrou | set | messages: + msg148675 |
2011-11-30 16:11:35 | eric.araujo | set | files:
+ archiving-modules-links.diff messages: + msg148671 |
2011-11-30 16:08:44 | amaury.forgeotdarc | set | messages: + msg148670 |
2011-11-30 15:57:23 | eric.araujo | set | messages: + msg148668 |
2011-11-30 15:18:53 | nadeem.vawda | set | messages: + msg148662 |
2011-11-30 15:11:45 | nadeem.vawda | set | messages: + msg148659 |
2011-11-30 15:09:20 | eric.araujo | set | messages: + msg148658 |
2011-11-30 14:46:06 | amaury.forgeotdarc | set | messages: + msg148654 |
2011-11-30 14:22:25 | proyvind | set | messages: + msg148653 |
2011-11-30 10:11:02 | ned.deily | set | nosy:
+ ned.deily messages: + msg148647 |
2011-11-30 01:54:32 | pitrou | set | messages: + msg148632 |
2011-11-29 23:19:02 | proyvind | set | messages: + msg148626 |
2011-11-29 23:08:19 | nadeem.vawda | set | messages: + msg148624 |
2011-11-29 22:36:59 | python-dev | set | nosy:
+ python-dev messages: + msg148622 |
2011-11-26 23:07:55 | vstinner | set | messages: + msg148427 |
2011-11-26 15:13:18 | nadeem.vawda | set | messages: + msg148412 |
2011-11-19 22:12:48 | nadeem.vawda | set | files: - 371a133b770a.diff |
2011-11-19 22:07:17 | nadeem.vawda | set | files: + 9276fc685c05.diff |
2011-11-17 01:48:23 | vstinner | set | messages: + msg147788 |
2011-11-17 00:49:52 | pitrou | set | messages: + msg147784 |
2011-11-16 21:18:21 | nadeem.vawda | set | messages: + msg147772 |
2011-11-16 19:10:00 | nadeem.vawda | set | messages: + msg147771 |
2011-11-16 18:43:52 | amaury.forgeotdarc | set | messages: + msg147770 |
2011-11-16 18:02:30 | pitrou | set | messages: + msg147769 |
2011-10-19 19:05:55 | nadeem.vawda | set | files: - bdf0afbbbd80.diff |
2011-10-19 19:05:09 | nadeem.vawda | set | files: + 371a133b770a.diff |
2011-10-19 16:09:19 | pitrou | set | messages: + msg145918 |
2011-10-17 17:32:07 | nadeem.vawda | set | messages: + msg145737 |
2011-10-13 15:29:15 | loewis | set | messages: + msg145456 |
2011-10-13 01:01:21 | nadeem.vawda | set | messages: + msg145446 |
2011-10-13 00:55:17 | nadeem.vawda | set | files: - 591277fe6b4a.diff |
2011-10-13 00:53:57 | nadeem.vawda | set | files: + bdf0afbbbd80.diff |
2011-10-12 16:21:03 | nadeem.vawda | set | messages: + msg145409 |
2011-10-12 16:16:51 | loewis | set | messages: + msg145407 |
2011-10-12 16:11:26 | loewis | set | messages: + msg145405 |
2011-10-11 11:39:02 | pitrou | set | messages: + msg145340 |
2011-10-11 10:03:32 | nadeem.vawda | set | messages: + msg145338 |
2011-10-11 00:42:13 | amaury.forgeotdarc | set | files:
+ lzma-win32.diff messages: + msg145317 |
2011-10-06 15:56:27 | amaury.forgeotdarc | set | messages: + msg145024 |
2011-10-06 15:51:49 | amaury.forgeotdarc | set | messages: + msg145023 |
2011-10-06 15:43:07 | nadeem.vawda | set | messages: + msg145022 |
2011-10-06 15:30:09 | amaury.forgeotdarc | set | messages: + msg145020 |
2011-10-06 14:35:26 | loewis | set | messages: + msg145016 |
2011-10-06 11:01:34 | nadeem.vawda | set | messages: + msg144992 |
2011-10-05 10:50:44 | jcea | set | messages: + msg144946 |
2011-10-05 10:09:21 | vstinner | set | nosy:
+ vstinner messages: + msg144942 |
2011-10-05 09:58:11 | loewis | set | messages: + msg144941 |
2011-10-05 09:31:15 | amaury.forgeotdarc | set | messages: + msg144940 |
2011-10-05 09:27:56 | pitrou | set | messages: + msg144939 |
2011-10-05 09:24:03 | jcea | set | messages: + msg144938 |
2011-10-04 18:08:55 | pitrou | set | messages: + msg144917 |
2011-10-04 18:08:09 | pitrou | set | messages: + msg144916 |
2011-10-04 18:05:56 | loewis | set | messages: + msg144915 |
2011-10-04 17:46:23 | pitrou | set | messages: + msg144912 |
2011-10-04 17:43:36 | loewis | set | messages: + msg144910 |
2011-10-04 17:07:59 | pitrou | set | messages: + msg144904 |
2011-10-04 17:00:00 | loewis | set | messages: + msg144902 |
2011-10-04 16:56:44 | loewis | set | messages: + msg144901 |
2011-10-04 16:13:23 | eric.araujo | set | files: - unnamed |
2011-10-04 08:12:36 | pitrou | set | messages: + msg144866 |
2011-10-02 22:55:41 | strombrg | set | files:
+ unnamed messages: + msg144790 |
2011-10-02 22:49:22 | nadeem.vawda | set | messages: + msg144788 |
2011-09-16 21:15:00 | amaury.forgeotdarc | set | messages: + msg144162 |
2011-09-16 10:27:51 | nadeem.vawda | set | messages: + msg144119 |
2011-09-16 10:10:42 | loewis | set | messages: + msg144118 |
2011-09-16 01:40:32 | jreese | set | nosy:
- jreese |
2011-09-15 22:03:25 | nadeem.vawda | set | messages: + msg144107 |
2011-09-15 21:54:24 | nadeem.vawda | set | files: - 3017ee1d548d.diff |
2011-09-15 21:53:14 | nadeem.vawda | set | files: + 591277fe6b4a.diff |
2011-09-15 16:14:16 | lars.gustaebel | set | messages: + msg144085 |
2011-09-15 08:15:20 | nadeem.vawda | set | messages:
+ msg144066 stage: needs patch -> patch review |
2011-09-15 08:13:31 | nadeem.vawda | set | files: - 95016f363e6c.diff |
2011-09-15 08:13:01 | nadeem.vawda | set | files: - fe2c9998f329.diff |
2011-09-15 08:12:33 | nadeem.vawda | set | files: - f3cf187208ea.diff |
2011-09-15 08:00:33 | nadeem.vawda | set | files: + 3017ee1d548d.diff |
2011-09-15 06:12:44 | nadeem.vawda | set | files: + 95016f363e6c.diff |
2011-09-11 21:01:34 | nadeem.vawda | set | messages: + msg143876 |
2011-09-11 20:57:00 | nadeem.vawda | set | files: + fe2c9998f329.diff |
2011-09-09 08:08:25 | nadeem.vawda | set | messages: + msg143757 |
2011-09-09 06:52:36 | Nam.Nguyen | set | nosy:
+ Nam.Nguyen messages: + msg143756 |
2011-08-29 18:53:08 | nadeem.vawda | set | messages: + msg143173 |
2011-08-29 18:49:14 | nadeem.vawda | set | files: + f3cf187208ea.diff |
2011-08-29 18:48:05 | nadeem.vawda | set | hgrepos: + hgrepo64 |
2011-08-29 12:21:53 | barry | set | nosy:
+ barry |
2011-06-23 12:23:56 | jeremybanks | set | nosy:
+ jeremybanks |
2011-05-16 11:25:55 | nadeem.vawda | set | assignee: nadeem.vawda messages: + msg136078 |
2011-05-16 07:13:27 | loewis | set | messages: + msg136070 |
2011-05-16 01:48:05 | strombrg | set | messages: + msg136059 |
2011-05-02 20:05:53 | tshepang | set | nosy:
+ tshepang |
2011-04-12 23:24:40 | vstinner | set | nosy:
- vstinner |
2011-04-08 01:36:16 | jcea | set | messages: + msg133285 |
2011-04-08 01:24:57 | jcea | set | nosy:
+ jcea |
2011-04-08 01:13:17 | jcea | set | nosy:
- jcea |
2011-04-08 01:12:26 | jcea | set | nosy:
+ Christophe Simonis |
2011-04-08 01:11:36 | jcea | set | nosy:
- Christophe Simonis |
2011-04-08 00:50:09 | strombrg | set | messages: + msg133282 |
2011-04-08 00:20:02 | nadeem.vawda | set | messages: + msg133280 |
2011-04-07 21:39:23 | loewis | set | nosy:
+ georg.brandl |
2011-04-07 21:39:06 | loewis | set | nosy:
- georg.brandl |
2011-04-07 21:38:48 | loewis | set | nosy:
+ Christophe Simonis |
2011-04-07 13:30:16 | jcea | set | messages: + msg133218 |
2011-04-07 13:10:14 | pitrou | set | messages: + msg133216 |
2011-04-07 13:02:04 | jcea | set | messages: + msg133214 |
2011-04-07 12:56:22 | jcea | set | nosy: + jcea, - Christophe Simonis |
2011-03-24 03:19:58 | asvetlov | set | nosy:
+ Christophe Simonis |
2011-03-24 03:18:53 | asvetlov | set | versions: + Python 3.3, - Python 3.2 |
2011-03-24 03:18:33 | asvetlov | set | nosy: + asvetlov, - Christophe Simonis |
2011-01-30 17:56:33 | devurandom | set | nosy:
- devurandom |
2011-01-30 07:49:34 | nadeem.vawda | set | nosy:
+ nadeem.vawda |
2011-01-29 07:52:05 | shirish | set | nosy:
+ shirish messages: + msg127384 |
2010-11-26 07:41:06 | loewis | set | messages: + msg122444 |
2010-11-26 07:32:54 | amaury.forgeotdarc | set | messages: + msg122443 |
2010-11-26 04:51:50 | strombrg | set | messages: + msg122435 |
2010-11-26 01:20:55 | eric.araujo | set | messages: + msg122426 |
2010-11-26 00:51:05 | strombrg | set | nosy:
+ strombrg messages: + msg122424 |
2010-10-31 21:46:25 | loewis | set | messages: + msg120091 |
2010-10-31 21:35:56 | loewis | set | messages: + msg120089 |
2010-10-31 21:24:25 | proyvind | set | messages: + msg120086 |
2010-10-31 21:15:28 | pitrou | set | messages: + msg120085 |
2010-10-31 20:57:33 | proyvind | set | messages: + msg120084 |
2010-10-31 16:43:24 | loewis | set | messages: + msg120068 |
2010-10-31 15:04:26 | proyvind | set | messages: + msg120050 |
2010-10-29 07:04:25 | georg.brandl | set | messages: + msg119866 |
2010-10-29 07:00:37 | pitrou | set | messages: + msg119865 |
2010-10-29 06:54:27 | georg.brandl | set | nosy:
+ georg.brandl messages: + msg119862 versions: + Python 3.2, - Python 3.3 |
2010-10-28 23:37:53 | pitrou | set | messages: + msg119826 |
2010-10-28 22:11:18 | proyvind | set | files:
+ teststring.xz messages: + msg119817 |
2010-10-28 22:10:48 | proyvind | set | files:
+ teststring.lzma messages: + msg119816 |
2010-10-28 22:09:58 | proyvind | set | files:
+ py3k-lzmamodule.patch keywords: + patch messages: + msg119815 |
2010-10-28 13:38:06 | loewis | set | messages: + msg119779 |
2010-10-28 13:25:20 | proyvind | set | messages: + msg119777 |
2010-10-27 23:40:41 | amaury.forgeotdarc | set | messages: + msg119743 |
2010-10-27 22:46:10 | pitrou | set | versions: + Python 3.3, - Python 3.2 |
2010-10-27 22:30:55 | proyvind | set | messages: + msg119742 |
2010-08-19 20:14:16 | jreese | set | nosy:
+ jreese |
2010-06-22 16:08:27 | rcoyner | set | nosy:
+ rcoyner |
2010-05-29 06:48:48 | proyvind | set | messages: + msg106710 |
2010-05-26 21:34:48 | pitrou | set | messages: + msg106581 |
2010-05-26 21:15:37 | doko | set | nosy:
+ doko |
2010-05-26 21:01:18 | loewis | set | messages: + msg106580 |
2010-05-26 20:47:25 | pitrou | set | messages: + msg106578 |
2010-05-26 19:54:04 | loewis | set | nosy:
+ loewis messages: + msg106572 |
2010-05-26 18:48:55 | proyvind | set | messages: + msg106567 |
2010-05-26 04:55:06 | lars.gustaebel | set | nosy:
+ lars.gustaebel |
2010-05-25 15:34:23 | pitrou | set | messages: + msg106441 |
2010-05-25 13:18:45 | ysj.ray | set | nosy:
+ ysj.ray |
2010-05-25 13:06:35 | proyvind | set | messages: + msg106433 |
2010-05-25 12:05:03 | amaury.forgeotdarc | set | messages: + msg106430 |
2010-05-25 11:20:28 | proyvind | set | nosy:
+ proyvind messages: + msg106427 |
2010-05-21 20:38:46 | Christophe Simonis | set | nosy:
+ Christophe Simonis |
2010-05-21 15:29:20 | vstinner | set | nosy:
+ vstinner |
2010-05-20 20:41:00 | skip.montanaro | set | nosy:
- skip.montanaro |
2010-05-08 14:30:20 | brian.curtin | set | versions: - Python 2.7 |
2010-05-08 14:18:54 | ockham-razor | set | nosy:
+ ockham-razor |
2010-04-09 13:07:44 | nikratio | set | nosy:
+ nikratio |
2010-04-09 08:06:02 | nicdumz | set | nosy:
+ nicdumz |
2010-03-30 14:41:19 | thedjatclubrock | set | nosy:
+ thedjatclubrock messages: + msg101941 |
2010-02-05 19:39:41 | eric.araujo | set | nosy:
+ eric.araujo |
2010-02-05 19:35:27 | pitrou | set | messages: + msg98899 |
2010-02-04 00:40:56 | pitrou | set | messages: + msg98806 |
2010-02-03 19:44:25 | arekm | set | nosy:
+ arekm messages: + msg98794 |
2010-02-03 06:34:59 | Garen | set | messages: + msg98776 |
2010-02-03 06:27:58 | Garen | set | nosy:
+ Garen messages: + msg98774 |
2010-01-27 15:58:57 | pitrou | link | issue5689 dependencies |
2010-01-27 15:58:41 | pitrou | set | priority: high dependencies: - Support xz compression in tarfile module |
2009-09-21 03:17:36 | leonov | set | nosy:
+ leonov |
2009-09-02 18:10:39 | devurandom | set | messages: + msg92174 |
2009-09-02 14:22:51 | amaury.forgeotdarc | set | messages: + msg92167 |
2009-09-02 11:02:12 | pitrou | set | nosy:
+ pitrou messages: + msg92163 |
2009-08-17 12:13:13 | devurandom | set | messages:
+ msg91661 title: xz compression support -> xz compressor support |
2009-08-17 11:51:44 | skip.montanaro | set | nosy:
+ skip.montanaro messages: + msg91660 |
2009-08-17 11:37:26 | amaury.forgeotdarc | set | versions:
- Python 2.6, Python 3.0, Python 3.1 nosy: + amaury.forgeotdarc messages: + msg91658 dependencies: + Support xz compression in tarfile module stage: needs patch |
2009-08-17 09:47:22 | devurandom | create |