This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Permit to easily use distutils "--formats=tar,gztar,bztar" on all systems
Type: enhancement Stage: resolved
Components: Distutils Versions: Python 3.2
Status: closed Resolution: duplicate
Dependencies: Superseder: make distutils use the tarfile module
View: 6048
Assigned To: tarek Nosy List: akuchling, eric.araujo, giampaolo.rodola, lars.gustaebel, nnorwitz, pje, rpetrov, tarek, techtonik, till
Priority: high Keywords: patch

Created on 2008-01-21 21:28 by giampaolo.rodola, last changed 2022-04-11 14:56 by admin. This issue is now closed.

File name Uploaded Description Edit
file.diff giampaolo.rodola, 2008-01-21 21:28 giampaolo.rodola, 2008-01-22 14:52 giampaolo.rodola, 2008-01-22 16:39
patch1886.use.tarfile.module.diff techtonik, 2008-12-26 16:12 patch
Messages (13)
msg61451 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2008-01-21 21:28
Currently creating tar, gztar, bztar source distributions using the
"--formats=tar,gztar,bztar" distutils option requires external utilities
(tar and possibly one of gzip, bzip2, or compress) to be installed on
the system since distutils uses them.
This is not a problem on most UNIX systems which provide these utilities
by default.  On all other systems (e.g. Windows) it is required to
install such utilities separately and have the executables available in
a directory on your PATH in order to have them work.

I discussed this on distutils mailing list [1] and tried to fix this
The patch in attachment permit to use the distutils
"--formats=tar,gztar,bztar" option without need of having tar/gzip/bzip2
utilities installed by using the tarfile module instead of os.spawn().
It also fixes bug #1885 [2].

msg61453 - (view) Author: PJ Eby (pje) * (Python committer) Date: 2008-01-21 21:42
On systems where the gzip or bz2 modules aren't installed, this patch
can raise a tarfile.CompressionError, which should be caught and handled
correctly.  The import of tarfile should also be inside the relevant
function (as is done for make_zipfile) instead of at the top level.

Other than those two points, the patch looks great.
msg61508 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2008-01-22 14:52
Both things fixed. The new patch is in attachment.

Just one thing: I tried a diff between a tar.gz file generated by
tarfile module and another one generated by the UNIX tar utility and it
seems there are some differences.
I don't know if this could represent a problem for distributions
generated by distutils and eventually integration.
Maybe someone who knows the tarfile implementation could clarify this point.
msg61514 - (view) Author: Lars Gustäbel (lars.gustaebel) * (Python committer) Date: 2008-01-22 16:02
I just did some tests and could not find any major difference. Which are
the differences you found?

BTW, in your patch the "dist" directory is not automatically created.
msg61515 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2008-01-22 16:39
> I just did some tests and could not find any major difference. 
> Which are the differences you found?

diff utility doesn't provide any output since the compared files are
binary. As far as I can tell the two files have a different size
(tarfile generates a little bigger file).
Moreover, by using 7zip on Windows I notice that the archive generated
by tarfile module looks different. Once I open it I see the following

dist (directory) -> archive_name.tar -> archive_name (directory) ->
other subfiles

...while the archive generated with the UNIX tar utility looks like this:

archive_name.tar -> archive_name (directory) -> other subfiles

> BTW, in your patch the "dist" directory is not automatically created.

You're right. file in attachment fixes that.
msg61518 - (view) Author: Lars Gustäbel (lars.gustaebel) * (Python committer) Date: 2008-01-22 18:33
Hm, on my Linux box both files are more or less identical. Sorry, I
cannot reproduce your problem.
msg78300 - (view) Author: anatoly techtonik (techtonik) Date: 2008-12-26 16:12
I attach an improved patch1886.use.tarfile.module.diff that fixes all
bugs addressed in this issue.

It also removes second call to compression program leaving that
privilege to tar.

>  I notice that the archive generated by tarfile module looks
> different. Once I open it I see the following structure:
> dist (directory) -> archive_name.tar -> archive_name (directory)
> -> other subfiles

This bug is illustrated and patched (not released yet as somebody needs
to reopen it) in issue 1885. The extra path component is taken from the
name passed to GzipFile.
msg78301 - (view) Author: anatoly techtonik (techtonik) Date: 2008-12-26 16:15
Oh, my mistake - the bug with extra dir component in archive is in issue4750

Second shell call to compression program is removed only for the case
when tarfile or any of required compression modules are not available.
msg78302 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-26 16:30
about patch1886.use.tarfile.module.diff - the original tar don't support
-j flag.
msg78303 - (view) Author: anatoly techtonik (techtonik) Date: 2008-12-26 16:55
What are the systems where does this original tar still exist as a
default utility?

If there is no tarfile module on this systems and "tar" version is so
old then you need a more modern system to wrap your releases or more
modern "tar". The ratio of such systems to windows systems without tar
at all is less than 1%, so it doesn't worth the hassle.

But the true reason why I won't revert the part where command line "tar"
is used, to a double call scheme, is that when you do not pipe "tar"
with compression program and use them separately with relative path -
you will get the same weird header in your tar.gz  That's why it is not
uncommon on windows to meet tar.gz python packages with weird paths
inside. Luckily, windows users are infrequently install something from
sources and if they do - the .zip option is often preferred.
msg78355 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-27 12:45
Did I misunderstood something about -j -z -Z flags ? /source/xref/onnv/onnv-gate/usr/src/cmd/tar/tar.c
msg78435 - (view) Author: anatoly techtonik (techtonik) Date: 2008-12-28 22:32
Roumen, could you be more specific about what are you trying to say with
this 200kB piece of code? Was it the intention to post a link to man or
another piece of spec?
msg113376 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-08-09 04:21
distutils in 2.7 and 3.1 use tarfile.
Date User Action Args
2022-04-11 14:56:30adminsetgithub: 46194
2010-08-09 04:21:07eric.araujosetstatus: open -> closed
resolution: duplicate
messages: + msg113376

superseder: make distutils use the tarfile module
stage: resolved
2010-08-09 03:07:18terry.reedysetversions: + Python 3.2, - Python 3.1, Python 2.7
2010-07-27 15:13:04eric.araujosetassignee: tarek

nosy: + eric.araujo, tarek
2010-07-26 13:02:32akuchlingsetassignee: akuchling -> (no value)
2009-07-18 16:10:41tillsetnosy: + till
2009-04-05 18:18:52tareksetpriority: normal -> high
versions: + Python 3.1, - Python 2.6
2009-04-05 18:18:33tareklinkissue5685 superseder
2008-12-28 22:32:53techtoniksetmessages: + msg78435
2008-12-27 12:45:40rpetrovsetmessages: + msg78355
2008-12-26 16:55:27techtoniksetmessages: + msg78303
2008-12-26 16:30:25rpetrovsetnosy: + rpetrov
messages: + msg78302
2008-12-26 16:15:40techtoniksetmessages: + msg78301
2008-12-26 16:12:17techtoniksetfiles: + patch1886.use.tarfile.module.diff
nosy: + techtonik
messages: + msg78300
versions: + Python 2.7
2008-10-11 00:55:26akuchlingsetassignee: akuchling
nosy: + akuchling
2008-01-22 18:33:33lars.gustaebelsetmessages: + msg61518
2008-01-22 16:39:44giampaolo.rodolasetfiles: +
messages: + msg61515
2008-01-22 16:02:10lars.gustaebelsetmessages: + msg61514
2008-01-22 14:52:29giampaolo.rodolasetfiles: +
nosy: + nnorwitz, lars.gustaebel
messages: + msg61508
2008-01-22 00:59:29christian.heimeslinkissue1885 superseder
2008-01-22 00:57:58christian.heimessetpriority: normal
keywords: + patch
type: enhancement
versions: + Python 2.6, - Python 2.5
2008-01-21 21:42:36pjesetnosy: + pje
messages: + msg61453
2008-01-21 21:28:39giampaolo.rodolacreate