Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Permit to easily use distutils "--formats=tar,gztar,bztar" on all systems #46194

Closed
giampaolo opened this issue Jan 21, 2008 · 13 comments
Closed

Permit to easily use distutils "--formats=tar,gztar,bztar" on all systems #46194

giampaolo opened this issue Jan 21, 2008 · 13 comments
Assignees
@tarekziade
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@giampaolo
Copy link
Contributor

BPO 1886
Nosy @akuchling, @pjeby, @gustaebel, @giampaolo, @tarekziade, @merwok
Superseder
  • bpo-6048: make distutils use the tarfile module
    		•
    Files
  • file.diff
  • patch.zip
  • patch2.zip
  • patch1886.use.tarfile.module.diff: patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tarekziade'
closed_at = <Date 2010-08-09.04:21:07.567>
created_at = <Date 2008-01-21.21:28:39.040>
labels = ['type-feature', 'library']
title = 'Permit to easily use distutils "--formats=tar,gztar,bztar" on all systems'
updated_at = <Date 2010-08-09.04:21:07.524>
user = 'https://github.com/giampaolo'

    bugs.python.org fields:

    activity = <Date 2010-08-09.04:21:07.524>
actor = 'eric.araujo'
assignee = 'tarek'
closed = True
closed_date = <Date 2010-08-09.04:21:07.567>
closer = 'eric.araujo'
components = ['Distutils']
creation = <Date 2008-01-21.21:28:39.040>
creator = 'giampaolo.rodola'
dependencies = []
files = ['9253', '9262', '9264', '12456']
hgrepos = []
issue_num = 1886
keywords = ['patch']
message_count = 13.0
messages = ['61451', '61453', '61508', '61514', '61515', '61518', '78300', '78301', '78302', '78303', '78355', '78435', '113376']
nosy_count = 10.0
nosy_names = ['akuchling', 'nnorwitz', 'pje', 'lars.gustaebel', 'techtonik', 'giampaolo.rodola', 'tarek', 'eric.araujo', 'rpetrov', 'till']
pr_nums = []
priority = 'high'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '6048'
type = 'enhancement'
url = 'https://bugs.python.org/issue1886'
versions = ['Python 3.2']
    @giampaolo
    Copy link
    Contributor Author

    Currently creating tar, gztar, bztar source distributions using the
    "--formats=tar,gztar,bztar" distutils option requires external utilities
    (tar and possibly one of gzip, bzip2, or compress) to be installed on
    the system since distutils uses them.
    This is not a problem on most UNIX systems which provide these utilities
    by default. On all other systems (e.g. Windows) it is required to
    install such utilities separately and have the executables available in
    a directory on your PATH in order to have them work.

    I discussed this on distutils mailing list [1] and tried to fix this
    hassle.
    The patch in attachment permit to use the distutils
    "--formats=tar,gztar,bztar" option without need of having tar/gzip/bzip2
    utilities installed by using the tarfile module instead of os.spawn().
    It also fixes bug bpo-1885 [2].

    [1] http://mail.python.org/pipermail/distutils-sig/2008-January/008654.html
    [2] http://bugs.python.org/issue1885

    @giampaolo giampaolo added the stdlib Python modules in the Lib dir label Jan 21, 2008
    @pjeby
    Copy link
    Mannequin

    pjeby mannequin commented Jan 21, 2008

    On systems where the gzip or bz2 modules aren't installed, this patch
    can raise a tarfile.CompressionError, which should be caught and handled
    correctly. The import of tarfile should also be inside the relevant
    function (as is done for make_zipfile) instead of at the top level.

    Other than those two points, the patch looks great.

    @tiran tiran added the type-feature A feature request or enhancement label Jan 22, 2008
    @giampaolo
    Copy link
    Contributor Author

    Both things fixed. The new patch is in attachment.

    Just one thing: I tried a diff between a tar.gz file generated by
    tarfile module and another one generated by the UNIX tar utility and it
    seems there are some differences.
    I don't know if this could represent a problem for distributions
    generated by distutils and eventually easy_install.py integration.
    Maybe someone who knows the tarfile implementation could clarify this point.

    @gustaebel
    Copy link
    Mannequin

    gustaebel mannequin commented Jan 22, 2008

    I just did some tests and could not find any major difference. Which are
    the differences you found?

    BTW, in your patch the "dist" directory is not automatically created.

    @giampaolo
    Copy link
    Contributor Author

    I just did some tests and could not find any major difference.
    Which are the differences you found?

    diff utility doesn't provide any output since the compared files are
    binary. As far as I can tell the two files have a different size
    (tarfile generates a little bigger file).
    Moreover, by using 7zip on Windows I notice that the archive generated
    by tarfile module looks different. Once I open it I see the following
    structure:

    dist (directory) -> archive_name.tar -> archive_name (directory) ->
    other subfiles

    ...while the archive generated with the UNIX tar utility looks like this:

    archive_name.tar -> archive_name (directory) -> other subfiles

    BTW, in your patch the "dist" directory is not automatically created.

    You're right. patch2.zip file in attachment fixes that.

    @gustaebel
    Copy link
    Mannequin

    gustaebel mannequin commented Jan 22, 2008

    Hm, on my Linux box both files are more or less identical. Sorry, I
    cannot reproduce your problem.

    @akuchling akuchling self-assigned this Oct 11, 2008
    @techtonik
    Copy link
    Mannequin

    techtonik mannequin commented Dec 26, 2008

    I attach an improved patch1886.use.tarfile.module.diff that fixes all
    bugs addressed in this issue.

    It also removes second call to compression program leaving that
    privilege to tar.

    I notice that the archive generated by tarfile module looks
    different. Once I open it I see the following structure:

    dist (directory) -> archive_name.tar -> archive_name (directory)
    -> other subfiles

    This bug is illustrated and patched (not released yet as somebody needs
    to reopen it) in bpo-1885. The extra path component is taken from the
    name passed to GzipFile.

    @techtonik
    Copy link
    Mannequin

    techtonik mannequin commented Dec 26, 2008

    Oh, my mistake - the bug with extra dir component in archive is in bpo-4750

    Second shell call to compression program is removed only for the case
    when tarfile or any of required compression modules are not available.

    @rpetrov
    Copy link
    Mannequin

    rpetrov mannequin commented Dec 26, 2008

    about patch1886.use.tarfile.module.diff - the original tar don't support
    -j flag.

    @techtonik
    Copy link
    Mannequin

    techtonik mannequin commented Dec 26, 2008

    What are the systems where does this original tar still exist as a
    default utility?

    If there is no tarfile module on this systems and "tar" version is so
    old then you need a more modern system to wrap your releases or more
    modern "tar". The ratio of such systems to windows systems without tar
    at all is less than 1%, so it doesn't worth the hassle.

    But the true reason why I won't revert the part where command line "tar"
    is used, to a double call scheme, is that when you do not pipe "tar"
    with compression program and use them separately with relative path -
    you will get the same weird header in your tar.gz That's why it is not
    uncommon on windows to meet tar.gz python packages with weird paths
    inside. Luckily, windows users are infrequently install something from
    sources and if they do - the .zip option is often preferred.

    @rpetrov
    Copy link
    Mannequin

    rpetrov mannequin commented Dec 27, 2008

    Did I misunderstood something about -j -z -Z flags ?
    http://src.opensolaris.org /source/xref/onnv/onnv-gate/usr/src/cmd/tar/tar.c

    @techtonik
    Copy link
    Mannequin

    techtonik mannequin commented Dec 28, 2008

    Roumen, could you be more specific about what are you trying to say with
    this 200kB piece of code? Was it the intention to post a link to man or
    another piece of spec?

    @akuchling akuchling removed their assignment Jul 26, 2010
    @merwok
    Copy link
    Member

    merwok commented Aug 9, 2010

    distutils in 2.7 and 3.1 use tarfile.

    @merwok merwok closed this as completed Aug 9, 2010
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @tarekziade tarekziade

    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @akuchling @tiran @merwok @giampaolo