classification
Title: distutils.core.setup - data_files misbehaviour ?
Type: behavior Stage: needs patch
Components: Distutils Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Prashant.Kumar, Thorsten.Simons, dstufft, eric.araujo, fdrake, tarek
Priority: normal Keywords: easy, patch

Created on 2011-01-18 12:30 by Thorsten.Simons, last changed 2014-06-28 01:57 by BreamoreBoy.

Files
File name Uploaded Description Edit
issue.txt Thorsten.Simons, 2011-01-18 12:30 Descriptive File (setup.py and its output)
fix-sdist.py.diff Thorsten.Simons, 2011-01-27 14:34 diff of fixed sdist.py (python 3.1.3)
patch.diff Prashant.Kumar, 2011-04-19 20:00 test added in test_command_sdist.py + manifest.py modified
test_command_sdist.diff Prashant.Kumar, 2011-04-23 07:00 test_command_sdist.py modified for contents in MANIFEST
Repositories containing patches
Messages (15)
msg126459 - (view) Author: Thorsten Simons (Thorsten.Simons) Date: 2011-01-18 12:30
Hi !

when specifying additional files to be distributed by distutils.core.setup, documentation says that:

"Each file name in files is interpreted relative to the setup.py script at the top of the package source distribution. No directory information from files is used to determine the final location of the installed file; only the name of the file is used"

(http://docs.python.org/py3k/distutils/setupscript.html#distutils-additional-files)

Well, this seems to be incorrect - I tried to specify a file from a higher level directory, which get's copies to a target based on the whole path I specified, not just the name...

Pls. see attached file for details
msg126545 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-01-19 17:42
Thanks for the report.

“No directory information from files is used to determine the final location of the installed file; only the name of the file is used”

I understand this line to mean that only the basename of the file will be used in the target directory, IOW that ('config', ['cfg/data.cfg']) will create a file in config/data.cfg, not config/cfg/data.cfg.  I’m not 100% sure about my reading though, this line is confusing.

Paths starting with '..' are relative, even if they’re not under the top-level directory of the project (do the docs say they should be?), so the ability to give '../../spam/whatever' as data file looks okay.  However, it should not be installed to target_dir/../../spam, that’s definitely a bug (coming from os.path.join in cmd.Command.copy_file, I think).

If you want to write a test and fix, there are some guidelines at http://wiki.python.org/moin/Distutils/FixingBugs
msg126547 - (view) Author: Fred L. Drake, Jr. (fdrake) (Python committer) Date: 2011-01-19 17:49
On Wed, Jan 19, 2011 at 12:42 PM, Éric Araujo <report@bugs.python.org> wrote:
> I understand this line to mean that only the basename of the file will be used
> in the target directory, IOW that ('config', ['cfg/data.cfg']) will create a
> file in config/data.cfg, not config/cfg/data.cfg.
>  I’m not 100% sure about my reading though, this line is confusing.

This is my reading as well.
msg126549 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-01-19 18:03
See also #9261.
msg127191 - (view) Author: Thorsten Simons (Thorsten.Simons) Date: 2011-01-27 14:34
Hi Éric, All,

thank you for the directions you gave me!
It's all about a call of os.path.basename() missing in sdist.py. Pls. see attached diff.

Regards,
Thorsten
msg127435 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-01-29 16:19
Thanks for the patch.  Would you like to complete it?  It lacks tests, and I think the fix may be in the wrong place: You fixed sdist but not bdists.  I think the root of the problem is in the manifest (distutils2) / filelist (distutils1) module.

(We want to fix bugs in distutils2 first and then backport them to d1, as explained in the short guidelines I linked to in msg126545, which I advise you to read.)
msg133721 - (view) Author: Prashant Kumar (Prashant.Kumar) Date: 2011-04-14 10:53
I have added a test for the modification in msg127191. Though, inside zip-file we get files without its parent dir, nothing changes in manifest file(should it change?). Please have a look at the changeset https://bitbucket.org/pkumar/distutils2_bugs/changeset/7473385c23f2.
msg133834 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-04-15 15:07
Looks great, thanks.  You haven’t addressed this part of my previous message: “I think the fix may be in the wrong place: You fixed sdist but not bdists.  I think the root of the problem is in the manifest (distutils2) / filelist (distutils1) module.”

I don’t understand this comment: “Though, inside zip-file we get files without its parent dir, nothing changes in manifest file(should it change?).”

This bug should be fixed in packaging and distutils2, but I’m not sure about distutils: it’s an important change in behavior.  Tarek, Fred, what do you think?
msg133933 - (view) Author: Prashant Kumar (Prashant.Kumar) Date: 2011-04-17 18:03
I made changes in manifest.py which can be viewed here: https://bitbucket.org/pkumar/distutils2_bugs/changeset/111c1253ea7a

I'm not sure if I should modify test_command_sdist.py for the failing tests of manifest contents(since it is mentioned not to edit). 

msg133834:
>> I don’t understand this comment: “Though, inside zip-file we get files without its parent dir, nothing changes in manifest file(should it change?)

code from test_command_sdist.py:

        zip_file = zipfile.ZipFile(join(dist_folder, 'fake-1.0.zip'))
        try:
            content = zip_file.namelist()
        finally:
            zip_file.close()
after the change as mentioned in msg127191, the value of content changed, i.e. 'fake-1.0/some/other_file.txt' was changed to 'fake-1.0/other_file.txt'.But modifications were still required for the manifest. Above change should serve the purpose, I guess.
msg133981 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-04-18 16:45
The “remote hg repo” is supposed to be the URI of a clone (not of a changeset) of cpython (not distutils2), so the repos you added don’t work.  Can you remove them and add a patch file instead?  Thanks.

The first changeset looks good; I don’t understand the second one.

> I'm not sure if I should modify test_command_sdist.py for the failing
> tests of manifest contents
Hm.  It looks like the tests were in contradiction with the docs.  The safest way here is to fix the behavior in packaging/d2 and mention the bug in the distutils docs.

> (since it is mentioned not to edit).
It’s a comment aimed at software developers reading the generated MANIFEST file, not at distutils2 hackers working on the tests :)  See #8688.
msg134057 - (view) Author: Prashant Kumar (Prashant.Kumar) Date: 2011-04-19 14:17
If I'm not wrong, the contents in the manifest file should be:
 'data.cfg' (not cfg/data.cgg)
 I've added a patch which fixes the above issue.
msg134075 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-04-19 16:11
Can you remove all obsolete repositories and patches from this report and upload one complete patch?
msg134093 - (view) Author: Prashant Kumar (Prashant.Kumar) Date: 2011-04-19 20:00
Sure, patch.diff is a complete one.
msg134226 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-04-21 16:08
Patch looks good.  Detail: I’d find a call to basename more readable than i.split('/')[-1:][0].

Does this need a test in test_manifest too?  A doc update?
msg134295 - (view) Author: Prashant Kumar (Prashant.Kumar) Date: 2011-04-23 07:00
Certainly, a test is needed to check that ('config', ['cfg/data.cfg']) would create a file in config/data.cfg and not config/cfg/data.cfg.

Though, I've added a patch 'test_command_sdist.diff' for the changes made in 'patch.diff'.
History
Date User Action Args
2014-06-28 01:57:18BreamoreBoysetnosy: + dstufft

components: - Distutils2
versions: + Python 3.4, Python 3.5, - 3rd party, Python 3.1, Python 3.2
2011-04-23 07:00:41Prashant.Kumarsetfiles: + test_command_sdist.diff

messages: + msg134295
2011-04-21 16:08:48eric.araujosetmessages: + msg134226
2011-04-19 20:00:03Prashant.Kumarsetfiles: + patch.diff

messages: + msg134093
2011-04-19 19:49:52Prashant.Kumarsetfiles: - fix-manifest.diff
2011-04-19 19:49:39Prashant.Kumarsetfiles: - test_command_sdist.diff
2011-04-19 16:11:38eric.araujosetmessages: + msg134075
2011-04-19 14:17:29Prashant.Kumarsetfiles: + fix-manifest.diff

messages: + msg134057
2011-04-19 14:04:40Prashant.Kumarsetfiles: + test_command_sdist.diff
2011-04-18 16:45:55eric.araujosetmessages: + msg133981
2011-04-17 18:03:25Prashant.Kumarsethgrepos: + hgrepo20
messages: + msg133933
2011-04-15 15:07:47eric.araujosetmessages: + msg133834
2011-04-15 15:02:41eric.araujosethgrepos: + hgrepo19
2011-04-14 10:53:18Prashant.Kumarsethgrepos: + hgrepo18

messages: + msg133721
nosy: + Prashant.Kumar
2011-01-29 16:19:52eric.araujosetnosy: fdrake, tarek, eric.araujo, Thorsten.Simons
messages: + msg127435
stage: needs patch
2011-01-27 14:34:04Thorsten.Simonssetfiles: + fix-sdist.py.diff

messages: + msg127191
keywords: + patch
nosy: fdrake, tarek, eric.araujo, Thorsten.Simons
2011-01-19 18:03:24eric.araujosetnosy: fdrake, tarek, eric.araujo, Thorsten.Simons
messages: + msg126549
2011-01-19 17:49:27fdrakesetnosy: fdrake, tarek, eric.araujo, Thorsten.Simons
messages: + msg126547
2011-01-19 17:42:28eric.araujosetassignee: tarek -> eric.araujo

components: + Distutils2
title: distutils.core.setup - data_files missbehaviour ? -> distutils.core.setup - data_files misbehaviour ?
keywords: + easy
nosy: + fdrake
versions: + 3rd party, Python 2.7, Python 3.2
messages: + msg126545
2011-01-18 12:30:23Thorsten.Simonscreate