classification
Title: distutils docs: data_files always uses sys.prefix
Type: behavior Stage: resolved
Components: Distutils, Documentation Versions: Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: berker.peksag, docs@python, dstufft, eric.araujo, jdemeyer, pitrou
Priority: normal Keywords: patch

Created on 2015-11-10 06:06 by jdemeyer, last changed 2019-02-02 10:17 by pitrou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9767 merged jdemeyer, 2018-10-09 13:12
PR 11701 merged miss-islington, 2019-01-30 15:50
PR 11701 merged miss-islington, 2019-01-30 15:50
PR 11701 merged miss-islington, 2019-01-30 15:50
PR 11734 merged miss-islington, 2019-02-02 10:01
PR 11734 merged miss-islington, 2019-02-02 10:01
PR 11734 merged miss-islington, 2019-02-02 10:01
Messages (21)
msg254432 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2015-11-10 06:06
The documentation for distutils claims that sys.exec_prefix is used in certain cases to install data_files, but this is simply not true (maybe it was true in the past or this sentence was copy/pasted from somewhere else?)
msg327403 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2018-10-09 14:10
If you’re not sure about the reason for that sentence, I think you should not remove it from the docs but investigate more (look at the history, test the conditions (package with extension module), etc).
msg327407 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2018-10-09 14:23
> If you’re not sure about the reason for that sentence, I think you should not remove it from the docs

If the docs are wrong, their history doesn't matter that much: the docs should be fixed regardless.

> test the conditions (package with extension module)

I did that before posting this bug report. I also looked at the distutils sources. I couldn't find any evidence that data_files are ever installed in sys.exec_prefix.
msg327408 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2018-10-09 14:24
Just for fun, let's look at the history. That piece of documentation goes back to

commit 632bda3aa06879396561dde5ed3d93ee8fb8900c
Author: Fred Drake <fdrake@acm.org>
Date:   Fri Mar 8 22:02:06 2002 +0000

    Add more explanation of how data_files is used (esp. where the files end up
    in the installation and how that location is determined).

I haven't tried to compile that Python version from 2002 but the distutils sources from that time look quite close to what they are today (scary, no?). So my gut feeling is that those docs were wrong even at the time that they were written.
msg327412 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2018-10-09 14:40
There is also

commit fa2f4b6d8e297eda09d8ee52dc4a3600b7d458e7
Author: Greg Ward <gward@python.net>
Date:   Sat Jun 24 17:22:39 2000 +0000

    Changed the default installation directory for data files (used by
    the "install_data" command to the installation base, which is usually just
    sys.prefix.  (Any setup scripts out there that specify data files will have
    to set the installation directory, relative to the base, explicitly.)

with commit message stating that data_files are installed in sys.prefix.
msg327413 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2018-10-09 14:45
> If the docs are wrong, their history doesn't matter that much

What I was saying is that I am not sure that the docs are wrong.

Distutils is more touchy that the rest of the stdlib and I err on the side of caution.
msg327415 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2018-10-09 15:00
Well, I did try it on a minimal Python project. I also read the distutils sources and understood why it installs data_files in sys.prefix by default. So what more do you need to be convinced?
msg327416 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2018-10-09 15:02
Did you try with a minimal project containing a C extension?

Did you install in a system where sys.prefix != sys.exec_prefix?
msg327455 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2018-10-10 06:16
> Did you try with a minimal project containing a C extension?
> Did you install in a system where sys.prefix != sys.exec_prefix?

Yes to both questions.
msg334492 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-28 22:17
Though it's difficult to say for certain (distutils' option selection logic is obscure and difficult to follow), it seems that Jeroen's analysis is right.
msg334511 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-01-29 12:17
> it seems that Jeroen's analysis is right.

So would you be willing to merge the PR then?
msg334565 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-30 15:49
New changeset 598e15d4feaee3849a91d92c9ca51f17baafe19c by Antoine Pitrou (jdemeyer) in branch 'master':
bpo-25592: Improve documentation of distutils data_files (GH-9767)
https://github.com/python/cpython/commit/598e15d4feaee3849a91d92c9ca51f17baafe19c
msg334566 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2019-01-30 15:54
Thanks for the analysis and PR!  Sorry that the slow process here was frustrating.
msg334567 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-30 15:56
New changeset ebae1ce9c40e62ce52dc968f86ed11578d2fcdfd by Antoine Pitrou (Miss Islington (bot)) in branch '3.7':
bpo-25592: Improve documentation of distutils data_files (GH-9767) (GH-11701)
https://github.com/python/cpython/commit/ebae1ce9c40e62ce52dc968f86ed11578d2fcdfd
msg334568 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-30 15:57
Fixing this on 2.7 would require additional investigation (distutils might have diverged), so I'm closing now.
msg334607 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-01-31 06:14
> Fixing this on 2.7 would require additional investigation (distutils might have diverged)

Let's be honest, we are talking about distutils here. So it's way more likely that it didn't diverge and that the behavior is exactly the same on 2.7 and 3.8. So I would suggest to backport it to 2.7 also.
msg334619 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-31 10:08
Jeroen you're right.  Could you still give it a quick check?  Then I'll mark the PR for a 2.7 backport.
msg334698 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-02-01 16:29
> Could you still give it a quick check?

I did just that. For reference, these are the steps:
- Checkout the "2.7" branch of the cpython git repo
- ./configure --prefix=/tmp/prefix --exec-prefix=/tmp/eprefix && make && make install
- Install pip (https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py)
- /tmp/prefix/pip install --no-deps --verbose pari_jupyter  # This requires the PARI/GP library to be installed
- Note that all Jupyter stuff is installed in /tmp/prefix/share/jupyter while the directory /tmp/eprefix/share does not exist
msg334700 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-02-01 16:30
(note typo in the above: /tmp/prefix/pip should be /tmp/prefix/bin/pip)
msg334740 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-02-02 10:16
New changeset 40a101df8c64a1e55cca2780248d7a92bdcccd57 by Antoine Pitrou (Miss Islington (bot)) in branch '2.7':
bpo-25592: Improve documentation of distutils data_files (GH-9767) (GH-11734)
https://github.com/python/cpython/commit/40a101df8c64a1e55cca2780248d7a92bdcccd57
msg334741 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-02-02 10:17
Ok, this is now pushed to 2.7 as well. Thank you Jeroen!
History
Date User Action Args
2019-02-02 10:17:12pitrousetmessages: + msg334741
2019-02-02 10:16:29pitrousetstatus: open -> closed
stage: patch review -> resolved
versions: + Python 2.7
2019-02-02 10:16:14pitrousetmessages: + msg334740
2019-02-02 10:01:30miss-islingtonsetstage: resolved -> patch review
pull_requests: + pull_request11635
2019-02-02 10:01:17miss-islingtonsetstage: resolved -> resolved
pull_requests: + pull_request11634
2019-02-02 10:01:05miss-islingtonsetstage: resolved -> resolved
pull_requests: + pull_request11633
2019-02-01 16:30:45jdemeyersetmessages: + msg334700
2019-02-01 16:29:15jdemeyersetmessages: + msg334698
2019-01-31 10:08:13pitrousetmessages: + msg334619
2019-01-31 10:04:27pitrousetstatus: closed -> open
2019-01-31 06:14:15jdemeyersetmessages: + msg334607
2019-01-30 15:57:30pitrousetstatus: open -> closed
versions: - Python 2.7
messages: + msg334568

resolution: fixed
stage: patch review -> resolved
2019-01-30 15:56:53pitrousetmessages: + msg334567
2019-01-30 15:54:05eric.araujosetmessages: + msg334566
2019-01-30 15:50:26miss-islingtonsetpull_requests: + pull_request11553
2019-01-30 15:50:13miss-islingtonsetpull_requests: + pull_request11552
2019-01-30 15:50:02miss-islingtonsetpull_requests: + pull_request11551
2019-01-30 15:49:44pitrousetmessages: + msg334565
2019-01-29 12:17:45jdemeyersetmessages: + msg334511
2019-01-28 22:17:02pitrousetnosy: + pitrou
messages: + msg334492
2019-01-28 22:09:07pitrousetversions: + Python 3.7, Python 3.8, - Python 3.5, Python 3.6
2018-10-10 06:16:01jdemeyersetmessages: + msg327455
2018-10-09 15:02:05eric.araujosetmessages: + msg327416
2018-10-09 15:00:07jdemeyersetmessages: + msg327415
2018-10-09 14:45:21eric.araujosetmessages: + msg327413
2018-10-09 14:40:16jdemeyersetmessages: + msg327412
2018-10-09 14:24:44jdemeyersetmessages: + msg327408
2018-10-09 14:23:53jdemeyersetmessages: + msg327407
2018-10-09 14:10:19eric.araujosetmessages: + msg327403
2018-10-09 13:15:34jdemeyersetfiles: - data_files_doc.patch
2018-10-09 13:12:01jdemeyersetpull_requests: + pull_request9154
2016-05-28 23:58:02berker.peksagsetversions: + Python 3.5, Python 3.6
nosy: + berker.peksag, eric.araujo, dstufft

components: + Distutils
type: behavior
stage: patch review
2016-05-28 23:56:58berker.peksaglinkissue9363 superseder
2015-11-10 06:06:12jdemeyercreate