classification
Title: Deprecate and remove extra_path distribution kwarg
Type: behavior Stage:
Components: Distutils Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, dstufft, jason.coombs, merwok, ncoghlan, python-dev
Priority: normal Keywords:

Created on 2016-08-31 19:06 by jason.coombs, last changed 2016-09-04 13:14 by jason.coombs.

Messages (7)
msg274060 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-08-31 19:06
extra_path is implicated in [this failure](https://github.com/jaraco/rwt/issues/7) and in general in any attempt to install to a path in PYTHONPATH that's not in site-packages. For example:

$ python -m pip install -t foo newrelic
Collecting newrelic
Installing collected packages: newrelic
Successfully installed newrelic
$ ls foo
newrelic-2.68.0.50		newrelic-2.68.0.50.dist-info	newrelic.pth
$ PYTHONPATH=foo python -c "import newrelic"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'newrelic'


As a result, extra_path (undocumented per issue901727) causes disruption in tools like rwt, and gives the packager the ability to break the convention for installing the package, relying on .pth files which only work in select locations.

I suggest we deprecate extra_path.
msg274061 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-08-31 19:10
Nick, what's your instinct regarding this issue?
msg274117 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-01 10:50
I think there are two time frames to look at here:

1. How do we make it easier for folks to work with existing packages like the newrelic one?

For that, it may make sense for at least setuptools to override extra_path when using installation targets other then sysconfig.get_path("purelib") and sysconfig.get_path("platlib") such that they get the same result as if "extra_path" wasn't there.

We know the setting isn't going to work, so ignoring it actually seems like it may be the more user friendly option.

2. How do we persuade publishers to stop using the "extra_path" feature in the first place?

I'm less sure of the benefits of that step, as I'm not sure why anyone would choose to use extra_path in the first place - the common "*.pth" file means you can still get conflicts, even with a versioned path name, and if you keep the dist-into directory, --single-version-externally-managed already gives you version info directly in the filesystem.

The use case does seem obscure enough that we could deprecate it as an undocumented feature that is incompatible with non-site-packages installs, and then see if anyone objects to the deprecation.
msg274157 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-01 17:55
New changeset 94710cbcac47 by Jason R. Coombs in branch 'default':
Issue #27919: Deprecate extra_path option in distutils.
https://hg.python.org/cpython/rev/94710cbcac47
msg274159 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-09-01 18:03
Thanks, Nick. I've also observed that the package that was mentioned in the comments was "Numeric Python" and I confirmed that numpy is not currently using this option, so that's a positive indication that it's unneeded. I've also reached out to New Relic and invited them to comment here about the value they get (if any) from the option.

I've committed the deprecation warning as well. I'll plan to mirror this deprecation warning in Setuptools to get faster feedback. I'm open to reverting this change before the final release of Python 3.6 if it turns out to be a necessary feature.

As for your first suggestion - having Setuptools disable the behavior, I'm not sure Setuptools has the context to disable the behavior, because at the time pip is invoking the install command, Setuptools doesn't know whether pip is going to install the packages into platlib or elsewhere. At least, that's my presumption; I haven't dived into the code. I'm okay with not having a workaround if later versions of Setuptools and Python can disable the functionality without breaking anything but the most obscure packages.
msg274269 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-09-02 18:40
Doing a quick search on Github, it seems there are [537 projects](https://github.com/search?utf8=%E2%9C%93&q=extra_path+filename%3Asetup.py&type=Code&ref=searchresults) indicating `extra_path` in their setup.py. Many (most?) of those are using the feature to group packages into a namespace (Numeric, pygoogle, aeosa) in the filesystem. For each of these groups, it's probably worth contacting someone in the group to reference this ticket and get feedback.

Some of those projects just reference the value but set it to the default (disabled) or it's commented out.

At least one is using the feature for a load-time behavior injection (https://github.com/jwasinger/mailman_cas/blob/2ad0631001cb57366bb2a3b9d5f8eeddf5472c4e/mailman-2.1.12/misc/KoreanCodecs-2.0.5/setup.py). In that case, I don't think distutils should remain complicit in the hack, but the hack should be owned entirely by the project.
msg274360 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-09-04 13:14
I've done more work breaking down the findings:

pygoogle (7): All forks of a project of the same name. Ignorable since the name matches the project name.

cancelbot (3): All forks of a project of the same name. Ignorable since the name matches the project name.

Numeric (8): All projects seem to be peripherally related to numpy, but probably had their setup.py copied from an earlier version of numpy.

Aeosa (58): The most recent activity is all in one repository, so I've filed [mattneub/appscript#14](https://github.com/mattneub/appscript/issues/14) to request the change.

quasi (7): All forks of a project of the same name. Ignorable since the name matches the project name.

Motion and Graph (2): Two projects in https://github.com/scanlime/navi-misc. Ignorable since the name matches the project name.

Ohai (1): Made a comment here (https://github.com/bukzor/ohai).

NumPtr (5): The project makes the comment "I want to be able to remove them if I screw up this script; 'Sandboxing', if you will" and uses extra_path with the same name as the project; ignorable.

eispice (3): All forks of a project of the same name. Ignorable since the name matches the project name.

rtaudio (1): All forks of a project of the same name. Ignorable since the name matches the project name.

pyryutil (1): All forks of a project of the same name. Ignorable since the name matches the project name.

pbs (~5): Several projects are using the 'pbs' extra_path to group their packages (https://github.com/search?utf8=%E2%9C%93&q=extra_path+pbs+filename%3Asetup.py&type=Code&ref=searchresults), even going as far as creating a setup.py.in template with that setting. I've posted a ticket with what I believe to be the parent project.

korean (1): Uses the feature for load-time import. I've posted a ticket suggesting the removal of that.

kodos (6): All forks of a project of the same name. Ignorable since the name matches the project name.

pygraphics (4): All forks of a project of the same name. Ignorable since the name matches the project name.

pythonutils(1): All forks of a project of the same name. Ignorable since the name matches the project name.

Spike (2): All forks of a project of the same name. Ignorable since the name matches the project name.

timml (2): All forks of a project of the same name. Ignorable since the name matches the project name.

orange-bioinformatics (1): Uses extra_path to set a custom destination dir for the package. I've filed a ticket referencing this one.

pyNameList (1): All forks of a project of the same name. Ignorable since the name matches the project name.

FontTools (42): All forks of a project of the same name. Ignorable since the name matches the project name.

flopy (1): All forks of a project of the same name. Ignorable since the name matches the project name.

At this point, I've gone through 11 of the 54 pages of (most relevant) search results from Github. I suspect that's a good enough sample to determine if this feature does indeed need to be retained.
History
Date User Action Args
2016-09-04 13:14:07jason.coombssetmessages: + msg274360
2016-09-02 18:40:23jason.coombssetmessages: + msg274269
2016-09-01 18:03:54jason.coombssetmessages: + msg274159
2016-09-01 17:55:44python-devsetnosy: + python-dev
messages: + msg274157
2016-09-01 10:50:06ncoghlansetmessages: + msg274117
2016-08-31 19:33:47rhettingersetnosy: + brett.cannon
2016-08-31 19:10:44jason.coombssetnosy: + ncoghlan
messages: + msg274061
2016-08-31 19:06:14jason.coombscreate