extra_path is implicated in this failure and in general in any attempt to install to a path in PYTHONPATH that's not in site-packages. For example:

$ python -m pip install -t foo newrelic
Collecting newrelic
Installing collected packages: newrelic
Successfully installed newrelic
$ ls foo
newrelic-2.68.0.50		newrelic-2.68.0.50.dist-info	newrelic.pth
$ PYTHONPATH=foo python -c "import newrelic"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'newrelic'

As a result, extra_path (undocumented per bpo-901727) causes disruption in tools like rwt, and gives the packager the ability to break the convention for installing the package, relying on .pth files which only work in select locations.

I suggest we deprecate extra_path.

Nick, what's your instinct regarding this issue?

I think there are two time frames to look at here:

1. How do we make it easier for folks to work with existing packages like the newrelic one?

For that, it may make sense for at least setuptools to override extra_path when using installation targets other then sysconfig.get_path("purelib") and sysconfig.get_path("platlib") such that they get the same result as if "extra_path" wasn't there.

We know the setting isn't going to work, so ignoring it actually seems like it may be the more user friendly option.

2. How do we persuade publishers to stop using the "extra_path" feature in the first place?

I'm less sure of the benefits of that step, as I'm not sure why anyone would choose to use extra_path in the first place - the common "*.pth" file means you can still get conflicts, even with a versioned path name, and if you keep the dist-into directory, --single-version-externally-managed already gives you version info directly in the filesystem.

The use case does seem obscure enough that we could deprecate it as an undocumented feature that is incompatible with non-site-packages installs, and then see if anyone objects to the deprecation.

New changeset 94710cbcac47 by Jason R. Coombs in branch 'default':
Issue bpo-27919: Deprecate extra_path option in distutils.
https://hg.python.org/cpython/rev/94710cbcac47

Thanks, Nick. I've also observed that the package that was mentioned in the comments was "Numeric Python" and I confirmed that numpy is not currently using this option, so that's a positive indication that it's unneeded. I've also reached out to New Relic and invited them to comment here about the value they get (if any) from the option.

I've committed the deprecation warning as well. I'll plan to mirror this deprecation warning in Setuptools to get faster feedback. I'm open to reverting this change before the final release of Python 3.6 if it turns out to be a necessary feature.

As for your first suggestion - having Setuptools disable the behavior, I'm not sure Setuptools has the context to disable the behavior, because at the time pip is invoking the install command, Setuptools doesn't know whether pip is going to install the packages into platlib or elsewhere. At least, that's my presumption; I haven't dived into the code. I'm okay with not having a workaround if later versions of Setuptools and Python can disable the functionality without breaking anything but the most obscure packages.

Doing a quick search on Github, it seems there are 537 projects indicating extra_path in their setup.py. Many (most?) of those are using the feature to group packages into a namespace (Numeric, pygoogle, aeosa) in the filesystem. For each of these groups, it's probably worth contacting someone in the group to reference this ticket and get feedback.

Some of those projects just reference the value but set it to the default (disabled) or it's commented out.

At least one is using the feature for a load-time behavior injection (https://github.com/jwasinger/mailman_cas/blob/2ad0631001cb57366bb2a3b9d5f8eeddf5472c4e/mailman-2.1.12/misc/KoreanCodecs-2.0.5/setup.py). In that case, I don't think distutils should remain complicit in the hack, but the hack should be owned entirely by the project.

I've done more work breaking down the findings:

pygoogle (7): All forks of a project of the same name. Ignorable since the name matches the project name.

cancelbot (3): All forks of a project of the same name. Ignorable since the name matches the project name.

Numeric (8): All projects seem to be peripherally related to numpy, but probably had their setup.py copied from an earlier version of numpy.

Aeosa (58): The most recent activity is all in one repository, so I've filed mattneub/appscript#14 to request the change.

quasi (7): All forks of a project of the same name. Ignorable since the name matches the project name.

Motion and Graph (2): Two projects in https://github.com/scanlime/navi-misc. Ignorable since the name matches the project name.

Ohai (1): Made a comment here (https://github.com/bukzor/ohai).

NumPtr (5): The project makes the comment "I want to be able to remove them if I screw up this script; 'Sandboxing', if you will" and uses extra_path with the same name as the project; ignorable.

eispice (3): All forks of a project of the same name. Ignorable since the name matches the project name.

rtaudio (1): All forks of a project of the same name. Ignorable since the name matches the project name.

pyryutil (1): All forks of a project of the same name. Ignorable since the name matches the project name.

pbs (~5): Several projects are using the 'pbs' extra_path to group their packages (https://github.com/search?utf8=%E2%9C%93&q=extra_path+pbs+filename%3Asetup.py&type=Code&ref=searchresults), even going as far as creating a setup.py.in template with that setting. I've posted a ticket with what I believe to be the parent project.

korean (1): Uses the feature for load-time import. I've posted a ticket suggesting the removal of that.

kodos (6): All forks of a project of the same name. Ignorable since the name matches the project name.

pygraphics (4): All forks of a project of the same name. Ignorable since the name matches the project name.

pythonutils(1): All forks of a project of the same name. Ignorable since the name matches the project name.

Spike (2): All forks of a project of the same name. Ignorable since the name matches the project name.

timml (2): All forks of a project of the same name. Ignorable since the name matches the project name.

orange-bioinformatics (1): Uses extra_path to set a custom destination dir for the package. I've filed a ticket referencing this one.

pyNameList (1): All forks of a project of the same name. Ignorable since the name matches the project name.

FontTools (42): All forks of a project of the same name. Ignorable since the name matches the project name.

flopy (1): All forks of a project of the same name. Ignorable since the name matches the project name.

At this point, I've gone through 11 of the 54 pages of (most relevant) search results from Github. I suspect that's a good enough sample to determine if this feature does indeed need to be retained.

Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools