classification
Title: Explicitly note that meta_path is not empty
Type: behavior Stage:
Components: Documentation Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: brett.cannon, docs@python, eric.snow, ncoghlan, pitrou, xmorel
Priority: normal Keywords: easy

Created on 2017-06-01 08:59 by xmorel, last changed 2017-06-03 05:30 by ncoghlan.

Messages (11)
msg294919 - (view) Author: Xavier Morel (xmorel) * Date: 2017-06-01 08:59
Encountered this issue porting Python 2 code manipulating meta_path to Python 3.

In Python 2, meta_path is empty by default and its documentation specifically notes that:

> sys.meta_path is searched before any implicit default finders or sys.path.

As a result, sys.meta_path.append() works perfectly well and is overwhelmingly common[0].

In Python 3, this note was removed but the documentation does not specifically state default finders are present in meta_path, and the old code using sys.meta_path.append may now break as the default finders will silently take precedence on the custom ones if they believe they can do the job.

I'd like to add a warning to the Python 3 documentation (3.5+?) noting the presence of existing default loaders, and to the Python 2 documentation suggesting `sys.meta_path.insert(0, finder)` for future-proofing, would there be objections or issues?

[0] https://github.com/search?q=meta_path.append&type=Code&utf8=
msg294924 - (view) Author: Xavier Morel (xmorel) * Date: 2017-06-01 09:45
Addendum: the warning was present (in the documentation) until Python 3.5, even though Python 3.3 is apparently where the "default finders" were moved to meta_path:

    > python3.2 -c 'import sys;print(sys.meta_path)'
    []
> python3.3 -c 'import sys;print(sys.meta_path)'
    [<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib.PathFinder'>]
msg294927 - (view) Author: Xavier Morel (xmorel) * Date: 2017-06-01 09:52
And it turns out the change was noted in the Python 3.3 release notes[0] though not all the consequences were spelled out (and the meta_path and path_hooks documentations were not changed)

[0] https://docs.python.org/3/whatsnew/3.3.html#visible-changes
msg294928 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-01 09:56
A warning is probably too strong.  Also, it's easy to check sys.meta_path at the interpreter prompt, so I'm not sure it's worth mentioning at all.
msg294967 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-06-01 20:53
I'm fine with adding a note to the Python 2 docs, but putting it in Python 3 seems to be going in the wrong direction as having sys.meta_path not be empty is how Python will be going forward.
msg295007 - (view) Author: Xavier Morel (xmorel) * Date: 2017-06-02 11:17
> I'm fine with adding a note to the Python 2 docs, but putting it in Python 3 seems to be going in the wrong direction as having sys.meta_path not be empty is how Python will be going forward.

I don't think putting a note is any hint that the Python 3 behaviour (which I appreciate) would change, it would simply tell the reader that the list is not empty by default and they ought decide whether they want their finders to take precedence over (and insert at head) or be fallback to (and append) the builtin finders.

It could also link to the standard/builtin finders.
msg295008 - (view) Author: Xavier Morel (xmorel) * Date: 2017-06-02 11:21
> A warning is probably too strong.  Also, it's easy to check sys.meta_path at the interpreter prompt, so I'm not sure it's worth mentioning at all.

It's easy if you think of it and did not miss a small bit of the Python 3.3 release note indicating that the *documented guarantees* of Python 2.x, 3.0, 3.1 and 3.2 had been dropped, even as they were incorrectly still present in the official documentation itself.

I spent a few hours trying to understand why our import hooks did not work correctly anymore in 3.5, and I never even considered printing sys.meta_path in an open interpreter, I realised the behavioural change because I ended up printing meta_path after the addition of our custom hooks to make sure they were actually there.
msg295010 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-02 11:36
Le 02/06/2017 à 13:21, Xavier Morel a écrit :
> 
> I spent a few hours trying to understand why our import hooks did not work correctly anymore in 3.5, and I never even considered printing sys.meta_path in an open interpreter, I realised the behavioural change because I ended up printing meta_path after the addition of our custom hooks to make sure they were actually there.

Fair enough. So we can just add a sentence informing readers that
`meta_path`, by default, holds entries to handle the standard kinds of
modules (.py files, extension modules...).
msg295012 - (view) Author: Xavier Morel (xmorel) * Date: 2017-06-02 12:45
> Fair enough. So we can just add a sentence informing readers that
`meta_path`, by default, holds entries to handle the standard kinds of
modules (.py files, extension modules…).

Yeah that's basically what I meant, talking about a "warning" in python 3 may have been the wrong wording or given an incorrect impression, sorry about that.
msg295049 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-06-02 18:23
Yeah, there's actually a specific 'warning' directive which puts text in the docs in a red box to make it a full-blown warning which is what I thought you wanted, having it say "this is different than Python 2!". Having a sentence that just states how things are without referring to Python 2 is totally fine.
msg295065 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-06-03 05:30
s/Warn/Explicitly note/ in the issue title :)

I think this is actually both important enough and subtle enough to warrant a ".. note::" callout in the docs.

In Python 3, it would say to use `sys.meta_path.insert(0, finder)` if you want your finder to take precedence over the default search locations, and `sys.meta_path.append(finder)` if you only want it handle cases that aren't already handled by the default machine.

In Python 2, it would say that custom finders on `sys.meta_path` will always take precedence over the default machinery, and if you want to do otherwise, you'll need to use `importlib2` and install its loaders into `sys.meta_path` ahead of your own.
History
Date User Action Args
2017-06-03 05:30:44ncoghlansetmessages: + msg295065
title: Warn that meta_path is not empty -> Explicitly note that meta_path is not empty
2017-06-02 18:23:47brett.cannonsetmessages: + msg295049
2017-06-02 15:06:45eric.snowsetkeywords: + easy
2017-06-02 12:45:57xmorelsetmessages: + msg295012
2017-06-02 11:36:02pitrousetmessages: + msg295010
2017-06-02 11:21:44xmorelsetmessages: + msg295008
2017-06-02 11:17:36xmorelsetmessages: + msg295007
2017-06-01 20:53:06brett.cannonsetmessages: + msg294967
2017-06-01 09:56:33pitrousetnosy: + pitrou, eric.snow, brett.cannon, ncoghlan
messages: + msg294928
2017-06-01 09:52:51xmorelsetmessages: + msg294927
2017-06-01 09:45:34xmorelsetmessages: + msg294924
2017-06-01 08:59:10xmorelcreate