http://bugs.python.org/msg202659

Also msg202663 and msg202664

See http://www.python.org/dev/peps/pep-0451/#deprecations.

Keep in mind we figured out post-PEP acceptance that there are still a
couple of obscure use cases that need the old loader API, so we may want to
keep that around as a "power user" mode rather than deprecating it (either
forever or until a replacement is defined in 3.5)

It's definitely worth it to be more explicit.  (Brett had referenced msg202663.)  When I get some time I'll update this ticket with a list of the specific deprecation items.

I do not think we should keep the old APIs around forever, so either we don't care about the obscure use cases or figure out a way to support them.

I think we should definitely support them, I just haven't thought of a way
to do that yet which is cleaner than the status quo (it's only the loader
part of the API that isn't fully covered - the rest of the legacy APIs can
be deprecated happily). I may come up with an idea once we start work on
migrating extension module loading in 3.5.

To add another point about load_module(), I just had to rip out exec_module() for Python 3.4 as working around _imp.init_builtin() and _imp.load_dynamic() are going to need to be done hand-in-hand. So load_module() should probably get documented as deprecated and nothing more. Everything else can be hard deprecated.

Here's a patch for the deprecations (and API additions) in the importlib docs.

New changeset 2c27c0e5bc50 by Eric Snow in branch 'default':
Issue #19713: Update importlib docs for module spec changes, including deprecations.
http://hg.python.org/cpython/rev/2c27c0e5bc50

New changeset b78de8029606 by Eric Snow in branch 'default':
Issue #19713: Fix mistakes in the import page of language reference.
http://hg.python.org/cpython/rev/b78de8029606

The doc deprecations should now be complete.

P.S. changeset b78de8029606 should have referred to #19710.

Here's a patch that simply adds all the deprecation warnings.  It does not include any effort to silence the zillion warnings that happen when the test suite is run with -Wall.  I plan on addressing all those when I have some time.  That will be a matter of refactoring tests or stdlib modules to not use deprecated APIs.

In cases when the warnings happen due to BuiltinImporter.load_module() or ExtensionFileLoader.load_module(), I'll need to silence the warnings in the tests.

2 questions:

* Could I push this changeset (after a few fixes) before adding in all the refactors, doing those separately?
* Could we special-case BuiltinImporter and ExtensionFileLoader in _SpecMethods._load_backward_compatible() so that they don't cause a warning?  I know we've also talked about skipping the deprecation warning entirely for load_module()...

There are also several places in _bootstrap where a method (A) may raise a warning and a method it calls (B) may raise a similar warning.  Would there be any probably with silencing the deprecation warning coming out of the called method (B)?

Could you please use correct stacklevel, such that warnings will point to places where deprecated function used, not where they are defined?

I'd meant to do that.  I'll update the patch when I have a chance.

* Yes, you can commit with the warnings being triggered to get help with resolving them, just expect to potentially do a lot of work to make sure they don't leak out into 3.4 final.

* Just don't deprecate load_module() yet in code. If we can't even get rid of all our usage of the method then it isn't fair to expect users to do the same. Deprecating in the docs is fine, though.

Here's an updated patch that does most of the deprecations and adjustments/silencings to accommodate the deprecations.  The main things left to adjust are many importlib tests and pkgutil/test_pkgutil.

I'm still on the fence about deprecation warnings for module_repr().  We have to keep the methods around for backward-compatibility and module.__repr__ uses them if they exist.  So there really isn't a good place to put a deprecation warning for now.  Thoughts?

Please don't emit a deprecation warning for loaders that only implement load_module - there are still things load_module can do that create/exec can't, and it's still possible it will remain the long term API for those use cases.

Plus builtins and extensions are still loaded with load_module - we can't deprecate an API we're still using.

I'm actually still -0 on doing any programmatic deprecations in 3.4 at all. What maintenance burden are we eliminating for ourselves that justifies the pain we're inflicting on users? I want import system specialists to be *excited* about PEP 451, but if we deprecate everything and *force* them to migrate immediately (rather than giving them until 3.5 before they get warnings), they're going to hate it :(

I'm glad you spoke up, Nick.  Holding off on the DeprecationWarnings is fine with me.  It's not like we are going to drop support for the APIs before Python 4!  That said, DeprecationWarnings are disabled by default.  So how big a deal do you think it is to leave the warnings in (minus loader.load_module)?  I simply don't have a sense of how often people would be running with all warnings enabled (and not want that to let them know about these deprecations).

The latest patch already yanks most of the load_module() warnings.  It had slipped my mind in the first patch.  Of the warnings that are left, I'd still like to leave them in for:

* importlib.find_loader()
* NamespaceLoader (still not sure what to think about that class)
* loader.module_repr() (if I can figure out a good way to make it work) since it will end up being a 3.3-only feature
* importlib.util.set_loader()
* importlib.util.set_package()

That basically leaves removing warnings for finders (and maybe one or two others I missed):

* finder.find_module()
* finder.find_loader()

As I said, I'm fine with doing so (or even pulling all the warnings).  I simply thought of the warnings as a service to Python users to make them aware of the deprecations if they chose to check.  However, I haven't been doing this nearly as long as you. :)  So I'll readily concede that I may be missing something.

Test suites enable deprecation warnings by default, and many projects
have a "no warnings allowed" rule. By adding programmatic deprecation
warnings for the old APIs where there's no other way to it in a 3.3
compatible way, we make things more difficult for people that still
need to support 3.3. I'm OK with doing that when there's a concrete
payoff in significantly reducing our maintenance burden, eliminating
an attractive nuisance (I took that as far as removing
contextlib.nested entirely before coming up with ExitStack as a
replacement), or when there's a concrete benefit to the *user* in
migrating, but otherwise lean towards the "I want major Python
upgrades to be exciting, not annoying" school of thought most of the
time.

And yes, I'm biased through working on package stuff - all of that
needs to straddle 2.6+ and 3.2+ due to platform support requirements,
and I want people to be free to work on metadata 2.0 rather than
figuring out how to avoid new deprecation warnings :)

Your updated list of suggested near-term deprecations sounds a lot
more reasonable to me - that stuff is all 3.3 specific, and the
packaging ecosystem hasn't really adjusted to those yet anyway (given
the heavy 2.6+ and 3.2+ bias). It was mostly the method deprecations
that bothered me, since providing the "lowest common denominator" APIs
is the easiest way to implement cross version compatible finders and
loaders.

Sounds good.  Thanks for the explanation.  It sounds like you're effectively concerned just with finder.find_module() and loader.load_module() (which is fine with me).  Everything else (including some of the ABCs) is 3.3-only.  For the sake of simplicity I could also leave the warnings off finder.find_loader() as well (if you want).

Sorry for all the fuss.  I've never *really* deprecated anything before. :)  This has been really helpful.  I see the value in the warnings (when things are going to be removed later vs. "there's something better"), but also see your point about the pain warnings can cause (especially when we don't have concrete plans for removal).  What a weird balance.

p.s. If only I had a time machine to put PendingDeprecationWarnings on those 3.3-only APIs!  <wink>

Here's a patch that updates a couple files to not use find_module/load_module.  These are the only changes like this (of consequence) outside pydoc, pkgutil, and importlib, which are covered by other tickets.

New changeset a18c1a4cf30a by Eric Snow in branch 'default':
Issue #19713: Move away from using find_module/load_module.
http://hg.python.org/cpython/rev/a18c1a4cf30a

At this point the only places using find_module and load_module are pydoc, importlib, and some oddballs that aren't worth worrying about.  Issue #19703 covers the pydoc changes.

I should clarify.  That last commit was not the patch that adds the warnings.  I'm going to update that patch and attach it here when I get the chance.

About last commit: Is there a way to avoid using private objects when removing uses of load_module?

Arfrever: not at this point. We really should have an importlib.util.load_from_spec() that hides those internal details.

Larry - can we get away with adding that? It didn't become obvious it was a missing API until Eric started doing these conversions.

Hmm.  That's a good question.  There really isn't a simple, public-API replacement.  I've opened issue #20125 to discuss our options.  Feel free to offer any suggestions there.  Thanks for bringing this up.

Okay, there were a few lingering changes (mostly related to importlib.find_loader).  Here's a patch.

Here's a patch that should round out the changes for this ticket, adding the various deprecation warnings.  Most of the patch involves silencing warnings or cleaning up importlib tests relative to the deprecated APIs.

...and the patch.

I can accept the fourth patch in its current state.

Is that a rollup patch, including all the previous patches, or is it independent?

Is there a patch I can look at for this new API?

New changeset 37caaf21f827 by Eric Snow in branch 'default':
Issue 19713: Add PEP 451-related deprecations.
http://hg.python.org/cpython/rev/37caaf21f827

Larry:  There wasn't any API change for this issue.  It was a matter of fixing up places that were still using find_module/find_loader/load_module.

Also:

changeset:   88332:bfcbe41e892d4451b53bb5674fc4fa4ae791ec8c
user:        Eric Snow <ericsnowcurrently@gmail.com>
date:        Mon Jan 06 20:42:59 2014 -0700
summary:     Remove more usage of APIs deprecated by PEP 451.

> New changeset 37caaf21f827 by Eric Snow in branch 'default':
> Issue 19713: Add PEP 451-related deprecations.
> http://hg.python.org/cpython/rev/37caaf21f827
> ...
> -        spec.loader.load_module(spec.name)
> +        try:
> +            _warnings
> +        except NameError:
> +            # We must be importing builtins in setup().
> +            spec.loader.load_module(spec.name)
> +        else:
> +            spec.loader.load_module(spec.name)

Is this correct?

Good catch.  I've gone through looking for other such left-overs from earlier patches that included deprecation warnings for loader.load_module(), but this is the only one I found.  However, I did notice that _SpecMethods.from_module() does not get used anywhere.  For now I'm going to remove it.

New changeset c0b0e7aef360 by Eric Snow in branch 'default':
Issue 19713: Remove PEP 451-related code that should have been factored out.
http://hg.python.org/cpython/rev/c0b0e7aef360