classification
Title: Deprecate various things in importlib thanks to PEP 451
Type: Stage: resolved
Components: Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: 19703 19708 Superseder:
Assigned To: eric.snow Nosy List: Arfrever, brett.cannon, eric.snow, larry, ncoghlan, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-11-22 16:50 by brett.cannon, last changed 2014-01-08 06:33 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue19713-importlib-docs.diff eric.snow, 2013-12-11 06:35 review
issue19713-switch-to-new-api.diff eric.snow, 2014-01-03 08:17 review
issue19713-more-API-adjustments.diff eric.snow, 2014-01-05 07:24 review
issue19713-deprecation-warnings.diff eric.snow, 2014-01-05 07:38 review
Messages (38)
msg203807 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-11-22 16:50
http://bugs.python.org/msg202659
msg203808 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-11-22 16:50
Also msg202663 and msg202664
msg203824 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-11-22 17:53
See http://www.python.org/dev/peps/pep-0451/#deprecations.
msg203920 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-22 23:37
Keep in mind we figured out post-PEP acceptance that there are still a
couple of obscure use cases that need the old loader API, so we may want to
keep that around as a "power user" mode rather than deprecating it (either
forever or until a replacement is defined in 3.5)
msg203933 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-11-23 00:42
It's definitely worth it to be more explicit.  (Brett had referenced msg202663.)  When I get some time I'll update this ticket with a list of the specific deprecation items.
msg204044 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-11-23 15:59
I do not think we should keep the old APIs around forever, so either we don't care about the obscure use cases or figure out a way to support them.
msg204163 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 02:27
I think we should definitely support them, I just haven't thought of a way
to do that yet which is cleaner than the status quo (it's only the loader
part of the API that isn't fully covered - the rest of the legacy APIs can
be deprecated happily). I may come up with an idea once we start work on
migrating extension module loading in 3.5.
msg204736 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-11-29 16:06
To add another point about load_module(), I just had to rip out exec_module() for Python 3.4 as working around _imp.init_builtin() and _imp.load_dynamic() are going to need to be done hand-in-hand. So load_module() should probably get documented as deprecated and nothing more. Everything else can be hard deprecated.
msg205886 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-11 06:35
Here's a patch for the deprecations (and API additions) in the importlib docs.
msg206402 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-17 06:11
New changeset 2c27c0e5bc50 by Eric Snow in branch 'default':
Issue #19713: Update importlib docs for module spec changes, including deprecations.
http://hg.python.org/cpython/rev/2c27c0e5bc50

New changeset b78de8029606 by Eric Snow in branch 'default':
Issue #19713: Fix mistakes in the import page of language reference.
http://hg.python.org/cpython/rev/b78de8029606
msg206404 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-17 06:14
The doc deprecations should now be complete.
msg206405 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-17 06:15
P.S. changeset b78de8029606 should have referred to #19710.
msg206407 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-17 06:55
Here's a patch that simply adds all the deprecation warnings.  It does not include any effort to silence the zillion warnings that happen when the test suite is run with -Wall.  I plan on addressing all those when I have some time.  That will be a matter of refactoring tests or stdlib modules to not use deprecated APIs.

In cases when the warnings happen due to BuiltinImporter.load_module() or ExtensionFileLoader.load_module(), I'll need to silence the warnings in the tests.

2 questions:

* Could I push this changeset (after a few fixes) before adding in all the refactors, doing those separately?
* Could we special-case BuiltinImporter and ExtensionFileLoader in _SpecMethods._load_backward_compatible() so that they don't cause a warning?  I know we've also talked about skipping the deprecation warning entirely for load_module()...

There are also several places in _bootstrap where a method (A) may raise a warning and a method it calls (B) may raise a similar warning.  Would there be any probably with silencing the deprecation warning coming out of the called method (B)?
msg206410 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-17 07:16
Could you please use correct stacklevel, such that warnings will point to places where deprecated function used, not where they are defined?
msg206411 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-17 07:23
I'd meant to do that.  I'll update the patch when I have a chance.
msg206440 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-12-17 14:04
* Yes, you can commit with the warnings being triggered to get help with resolving them, just expect to potentially do a lot of work to make sure they don't leak out into 3.4 final.

* Just don't deprecate load_module() yet in code. If we can't even get rid of all our usage of the method then it isn't fair to expect users to do the same. Deprecating in the docs is fine, though.
msg206583 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-19 06:20
Here's an updated patch that does most of the deprecations and adjustments/silencings to accommodate the deprecations.  The main things left to adjust are many importlib tests and pkgutil/test_pkgutil.

I'm still on the fence about deprecation warnings for module_repr().  We have to keep the methods around for backward-compatibility and module.__repr__ uses them if they exist.  So there really isn't a good place to put a deprecation warning for now.  Thoughts?
msg206599 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-19 11:06
Please don't emit a deprecation warning for loaders that only implement load_module - there are still things load_module can do that create/exec can't, and it's still possible it will remain the long term API for those use cases.

Plus builtins and extensions are still loaded with load_module - we can't deprecate an API we're still using.

I'm actually still -0 on doing any programmatic deprecations in 3.4 at all. What maintenance burden are we eliminating for ourselves that justifies the pain we're inflicting on users? I want import system specialists to be *excited* about PEP 451, but if we deprecate everything and *force* them to migrate immediately (rather than giving them until 3.5 before they get warnings), they're going to hate it :(
msg206654 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-19 22:54
I'm glad you spoke up, Nick.  Holding off on the DeprecationWarnings is fine with me.  It's not like we are going to drop support for the APIs before Python 4!  That said, DeprecationWarnings are disabled by default.  So how big a deal do you think it is to leave the warnings in (minus loader.load_module)?  I simply don't have a sense of how often people would be running with all warnings enabled (and not want that to let them know about these deprecations).

The latest patch already yanks most of the load_module() warnings.  It had slipped my mind in the first patch.  Of the warnings that are left, I'd still like to leave them in for:

* importlib.find_loader()
* NamespaceLoader (still not sure what to think about that class)
* loader.module_repr() (if I can figure out a good way to make it work) since it will end up being a 3.3-only feature
* importlib.util.set_loader()
* importlib.util.set_package()

That basically leaves removing warnings for finders (and maybe one or two others I missed):

* finder.find_module()
* finder.find_loader()

As I said, I'm fine with doing so (or even pulling all the warnings).  I simply thought of the warnings as a service to Python users to make them aware of the deprecations if they chose to check.  However, I haven't been doing this nearly as long as you. :)  So I'll readily concede that I may be missing something.
msg206658 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-20 01:52
Test suites enable deprecation warnings by default, and many projects
have a "no warnings allowed" rule. By adding programmatic deprecation
warnings for the old APIs where there's no other way to it in a 3.3
compatible way, we make things more difficult for people that still
need to support 3.3. I'm OK with doing that when there's a concrete
payoff in significantly reducing our maintenance burden, eliminating
an attractive nuisance (I took that as far as removing
contextlib.nested entirely before coming up with ExitStack as a
replacement), or when there's a concrete benefit to the *user* in
migrating, but otherwise lean towards the "I want major Python
upgrades to be exciting, not annoying" school of thought most of the
time.

And yes, I'm biased through working on package stuff - all of that
needs to straddle 2.6+ and 3.2+ due to platform support requirements,
and I want people to be free to work on metadata 2.0 rather than
figuring out how to avoid new deprecation warnings :)

Your updated list of suggested near-term deprecations sounds a lot
more reasonable to me - that stuff is all 3.3 specific, and the
packaging ecosystem hasn't really adjusted to those yet anyway (given
the heavy 2.6+ and 3.2+ bias). It was mostly the method deprecations
that bothered me, since providing the "lowest common denominator" APIs
is the easiest way to implement cross version compatible finders and
loaders.
msg206660 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-20 02:23
Sounds good.  Thanks for the explanation.  It sounds like you're effectively concerned just with finder.find_module() and loader.load_module() (which is fine with me).  Everything else (including some of the ABCs) is 3.3-only.  For the sake of simplicity I could also leave the warnings off finder.find_loader() as well (if you want).

Sorry for all the fuss.  I've never *really* deprecated anything before. :)  This has been really helpful.  I see the value in the warnings (when things are going to be removed later vs. "there's something better"), but also see your point about the pain warnings can cause (especially when we don't have concrete plans for removal).  What a weird balance.

p.s. If only I had a time machine to put PendingDeprecationWarnings on those 3.3-only APIs!  <wink>
msg207214 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-03 08:17
Here's a patch that updates a couple files to not use find_module/load_module.  These are the only changes like this (of consequence) outside pydoc, pkgutil, and importlib, which are covered by other tickets.
msg207314 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-04 22:13
New changeset a18c1a4cf30a by Eric Snow in branch 'default':
Issue #19713: Move away from using find_module/load_module.
http://hg.python.org/cpython/rev/a18c1a4cf30a
msg207318 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-04 22:19
At this point the only places using find_module and load_module are pydoc, importlib, and some oddballs that aren't worth worrying about.  Issue #19703 covers the pydoc changes.
msg207322 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-04 22:57
I should clarify.  That last commit was not the patch that adds the warnings.  I'm going to update that patch and attach it here when I get the chance.
msg207324 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-01-05 01:27
About last commit: Is there a way to avoid using private objects when removing uses of load_module?
msg207325 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-01-05 02:16
Arfrever: not at this point. We really should have an importlib.util.load_from_spec() that hides those internal details.

Larry - can we get away with adding that? It didn't become obvious it was a missing API until Eric started doing these conversions.
msg207327 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 02:19
Hmm.  That's a good question.  There really isn't a simple, public-API replacement.  I've opened issue #20125 to discuss our options.  Feel free to offer any suggestions there.  Thanks for bringing this up.
msg207359 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 07:24
Okay, there were a few lingering changes (mostly related to importlib.find_loader).  Here's a patch.
msg207360 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 07:38
Here's a patch that should round out the changes for this ticket, adding the various deprecation warnings.  Most of the patch involves silencing warnings or cleaning up importlib tests relative to the deprecated APIs.
msg207361 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 07:38
...and the patch.
msg207367 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-01-05 08:23
I can accept the fourth patch in its current state.

Is that a rollup patch, including all the previous patches, or is it independent?

Is there a patch I can look at for this new API?
msg207501 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-07 03:51
New changeset 37caaf21f827 by Eric Snow in branch 'default':
Issue 19713: Add PEP 451-related deprecations.
http://hg.python.org/cpython/rev/37caaf21f827
msg207503 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-07 03:55
Larry:  There wasn't any API change for this issue.  It was a matter of fixing up places that were still using find_module/find_loader/load_module.
msg207504 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-07 03:57
Also:

changeset:   88332:bfcbe41e892d4451b53bb5674fc4fa4ae791ec8c
user:        Eric Snow <ericsnowcurrently@gmail.com>
date:        Mon Jan 06 20:42:59 2014 -0700
summary:     Remove more usage of APIs deprecated by PEP 451.
msg207567 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-01-07 17:27
> New changeset 37caaf21f827 by Eric Snow in branch 'default':
> Issue 19713: Add PEP 451-related deprecations.
> http://hg.python.org/cpython/rev/37caaf21f827
> ...
> -        spec.loader.load_module(spec.name)
> +        try:
> +            _warnings
> +        except NameError:
> +            # We must be importing builtins in setup().
> +            spec.loader.load_module(spec.name)
> +        else:
> +            spec.loader.load_module(spec.name)

Is this correct?
msg207674 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-08 06:24
Good catch.  I've gone through looking for other such left-overs from earlier patches that included deprecation warnings for loader.load_module(), but this is the only one I found.  However, I did notice that _SpecMethods.from_module() does not get used anywhere.  For now I'm going to remove it.
msg207675 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-08 06:33
New changeset c0b0e7aef360 by Eric Snow in branch 'default':
Issue 19713: Remove PEP 451-related code that should have been factored out.
http://hg.python.org/cpython/rev/c0b0e7aef360
History
Date User Action Args
2014-01-08 06:33:35python-devsetmessages: + msg207675
2014-01-08 06:24:00eric.snowsetmessages: + msg207674
2014-01-07 17:27:42Arfreversetmessages: + msg207567
2014-01-07 03:57:04eric.snowsetmessages: + msg207504
2014-01-07 03:55:08eric.snowsetstatus: open -> closed
resolution: fixed
messages: + msg207503

stage: test needed -> resolved
2014-01-07 03:51:07python-devsetmessages: + msg207501
2014-01-05 08:23:54larrysetmessages: + msg207367
2014-01-05 07:39:53eric.snowsetassignee: eric.snow
2014-01-05 07:39:18eric.snowsetfiles: - issue19713-deprecation-warnings.diff
2014-01-05 07:39:07eric.snowsetdependencies: + Check pkgutil for anything missing for PEP 451
2014-01-05 07:38:57eric.snowsetfiles: + issue19713-deprecation-warnings.diff

messages: + msg207361
2014-01-05 07:38:17eric.snowsetmessages: + msg207360
2014-01-05 07:24:05eric.snowsetfiles: + issue19713-more-API-adjustments.diff

messages: + msg207359
2014-01-05 02:19:25eric.snowsetmessages: + msg207327
2014-01-05 02:16:39ncoghlansetnosy: + larry
messages: + msg207325
2014-01-05 01:27:21Arfreversetmessages: + msg207324
2014-01-04 22:57:49eric.snowsetmessages: + msg207322
2014-01-04 22:19:38eric.snowsetdependencies: + Update pydoc to PEP 451
messages: + msg207318
2014-01-04 22:13:48python-devsetmessages: + msg207314
2014-01-03 08:17:10eric.snowsetfiles: + issue19713-switch-to-new-api.diff

messages: + msg207214
2013-12-20 02:23:31eric.snowsetmessages: + msg206660
2013-12-20 01:52:48ncoghlansetmessages: + msg206658
2013-12-19 22:54:12eric.snowsetmessages: + msg206654
2013-12-19 11:06:51ncoghlansetmessages: + msg206599
2013-12-19 06:20:57eric.snowsetfiles: + issue19713-deprecation-warnings.diff

messages: + msg206583
2013-12-19 06:13:24eric.snowsetfiles: - issue19713-deprecation-warnings.diff
2013-12-17 14:04:11brett.cannonsetmessages: + msg206440
2013-12-17 07:23:58eric.snowsetmessages: + msg206411
2013-12-17 07:16:01serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg206410
2013-12-17 06:55:17eric.snowsetfiles: + issue19713-deprecation-warnings.diff

messages: + msg206407
2013-12-17 06:15:03eric.snowsetmessages: + msg206405
2013-12-17 06:14:03eric.snowsetmessages: + msg206404
2013-12-17 06:11:56python-devsetnosy: + python-dev
messages: + msg206402
2013-12-11 06:35:57eric.snowsetfiles: + issue19713-importlib-docs.diff
keywords: + patch
messages: + msg205886
2013-11-29 16:06:07brett.cannonsetmessages: + msg204736
2013-11-24 02:27:26ncoghlansetmessages: + msg204163
2013-11-23 19:54:11Arfreversetnosy: + Arfrever
2013-11-23 15:59:08brett.cannonsetmessages: + msg204044
2013-11-23 00:42:50eric.snowsetmessages: + msg203933
2013-11-22 23:37:43ncoghlansetmessages: + msg203920
2013-11-22 17:53:37eric.snowsetmessages: + msg203824
2013-11-22 16:50:53brett.cannonsetmessages: + msg203808
2013-11-22 16:50:16brett.cannonlinkissue18864 dependencies
2013-11-22 16:50:06brett.cannoncreate