classification
Title: Deprecate imp
Type: enhancement Stage: test needed
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: 14797 17907 18192 18193 18194 Superseder:
Assigned To: brett.cannon Nosy List: Arfrever, benjamin.peterson, brett.cannon, eric.araujo, ezio.melotti, haypo, python-dev, r.david.murray
Priority: normal Keywords:

Created on 2013-02-10 20:08 by brett.cannon, last changed 2013-06-19 00:52 by brett.cannon. This issue is now closed.

Messages (38)
msg181849 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-02-10 20:08
I need a make a decision as to what (if anything) belongs in imp and then document what stays and deprecate everything else.

Everything in imp falls into one of the following categories:

* From importlib
  - get_magic()
  - source_from_cache()
  - cache_from_source()
* From sys
  - get_tag()
* Platform-implemented
  - lock_held()
  - acquire_lock()
  - release_lock()
  - Undocumented stuff related to builtins, frozen, and load_dynamic()
* Helper
  - reload()

The question is what to keep/expose in imp and what to deprecate. Basically I need to figure out what imp's role is supposed to be in the face of importlib. My gut says either expose platform-dependent stuff and reload() and move the rest to importlib/deprecate, or to completely do away with the module and force people to use the APIs in importlib for consistency (and to stop people from mucking around with import from outside of importlib).
msg182152 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-02-15 17:32
I’ve always perceived imp as an implementation detail of the import system that people were sometimes forced to use to get to some things.  Now that we have importlib, deprecating (with a nice long compatibility-with-warnings period) sounds good.

Would checking with the other VMs be a good idea?
msg182155 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-02-15 17:54
Checking with the other VMs wouldn't benefit anyone. They still have to implement whatever is necessary from _imp in order for importlib to work. Otherwise it's just another stdlib module.
msg182165 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-02-15 20:15
> Basically I need to figure out what imp's role is supposed to be in the face of importlib.

If it's not even clear for you, even if you eventually figure that out, having two import-related modules is likely to be confusing for users.
OTOH deprecating/removing imp would affect the somewhat popular imp.reload(), that has already been moved for Python 3, and will now be moved again (this also means that the -3 warnings on 2.7 will no longer be valid).
msg182168 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-02-15 20:35
I would probably document the deprecation but otherwise leave it alone at the module level. Considering I only documented imp.find_module() and .load_module() as documented thanks to their funky interfaces making nearly impossible to do a direct translation I don't see how I can justify raising a DeprecationWarning for the whole thing.

But it's gone in Python 4. =)
msg182173 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-02-15 20:57
If you move them to importlib and just leave an alias in imp, you should still add a DeprecationWarning IMHO, even if you plan to remove them in Python 4.
msg187251 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-18 13:56
The documentation deprecation has been added as part of issue 17135.

Leaving this open to address Ezio's last comment.
msg187305 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-04-18 22:42
If the imp module is marked as deprecated, it would help to explain how to port code from imp to importlib. As it was done for os.spawn(), platform.popen(), etc.

I'm using imp.load_source() in my setup.py to load a version.py file which only contains constants, to be able to build my module even if executing the module fails (ex: because of a runtime dependency). I don't know how to change my code to use imp.load_source().
msg187306 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-04-18 22:43
> I don't know how to change my code to use imp.load_source().

Woops, I mean "to use importlib".
msg187361 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-04-19 14:08
The deprecation warnings for the explicitly deprecated functions already have the instructions on how to port. A note could probably be added to turn on deprecation warnings if one needs help with moving code over.

But the deprecation isn't even officially full-on yet. While it's documented as such, not all functionality has moved over to importlib, only bits and pieces.
msg187364 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-19 14:40
Should we back out that module level deprecation notice for the moment, then?
msg187366 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-04-19 15:35
Nah, consider it motivation for me to get it done in Python 3.4 else I have to back it out before release.
msg188387 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-05-04 18:23
Just to keep on top of what is left to do for this to happen:

* get_magic(): expose in importlib
* find_module()/load_module(): eliminate all uses in the stdlib and fully deprecate
* new_module(): use types.ModuleType() instead
* reload(): relocate
* cache_from_source(): relocate
* source_from_cache(): relocate
* get_tag(): deprecate
* lock_held()/acquire_lock()/release_lock(): deprecate
* NullImporter: deprecate

Everything else can stay undocumented as-is (deprecated or simply exposed through _imp which can stay undocumented).

I plan to make individual issues for these things as I solidify the decision of how to handle them.
msg188390 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-05-04 18:38
While thinking about asking on G+ for feedback, I realized that if I get this to the point of a module-level warning, it won't be a big deal. It would be just like warnings you deal with when supporting old versions of Python along with newer versions where things are now deprecated. That way  users of imp could just do::

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    import imp

and be done with it. This means still striving to deprecate every function in imp so that it can all be replaced with a single module-level deprecation is a reasonable goal to have.
msg188391 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-05-04 18:43
This might be OK, however you won't be able to point to the alternatives in the deprecation message (e.g. "new_module() is deprecated, use types.ModuleType() instead").
This can still be done in the docs though.
msg188392 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-05-04 18:46
Right, the docs would say "the imp modules is deprecated; please see the module documentation for alternatives" or something.
msg190984 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-06-11 20:47
There should now be issues for each of the required relocations to make imp redundant.
msg191187 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-06-15 02:42
Everything that is going to be kept around has been moved. That just leaves:

1. Removing all uses of imp from the stdlib
2. Document as deprecated the last few things in imp which are simply being removed (including adding a note in the docstrings)
3. Remove all function/class-level deprecation warnings and add a module-level pending deprecation one
4. Run the test suite with -W always to make sure I didn't miss anything
msg191188 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-06-15 02:44
And in case anyone is interested, it could be as many as 98 files to go through. =)
msg191189 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-06-15 02:47
Sorry, bad regex: it's more like 60 files.
msg191190 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 02:49
New changeset 3d3b9d456eb8 by Brett Cannon in branch 'default':
Issue #17177: Update the programming FAQ to use importlib
http://hg.python.org/cpython/rev/3d3b9d456eb8
msg191191 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-06-15 02:52
To have it on record, the command I am using to find uses of imp is::

 ack "import imp(,|\n)|from imp import" -l
msg191192 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-06-15 02:53
ack "import (\w+, *)*imp(,|\n)|from imp import" -l
msg191193 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 03:04
New changeset cc27d50bd91a by Brett Cannon in branch 'default':
Issue #17177: stop using imp for compileall.
http://hg.python.org/cpython/rev/cc27d50bd91a
msg191215 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 17:00
New changeset 30aa032c4bd0 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in distutils
http://hg.python.org/cpython/rev/30aa032c4bd0
msg191216 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 17:23
New changeset 0b96a16656b7 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in multiprocessing
http://hg.python.org/cpython/rev/0b96a16656b7
msg191218 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 18:25
New changeset 91467f342977 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp with py_compile
http://hg.python.org/cpython/rev/91467f342977

New changeset 81cf3d6e6756 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in pydoc
http://hg.python.org/cpython/rev/81cf3d6e6756
msg191220 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 18:27
New changeset 8fff5554125d by Brett Cannon in branch 'default':
Issue #17177: switch from imp.new_module to types.ModuleType for runpy
http://hg.python.org/cpython/rev/8fff5554125d
msg191221 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 18:32
New changeset 1e141c2cc0d7 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in sysconfig
http://hg.python.org/cpython/rev/1e141c2cc0d7
msg191235 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 21:11
New changeset bc18b5d4920e by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in a bunch of tests
http://hg.python.org/cpython/rev/bc18b5d4920e
msg191237 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 21:33
New changeset f96eb1dc335f by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in zipfile
http://hg.python.org/cpython/rev/f96eb1dc335f

New changeset b2b2bdc4cf6f by Brett Cannon in branch 'default':
Issue # 17177: Stop using imp in turtledemo
http://hg.python.org/cpython/rev/b2b2bdc4cf6f
msg191238 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 21:53
New changeset 4a1161eaed99 by Brett Cannon in branch 'default':
Issue # 17177: Stop using imp in setup.py
http://hg.python.org/cpython/rev/4a1161eaed99
msg191240 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 22:10
New changeset ca3bdac1f88a by Brett Cannon in branch 'default':
Issue #17177: update checkpyc to stop using imp
http://hg.python.org/cpython/rev/ca3bdac1f88a
msg191243 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-15 22:39
New changeset 5e8b377942f7 by Brett Cannon in branch 'default':
Issue #17177: stop using imp in test_importlib
http://hg.python.org/cpython/rev/5e8b377942f7
msg191247 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-06-15 23:13
At this point I have updated all the code that does not directly exposes imp somehow or in Tools.

At this point the only thing I need to decide is whether I want to continue to expose the lock stuff from imp or leave it private. After that the pending deprecation is ready to go.
msg191274 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-16 17:14
New changeset 1b8f08c4efd5 by Brett Cannon in branch 'default':
Issue #17177: The imp module is pending deprecation.
http://hg.python.org/cpython/rev/1b8f08c4efd5
msg191422 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-06-18 18:00
Going to temporarily re-open to remind me to add better deprecation docs for imp.load_module() to point to specific loader in importlib.machinery.

Should also add example usage as well.
msg191442 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-19 00:51
New changeset ded443c603f0 by Brett Cannon in branch 'default':
Issue #17177: Clarify some deprecations
http://hg.python.org/cpython/rev/ded443c603f0
History
Date User Action Args
2013-06-19 00:52:10brett.cannonsetstatus: open -> closed
2013-06-19 00:51:52python-devsetmessages: + msg191442
2013-06-18 18:00:31brett.cannonsetstatus: closed -> open

messages: + msg191422
2013-06-16 17:16:28brett.cannonsetstatus: open -> closed
resolution: fixed
2013-06-16 17:14:14python-devsetmessages: + msg191274
2013-06-15 23:13:59brett.cannonsetmessages: + msg191247
2013-06-15 22:39:29python-devsetmessages: + msg191243
2013-06-15 22:10:25python-devsetmessages: + msg191240
2013-06-15 21:53:07python-devsetmessages: + msg191238
2013-06-15 21:33:37python-devsetmessages: + msg191237
2013-06-15 21:11:34python-devsetmessages: + msg191235
2013-06-15 18:32:27python-devsetmessages: + msg191221
2013-06-15 18:27:29python-devsetmessages: + msg191220
2013-06-15 18:25:12python-devsetmessages: + msg191218
2013-06-15 17:23:08python-devsetmessages: + msg191216
2013-06-15 17:00:01python-devsetmessages: + msg191215
2013-06-15 03:04:11python-devsetmessages: + msg191193
2013-06-15 02:53:42brett.cannonsetmessages: + msg191192
2013-06-15 02:52:07brett.cannonsetmessages: + msg191191
2013-06-15 02:49:09python-devsetnosy: + python-dev
messages: + msg191190
2013-06-15 02:47:07brett.cannonsetmessages: + msg191189
2013-06-15 02:44:23brett.cannonsetmessages: + msg191188
2013-06-15 02:42:24brett.cannonsetmessages: + msg191187
2013-06-14 19:06:17brett.cannonsettitle: Document/deprecate imp -> Deprecate imp
2013-06-11 20:47:48brett.cannonsetdependencies: + Move imp.source_from_cache/cache_from_source to importlib
messages: + msg190984
2013-06-11 20:45:25brett.cannonsetdependencies: + Move imp.reload() to importlib
2013-06-11 20:44:21brett.cannonsetdependencies: + Move imp.get_magic() to importlib
2013-06-08 22:35:29brett.cannonlinkissue18034 dependencies
2013-05-04 23:49:29Arfreversetnosy: + Arfrever
2013-05-04 18:46:23brett.cannonsetmessages: + msg188392
2013-05-04 18:43:34ezio.melottisetmessages: + msg188391
2013-05-04 18:38:33brett.cannonsetmessages: + msg188390
2013-05-04 18:25:28brett.cannonsetdependencies: + Deprecate imp.new_module() in favour of types.ModuleType
2013-05-04 18:23:44brett.cannonsetnosy: + benjamin.peterson
messages: + msg188387
2013-04-19 15:35:27brett.cannonsetmessages: + msg187366
2013-04-19 14:40:55r.david.murraysetmessages: + msg187364
2013-04-19 14:08:40brett.cannonsetmessages: + msg187361
2013-04-18 22:43:17hayposetmessages: + msg187306
2013-04-18 22:42:52hayposetnosy: + haypo
messages: + msg187305
2013-04-18 13:56:44r.david.murraysetnosy: + r.david.murray
messages: + msg187251
2013-02-27 20:03:15brett.cannonsetdependencies: + Deprecate imp.find_module()/load_module()
2013-02-15 20:57:35ezio.melottisetmessages: + msg182173
2013-02-15 20:35:06brett.cannonsetmessages: + msg182168
2013-02-15 20:15:34ezio.melottisettype: enhancement

messages: + msg182165
nosy: + ezio.melotti
2013-02-15 17:54:48brett.cannonsetmessages: + msg182155
2013-02-15 17:32:45eric.araujosetnosy: + eric.araujo
messages: + msg182152
2013-02-10 20:08:31brett.cannoncreate