I need a make a decision as to what (if anything) belongs in imp and then document what stays and deprecate everything else.

Everything in imp falls into one of the following categories:

* From importlib
  - get_magic()
  - source_from_cache()
  - cache_from_source()
* From sys
  - get_tag()
* Platform-implemented
  - lock_held()
  - acquire_lock()
  - release_lock()
  - Undocumented stuff related to builtins, frozen, and load_dynamic()
* Helper
  - reload()

The question is what to keep/expose in imp and what to deprecate. Basically I need to figure out what imp's role is supposed to be in the face of importlib. My gut says either expose platform-dependent stuff and reload() and move the rest to importlib/deprecate, or to completely do away with the module and force people to use the APIs in importlib for consistency (and to stop people from mucking around with import from outside of importlib).

I’ve always perceived imp as an implementation detail of the import system that people were sometimes forced to use to get to some things.  Now that we have importlib, deprecating (with a nice long compatibility-with-warnings period) sounds good.

Would checking with the other VMs be a good idea?

Checking with the other VMs wouldn't benefit anyone. They still have to implement whatever is necessary from _imp in order for importlib to work. Otherwise it's just another stdlib module.

> Basically I need to figure out what imp's role is supposed to be in the face of importlib.

If it's not even clear for you, even if you eventually figure that out, having two import-related modules is likely to be confusing for users.
OTOH deprecating/removing imp would affect the somewhat popular imp.reload(), that has already been moved for Python 3, and will now be moved again (this also means that the -3 warnings on 2.7 will no longer be valid).

I would probably document the deprecation but otherwise leave it alone at the module level. Considering I only documented imp.find_module() and .load_module() as documented thanks to their funky interfaces making nearly impossible to do a direct translation I don't see how I can justify raising a DeprecationWarning for the whole thing.

But it's gone in Python 4. =)

If you move them to importlib and just leave an alias in imp, you should still add a DeprecationWarning IMHO, even if you plan to remove them in Python 4.

The documentation deprecation has been added as part of issue 17135.

Leaving this open to address Ezio's last comment.

If the imp module is marked as deprecated, it would help to explain how to port code from imp to importlib. As it was done for os.spawn(), platform.popen(), etc.

I'm using imp.load_source() in my setup.py to load a version.py file which only contains constants, to be able to build my module even if executing the module fails (ex: because of a runtime dependency). I don't know how to change my code to use imp.load_source().

> I don't know how to change my code to use imp.load_source().

Woops, I mean "to use importlib".

The deprecation warnings for the explicitly deprecated functions already have the instructions on how to port. A note could probably be added to turn on deprecation warnings if one needs help with moving code over.

But the deprecation isn't even officially full-on yet. While it's documented as such, not all functionality has moved over to importlib, only bits and pieces.

Should we back out that module level deprecation notice for the moment, then?

Nah, consider it motivation for me to get it done in Python 3.4 else I have to back it out before release.

Just to keep on top of what is left to do for this to happen:

* get_magic(): expose in importlib
* find_module()/load_module(): eliminate all uses in the stdlib and fully deprecate
* new_module(): use types.ModuleType() instead
* reload(): relocate
* cache_from_source(): relocate
* source_from_cache(): relocate
* get_tag(): deprecate
* lock_held()/acquire_lock()/release_lock(): deprecate
* NullImporter: deprecate

Everything else can stay undocumented as-is (deprecated or simply exposed through _imp which can stay undocumented).

I plan to make individual issues for these things as I solidify the decision of how to handle them.

While thinking about asking on G+ for feedback, I realized that if I get this to the point of a module-level warning, it won't be a big deal. It would be just like warnings you deal with when supporting old versions of Python along with newer versions where things are now deprecated. That way  users of imp could just do::

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    import imp

and be done with it. This means still striving to deprecate every function in imp so that it can all be replaced with a single module-level deprecation is a reasonable goal to have.

This might be OK, however you won't be able to point to the alternatives in the deprecation message (e.g. "new_module() is deprecated, use types.ModuleType() instead").
This can still be done in the docs though.

Right, the docs would say "the imp modules is deprecated; please see the module documentation for alternatives" or something.

There should now be issues for each of the required relocations to make imp redundant.

Everything that is going to be kept around has been moved. That just leaves:

1. Removing all uses of imp from the stdlib
2. Document as deprecated the last few things in imp which are simply being removed (including adding a note in the docstrings)
3. Remove all function/class-level deprecation warnings and add a module-level pending deprecation one
4. Run the test suite with -W always to make sure I didn't miss anything

And in case anyone is interested, it could be as many as 98 files to go through. =)

Sorry, bad regex: it's more like 60 files.

New changeset 3d3b9d456eb8 by Brett Cannon in branch 'default':
Issue #17177: Update the programming FAQ to use importlib
http://hg.python.org/cpython/rev/3d3b9d456eb8

To have it on record, the command I am using to find uses of imp is::

 ack "import imp(,|\n)|from imp import" -l

ack "import (\w+, *)*imp(,|\n)|from imp import" -l

New changeset cc27d50bd91a by Brett Cannon in branch 'default':
Issue #17177: stop using imp for compileall.
http://hg.python.org/cpython/rev/cc27d50bd91a

New changeset 30aa032c4bd0 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in distutils
http://hg.python.org/cpython/rev/30aa032c4bd0

New changeset 0b96a16656b7 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in multiprocessing
http://hg.python.org/cpython/rev/0b96a16656b7

New changeset 91467f342977 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp with py_compile
http://hg.python.org/cpython/rev/91467f342977

New changeset 81cf3d6e6756 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in pydoc
http://hg.python.org/cpython/rev/81cf3d6e6756

New changeset 8fff5554125d by Brett Cannon in branch 'default':
Issue #17177: switch from imp.new_module to types.ModuleType for runpy
http://hg.python.org/cpython/rev/8fff5554125d

New changeset 1e141c2cc0d7 by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in sysconfig
http://hg.python.org/cpython/rev/1e141c2cc0d7

New changeset bc18b5d4920e by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in a bunch of tests
http://hg.python.org/cpython/rev/bc18b5d4920e

New changeset f96eb1dc335f by Brett Cannon in branch 'default':
Issue #17177: Stop using imp in zipfile
http://hg.python.org/cpython/rev/f96eb1dc335f

New changeset b2b2bdc4cf6f by Brett Cannon in branch 'default':
Issue # 17177: Stop using imp in turtledemo
http://hg.python.org/cpython/rev/b2b2bdc4cf6f

New changeset 4a1161eaed99 by Brett Cannon in branch 'default':
Issue # 17177: Stop using imp in setup.py
http://hg.python.org/cpython/rev/4a1161eaed99

New changeset ca3bdac1f88a by Brett Cannon in branch 'default':
Issue #17177: update checkpyc to stop using imp
http://hg.python.org/cpython/rev/ca3bdac1f88a

New changeset 5e8b377942f7 by Brett Cannon in branch 'default':
Issue #17177: stop using imp in test_importlib
http://hg.python.org/cpython/rev/5e8b377942f7

At this point I have updated all the code that does not directly exposes imp somehow or in Tools.

At this point the only thing I need to decide is whether I want to continue to expose the lock stuff from imp or leave it private. After that the pending deprecation is ready to go.

New changeset 1b8f08c4efd5 by Brett Cannon in branch 'default':
Issue #17177: The imp module is pending deprecation.
http://hg.python.org/cpython/rev/1b8f08c4efd5

Going to temporarily re-open to remind me to add better deprecation docs for imp.load_module() to point to specific loader in importlib.machinery.

Should also add example usage as well.

New changeset ded443c603f0 by Brett Cannon in branch 'default':
Issue #17177: Clarify some deprecations
http://hg.python.org/cpython/rev/ded443c603f0