Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(23135)

Side by Side Diff: Doc/reference/import.rst

Issue 18864: Implementation for PEP 451 (importlib.machinery.ModuleSpec)
Patch Set: Created 5 years, 7 months ago
Left:
Right:
Use n/p to move between diff chunks; N/P to move between comments. Please Sign in to add in-line comments.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | no next file » | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 1
2 .. _importsystem: 2 .. _importsystem:
3 3
4 ***************** 4 *****************
5 The import system 5 The import system
6 ***************** 6 *****************
7 7
8 .. index:: single: import machinery 8 .. index:: single: import machinery
9 9
10 Python code in one :term:`module` gains access to the code in another module 10 Python code in one :term:`module` gains access to the code in another module
(...skipping 192 matching lines...) Expand 10 before | Expand all | Expand 10 after
203 :func:`imp.reload` will reuse the *same* module object, and simply 203 :func:`imp.reload` will reuse the *same* module object, and simply
204 reinitialise the module contents by rerunning the module's code. 204 reinitialise the module contents by rerunning the module's code.
205 205
206 206
207 Finders and loaders 207 Finders and loaders
208 ------------------- 208 -------------------
209 209
210 .. index:: 210 .. index::
211 single: finder 211 single: finder
212 single: loader 212 single: loader
213 single: module spec
213 214
214 If the named module is not found in :data:`sys.modules`, then Python's import 215 If the named module is not found in :data:`sys.modules`, then Python's import
215 protocol is invoked to find and load the module. This protocol consists of 216 protocol is invoked to find and load the module. This protocol consists of
216 two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`. 217 two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`.
217 A finder's job is to determine whether it can find the named module using 218 A finder's job is to determine whether it can find the named module using
218 whatever strategy it knows about. Objects that implement both of these 219 whatever strategy it knows about. Objects that implement both of these
219 interfaces are referred to as :term:`importers <importer>` - they return 220 interfaces are referred to as :term:`importers <importer>` - they return
220 themselves when they find that they can load the requested module. 221 themselves when they find that they can load the requested module.
221 222
222 Python includes a number of default finders and importers. The first one 223 Python includes a number of default finders and importers. The first one
223 knows how to locate built-in modules, and the second knows how to locate 224 knows how to locate built-in modules, and the second knows how to locate
224 frozen modules. A third default finder searches an :term:`import path` 225 frozen modules. A third default finder searches an :term:`import path`
225 for modules. The :term:`import path` is a list of locations that may 226 for modules. The :term:`import path` is a list of locations that may
226 name file system paths or zip files. It can also be extended to search 227 name file system paths or zip files. It can also be extended to search
227 for any locatable resource, such as those identified by URLs. 228 for any locatable resource, such as those identified by URLs.
228 229
229 The import machinery is extensible, so new finders can be added to extend the 230 The import machinery is extensible, so new finders can be added to extend the
230 range and scope of module searching. 231 range and scope of module searching.
231 232
232 Finders do not actually load modules. If they can find the named module, they 233 Finders do not actually load modules. If they can find the named module, they
233 return a :term:`loader`, which the import machinery then invokes to load the 234 return a :term:`module spec`, an encapsulation of the module's import-related
234 module and create the corresponding module object. 235 information, which the import machinery then uses when loading the module.
235 236
236 The following sections describe the protocol for finders and loaders in more 237 The following sections describe the protocol for finders and loaders in more
237 detail, including how you can create and register new ones to extend the 238 detail, including how you can create and register new ones to extend the
238 import machinery. 239 import machinery.
239 240
241 .. versionchanged:: 3.4
242 In previous versions of Python, finders returned :term:`loaders <loader>`
243 directly, whereas now they return module specs which *contain* loaders.
244 Loaders are still used during import but have fewer responsibilities.
240 245
241 Import hooks 246 Import hooks
242 ------------ 247 ------------
243 248
244 .. index:: 249 .. index::
245 single: import hooks 250 single: import hooks
246 single: meta hooks 251 single: meta hooks
247 single: path hooks 252 single: path hooks
248 pair: hooks; import 253 pair: hooks; import
249 pair: hooks; meta 254 pair: hooks; meta
(...skipping 13 matching lines...) Expand all
263 ``package.__path__``) processing, at the point where their associated path 268 ``package.__path__``) processing, at the point where their associated path
264 item is encountered. Import path hooks are registered by adding new callables 269 item is encountered. Import path hooks are registered by adding new callables
265 to :data:`sys.path_hooks` as described below. 270 to :data:`sys.path_hooks` as described below.
266 271
267 272
268 The meta path 273 The meta path
269 ------------- 274 -------------
270 275
271 .. index:: 276 .. index::
272 single: sys.meta_path 277 single: sys.meta_path
273 pair: finder; find_module 278 pair: finder; find_spec
274 pair: finder; find_loader
275 279
276 When the named module is not found in :data:`sys.modules`, Python next 280 When the named module is not found in :data:`sys.modules`, Python next
277 searches :data:`sys.meta_path`, which contains a list of meta path finder 281 searches :data:`sys.meta_path`, which contains a list of meta path finder
278 objects. These finders are queried in order to see if they know how to handle 282 objects. These finders are queried in order to see if they know how to handle
279 the named module. Meta path finders must implement a method called 283 the named module. Meta path finders must implement a method called
280 :meth:`find_module()` which takes two arguments, a name and an import path. 284 :meth:`find_spec()` which takes two arguments, a name and an import path.
281 The meta path finder can use any strategy it wants to determine whether it can 285 The meta path finder can use any strategy it wants to determine whether it can
282 handle the named module or not. 286 handle the named module or not.
283 287
284 If the meta path finder knows how to handle the named module, it returns a 288 If the meta path finder knows how to handle the named module, it returns a
285 loader object. If it cannot handle the named module, it returns ``None``. If 289 spec object. If it cannot handle the named module, it returns ``None``. If
286 :data:`sys.meta_path` processing reaches the end of its list without returning 290 :data:`sys.meta_path` processing reaches the end of its list without returning
287 a loader, then an :exc:`ImportError` is raised. Any other exceptions raised 291 a spec, then an :exc:`ImportError` is raised. Any other exceptions raised
288 are simply propagated up, aborting the import process. 292 are simply propagated up, aborting the import process.
289 293
290 The :meth:`find_module()` method of meta path finders is called with two 294 The :meth:`find_spec()` method of meta path finders is called with two
291 arguments. The first is the fully qualified name of the module being 295 arguments. The first is the fully qualified name of the module being
292 imported, for example ``foo.bar.baz``. The second argument is the path 296 imported, for example ``foo.bar.baz``. The second argument is the path
293 entries to use for the module search. For top-level modules, the second 297 entries to use for the module search. For top-level modules, the second
294 argument is ``None``, but for submodules or subpackages, the second 298 argument is ``None``, but for submodules or subpackages, the second
295 argument is the value of the parent package's ``__path__`` attribute. If 299 argument is the value of the parent package's ``__path__`` attribute. If
296 the appropriate ``__path__`` attribute cannot be accessed, an 300 the appropriate ``__path__`` attribute cannot be accessed, an
297 :exc:`ImportError` is raised. 301 :exc:`ImportError` is raised.
302
303 Meta path finders in earlier versions of Python implemented a
brett.cannon 2013/10/25 17:37:14 I would make this a versionchanged comment.
eric.snow 2013/10/26 23:35:03 Done.
304 :meth:`find_module()` method instead of find_spec(). The method takes
305 the same arguments but returns just the loader (instead of the whole spec).
306 However, find_module() is deprecated. While it will continue to work
307 without change, the import machinery will try it only if the finder does
308 not implement find_spec().
298 309
299 The meta path may be traversed multiple times for a single import request. 310 The meta path may be traversed multiple times for a single import request.
300 For example, assuming none of the modules involved has already been cached, 311 For example, assuming none of the modules involved has already been cached,
301 importing ``foo.bar.baz`` will first perform a top level import, calling 312 importing ``foo.bar.baz`` will first perform a top level import, calling
302 ``mpf.find_module("foo", None)`` on each meta path finder (``mpf``). After 313 ``mpf.find_spec("foo", None)`` on each meta path finder (``mpf``). After
303 ``foo`` has been imported, ``foo.bar`` will be imported by traversing the 314 ``foo`` has been imported, ``foo.bar`` will be imported by traversing the
304 meta path a second time, calling 315 meta path a second time, calling
305 ``mpf.find_module("foo.bar", foo.__path__)``. Once ``foo.bar`` has been 316 ``mpf.find_spec("foo.bar", foo.__path__)``. Once ``foo.bar`` has been
306 imported, the final traversal will call 317 imported, the final traversal will call
307 ``mpf.find_module("foo.bar.baz", foo.bar.__path__)``. 318 ``mpf.find_spec("foo.bar.baz", foo.bar.__path__)``.
308 319
309 Some meta path finders only support top level imports. These importers will 320 Some meta path finders only support top level imports. These importers will
310 always return ``None`` when anything other than ``None`` is passed as the 321 always return ``None`` when anything other than ``None`` is passed as the
311 second argument. 322 second argument.
312 323
313 Python's default :data:`sys.meta_path` has three meta path finders, one that 324 Python's default :data:`sys.meta_path` has three meta path finders, one that
314 knows how to import built-in modules, one that knows how to import frozen 325 knows how to import built-in modules, one that knows how to import frozen
315 modules, and one that knows how to import modules from an :term:`import path` 326 modules, and one that knows how to import modules from an :term:`import path`
316 (i.e. the :term:`path based finder`). 327 (i.e. the :term:`path based finder`).
317 328
329 .. versionchanged:: 3.4
330 The find_spec() method of meta path finders replaced find_module().
331
332
333 Loading
334 =======
335
336 If and when a module spec is found, the import machinery will use it (and
337 the loader it contains) when loading the module. Here is an approximation
338 of what happens during the loading portion of import::
339
340 module = None
341 if hasattr(spec.loader, 'create_module'):
342 module = spec.loader.create_module(spec)
343 if module is None:
344 module = ModuleType(spec.name)
345 init_module_attrs(module)
346
347 sys.modules[spec.name] = module
348 try:
349 spec.loader.exec_module(module)
350 except BaseException:
351 try:
352 del sys.modules[spec.name]
353 except KeyError:
354 pass
355 raise
356
357 Note the following details:
358
359 * If there is an existing module object with the given name in
360 :data:`sys.modules`, import have already returned it.
brett.cannon 2013/10/25 17:37:14 "import will have"
eric.snow 2013/10/26 23:35:03 Done.
361
362 * The module will exist in :data:`sys.modules` before the loader
363 executes the module code. This is crucial because the module code may
364 (directly or indirectly) import itself; adding it to :data:`sys.modules`
365 beforehand prevents unbounded recursion in the worst case and multiple
366 loading in the best.
367
368 * If loading fails, the failing module--and only the failing module--gets
brett.cannon 2013/10/25 17:37:14 Spaces around the emdashes
eric.snow 2013/10/26 23:35:03 Done.
369 removed from :data:`sys.modules`. Any module already in the
370 :data:`sys.modules` cache, and any module that was successfully loaded
371 as a side-effect, must remain in the cache. This contrasts with
372 reloading where even the failing module is left in :data:`sys.modules`.
373
374 * After the module is created but before execution, the import machinery
375 sets the import-related module attributes ("init_module_attrs"), as
376 summarized in a `later section <Import-related module attributes>`_.
377
378 * Module execution is the key moment of loading in which the module's
379 namespace gets populated. Execution is entirely delegated to the
380 loader, which gets to decide what gets populated and how.
381
382 * The module created during loading and passed to exec_module() may
383 not be the one returned at the end of import [#fnlo]_.
384
385 .. versionchanged:: 3.4
386 The import system has taken over the boilerplate responsibilities of
387 loaders. These were previously performed by the :meth:`load_module()`
388 method.
318 389
319 Loaders 390 Loaders
320 ======= 391 -------
321 392
322 If and when a module loader is found its 393 Module loaders provide the critical function of loading: module execution.
323 :meth:`~importlib.abc.Loader.load_module` method is called, with a single 394 The import machinery calls the :meth:`~importlib.abc.Loader.exec_module()`
324 argument, the fully qualified name of the module being imported. This method 395 method with a single argument, the module object to execute. Any value
325 has several responsibilities, and should return the module object it has 396 returned from exec_module() is ignored.
326 loaded [#fnlo]_. If it cannot load the module, it should raise an
327 :exc:`ImportError`, although any other exception raised during
328 :meth:`load_module()` will be propagated.
329
330 In many cases, the finder and loader can be the same object; in such cases the
331 :meth:`finder.find_module()` would just return ``self``.
332 397
333 Loaders must satisfy the following requirements: 398 Loaders must satisfy the following requirements:
334 399
400 * If the module is a Python module (as opposed to a built-in module or a
brett.cannon 2013/10/25 17:37:14 Simplify this to something like "Execute the modul
eric.snow 2013/10/26 23:35:03 I can do that. Keep in mind that this passage was
401 dynamically loaded extension), the loader should execute the module's code
402 in the module's global name space (``module.__dict__``).
403
404 * If loader cannot execute the module, it should raise an
405 :exc:`ImportError`, although any other exception raised during
406 :meth:`exec_module()` will be propagated.
407
408 In many cases, the finder and loader can be the same object; in such cases the
409 :meth:`finder.find_spec()` would just return a spec with the loader set
410 to ``self``.
411
412 For compatibility with existing loaders, the import machinery will use
brett.cannon 2013/10/25 17:37:14 Make all of this a versionchanged comment; no nee
eric.snow 2013/10/26 23:35:03 Done.
413 the :meth:`~importlib.abc.Loader.load_module()` method of loaders if it
414 exists and the loader does not also implement exec_module(). However,
415 load_module() has been deprecated and loaders should implement
416 exec_module() instead.
417
418 The load_module() method must implement all the boilerplate loading
419 functionality described above in addition to executing the module. All
420 the same constraints apply, with some additional clarification:
421
335 * If there is an existing module object with the given name in 422 * If there is an existing module object with the given name in
336 :data:`sys.modules`, the loader must use that existing module. (Otherwise, 423 :data:`sys.modules`, the loader must use that existing module. (Otherwise,
337 :func:`imp.reload` will not work correctly.) If the named module does 424 :func:`imp.reload` will not work correctly.) If the named module does
338 not exist in :data:`sys.modules`, the loader must create a new module 425 not exist in :data:`sys.modules`, the loader must create a new module
339 object and add it to :data:`sys.modules`. 426 object and add it to :data:`sys.modules`.
340 427
341 Note that the module *must* exist in :data:`sys.modules` before the loader 428 * The module *must* exist in :data:`sys.modules` before the loader
342 executes the module code. This is crucial because the module code may 429 executes the module code, to prevent unbounded recursion or multiple
343 (directly or indirectly) import itself; adding it to :data:`sys.modules` 430 loading.
344 beforehand prevents unbounded recursion in the worst case and multiple
345 loading in the best.
346 431
347 If loading fails, the loader must remove any modules it has inserted into 432 * If loading fails, the loader must remove any modules it has inserted into
348 :data:`sys.modules`, but it must remove **only** the failing module, and 433 :data:`sys.modules`, but it must remove **only** the failing module, and
349 only if the loader itself has loaded it explicitly. Any module already in 434 only if the loader itself has loaded it explicitly.
350 the :data:`sys.modules` cache, and any module that was successfully loaded
351 as a side-effect, must remain in the cache.
352 435
353 * The loader may set the ``__file__`` attribute of the module. If set, this 436 Module loaders may opt in to creating the module object during loading
354 attribute's value must be a string. The loader may opt to leave 437 by implementing a :meth:`create_module()` method. It takes one argument,
355 ``__file__`` unset if it has no semantic meaning (e.g. a module loaded from 438 the module spec, and returns the new module object to use during loading.
356 a database). If ``__file__`` is set, it may also be appropriate to set the 439 create_module() does not need to set any attributes on the module object.
357 ``__cached__`` attribute which is the path to any compiled version of the 440 If the loader does not define create_module(), the import machinery will
358 code (e.g. byte-compiled file). The file does not need to exist to set this 441 create the new module itself.
359 attribute; the path can simply point to whether the compiled file would
360 exist (see :pep:`3147`).
361 442
362 * The loader may set the ``__name__`` attribute of the module. While not 443 .. versionadded:: 3.4
363 required, setting this attribute is highly recommended so that the 444 The create_module() method of loaders.
364 :meth:`repr()` of the module is more informative.
365 445
366 * If the module is a package (either regular or namespace), the loader must 446 .. versionchanged:: 3.4
447 The load_module() method was replaced by exec_module() and the import
448 machinery assumed all the boilerplate responsibilities of loading.
449
450 Module spec
451 -----------
452
453 The import machinery uses a variety of information about each module
454 during import, especially before loading. Most of the information is
455 common to all modules. The purpose of a module's spec is to encapsulate
456 this import-related information on a per-module basis.
457
458 Using a spec during import allows state to be transferred between import
459 system components, e.g. between the finder that creates the module spec
460 and the loader that executes it. Most importantly, it allows the
461 import machinery to perform the boilerplate operations of loading,
462 whereas without a module spec the loader had that responsibility.
463
464 See :class:`~importlib.machinery.ModuleSpec` for more specifics on what
465 information a module's spec may hold.
466
467 .. versionadded:: 3.4
468
469 Import-related module attributes
470 --------------------------------
brett.cannon 2013/10/25 17:37:14 Probably want a comment/note/something that init_m
eric.snow 2013/10/26 23:35:03 Done.
471
472 .. attribute:: __name__
473
474 The import machinery may set the ``__name__`` attribute of the
475 module. While not required, setting this attribute is highly
476 recommended so that the :meth:`repr()` of the module is more
477 informative.
478
479 .. attribute:: __loader__
480
481 The ``__loader__`` attribute must be set to the loader object that
482 loaded the module. This is mostly for introspection and reloading,
483 but can be used for additional loader-specific functionality, for
484 example getting data associated with a loader. If the attribute is
485 missing or set to ``None`` then the import machinery will
486 automatically set it **after** the module has been imported.
487
488 .. attribute:: __package__
489
490 The module's ``__package__`` attribute must be set. Its value must
491 be a string, but it can be the same value as its ``__name__``. If
492 the attribute is set to ``None`` or is missing, the import system
493 will fill it in with a more appropriate value **after** the module
494 has been imported. When the module is a package, its ``__package__``
brett.cannon 2013/10/25 17:37:14 I don't know if we want to keep this requirement a
brett.cannon 2013/10/25 17:42:09 I meant __getattrribute__, not __setattr__
eric.snow 2013/10/26 23:35:03 Done.
495 value should be set to its ``__name__``. When the module is not a
496 package, ``__package__`` should be set to the empty string for
497 top-level modules, or for submodules, to the parent package's name.
498 See :pep:`366` for further details.
499
500 This attribute is used instead of ``__name__`` to calculate explicit
501 relative imports for main modules, as defined in :pep:`366`.
502
503 .. attribute:: __spec__
504
505 The module spec that was used when importing the module.
506
507 .. attribute:: __path__
508
509 If the module is a package (either regular or namespace), the loader must
367 set the module object's ``__path__`` attribute. The value must be 510 set the module object's ``__path__`` attribute. The value must be
368 iterable, but may be empty if ``__path__`` has no further significance 511 iterable, but may be empty if ``__path__`` has no further significance
369 to the loader. If ``__path__`` is not empty, it must produce strings 512 to the loader. If ``__path__`` is not empty, it must produce strings
370 when iterated over. More details on the semantics of ``__path__`` are 513 when iterated over. More details on the semantics of ``__path__`` are
371 given :ref:`below <package-path-rules>`. 514 given :ref:`below <package-path-rules>`.
372 515
373 * The ``__loader__`` attribute must be set to the loader object that loaded 516 .. attribute:: __file__
374 the module. This is mostly for introspection and reloading, but can be 517 .. attribute:: __cached__
375 used for additional loader-specific functionality, for example getting
376 data associated with a loader. If the attribute is missing or set to ``None``
377 then the import machinery will automatically set it **after** the module has
378 been imported.
379 518
380 * The module's ``__package__`` attribute must be set. Its value must be a 519 ``__file__`` is optional. If set, this attribute's value must be a
381 string, but it can be the same value as its ``__name__``. If the attribute 520 string. The import system may opt to leave ``__file__`` unset if it
382 is set to ``None`` or is missing, the import system will fill it in with a 521 has no semantic meaning (e.g. a module loaded from a database). If
383 more appropriate value **after** the module has been imported. 522 ``__file__`` is set, it may also be appropriate to set the
384 When the module is a package, its ``__package__`` value should be set to its 523 ``__cached__`` attribute which is the path to any compiled version of
385 ``__name__``. When the module is not a package, ``__package__`` should be 524 the code (e.g. byte-compiled file). The file does not need to exist
386 set to the empty string for top-level modules, or for submodules, to the 525 to set this attribute; the path can simply point to whether the
387 parent package's name. See :pep:`366` for further details. 526 compiled file would exist (see :pep:`3147`).
388
389 This attribute is used instead of ``__name__`` to calculate explicit
390 relative imports for main modules, as defined in :pep:`366`.
391
392 * If the module is a Python module (as opposed to a built-in module or a
393 dynamically loaded extension), the loader should execute the module's code
394 in the module's global name space (``module.__dict__``).
395
396
397 Module reprs
398 ------------
399
400 By default, all modules have a usable repr, however depending on the
401 attributes set above, and hooks in the loader, you can more explicitly control
402 the repr of module objects.
403
404 Loaders may implement a :meth:`module_repr()` method which takes a single
405 argument, the module object. When ``repr(module)`` is called for a module
406 with a loader supporting this protocol, whatever is returned from
407 ``module.__loader__.module_repr(module)`` is returned as the module's repr
408 without further processing. This return value must be a string.
409
410 If the module has no ``__loader__`` attribute, or the loader has no
411 :meth:`module_repr()` method, then the module object implementation itself
412 will craft a default repr using whatever information is available. It will
413 try to use the ``module.__name__``, ``module.__file__``, and
414 ``module.__loader__`` as input into the repr, with defaults for whatever
415 information is missing.
416
417 Here are the exact rules used:
418
419 * If the module has a ``__loader__`` and that loader has a
420 :meth:`module_repr()` method, call it with a single argument, which is the
421 module object. The value returned is used as the module's repr.
422
423 * If an exception occurs in :meth:`module_repr()`, the exception is caught
424 and discarded, and the calculation of the module's repr continues as if
425 :meth:`module_repr()` did not exist.
426
427 * If the module has a ``__file__`` attribute, this is used as part of the
428 module's repr.
429
430 * If the module has no ``__file__`` but does have a ``__loader__`` that is not
431 ``None``, then the loader's repr is used as part of the module's repr.
432
433 * Otherwise, just use the module's ``__name__`` in the repr.
434
435 This example, from :pep:`420` shows how a loader can craft its own module
436 repr::
437
438 class NamespaceLoader:
439 @classmethod
440 def module_repr(cls, module):
441 return "<module '{}' (namespace)>".format(module.__name__)
442
443 527
444 .. _package-path-rules: 528 .. _package-path-rules:
445 529
446 module.__path__ 530 module.__path__
447 --------------- 531 ---------------
448 532
449 By definition, if a module has an ``__path__`` attribute, it is a package, 533 By definition, if a module has an ``__path__`` attribute, it is a package,
450 regardless of its value. 534 regardless of its value.
451 535
452 A package's ``__path__`` attribute is used during imports of its subpackages. 536 A package's ``__path__`` attribute is used during imports of its subpackages.
453 Within the import machinery, it functions much the same as :data:`sys.path`, 537 Within the import machinery, it functions much the same as :data:`sys.path`,
454 i.e. providing a list of locations to search for modules during import. 538 i.e. providing a list of locations to search for modules during import.
455 However, ``__path__`` is typically much more constrained than 539 However, ``__path__`` is typically much more constrained than
456 :data:`sys.path`. 540 :data:`sys.path`.
457 541
458 ``__path__`` must be an iterable of strings, but it may be empty. 542 ``__path__`` must be an iterable of strings, but it may be empty.
459 The same rules used for :data:`sys.path` also apply to a package's 543 The same rules used for :data:`sys.path` also apply to a package's
460 ``__path__``, and :data:`sys.path_hooks` (described below) are 544 ``__path__``, and :data:`sys.path_hooks` (described below) are
461 consulted when traversing a package's ``__path__``. 545 consulted when traversing a package's ``__path__``.
462 546
463 A package's ``__init__.py`` file may set or alter the package's ``__path__`` 547 A package's ``__init__.py`` file may set or alter the package's ``__path__``
464 attribute, and this was typically the way namespace packages were implemented 548 attribute, and this was typically the way namespace packages were implemented
465 prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no 549 prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no
466 longer need to supply ``__init__.py`` files containing only ``__path__`` 550 longer need to supply ``__init__.py`` files containing only ``__path__``
467 manipulation code; the namespace loader automatically sets ``__path__`` 551 manipulation code; the import machinery automatically sets ``__path__``
468 correctly for the namespace package. 552 correctly for the namespace package.
553
554 Module reprs
555 ------------
556
557 By default, all modules have a usable repr, however depending on the
558 attributes set above, and in the module's spec, you can more explicitly
559 control the repr of module objects.
560
561 If the module has a spec (``__spec__``), the import machinery will try
562 to generate a repr from it. If that fails or there is no spec, the import
563 system will craft a default repr using whatever information is available
564 on the module. It will try to use the ``module.__name__``,
565 ``module.__file__``, and ``module.__loader__`` as input into the repr,
566 with defaults for whatever information is missing.
567
568 For backward compatibility with Python 3.3, the module repr will be
brett.cannon 2013/10/25 17:37:14 Make this a versionchanged comment
eric.snow 2013/10/26 23:35:03 Done.
569 generated by calling the loader's :meth:`module_repr()` method, if
570 defined, before trying either approach described above. However, the
571 method is deprecated.
572
573 Here are the exact rules used:
574
575 * If the module has a ``__loader__`` and that loader has a
576 :meth:`module_repr()` method, call it with a single argument, which is the
577 module object. The value returned is used as the module's repr.
578
579 * If an exception occurs in :meth:`module_repr()`, the exception is caught
580 and discarded, and the calculation of the module's repr continues as if
581 :meth:`module_repr()` did not exist.
582
583 * If the module has a ``__spec__`` attribute, the information in the spec
584 is used to generate the repr. The "name", "loader", "origin", and
585 "has_location" attributes are consulted.
586
587 * If the module has a ``__file__`` attribute, this is used as part of the
588 module's repr.
589
590 * If the module has no ``__file__`` but does have a ``__loader__`` that is not
591 ``None``, then the loader's repr is used as part of the module's repr.
592
593 * Otherwise, just use the module's ``__name__`` in the repr.
594
595 .. versionchanged:: 3.4
596 Use of loader.module_repr() has been deprecated and the module spec
597 is now used by the import machinery to generate a module repr.
469 598
470 599
471 The Path Based Finder 600 The Path Based Finder
472 ===================== 601 =====================
473 602
474 .. index:: 603 .. index::
475 single: path based finder 604 single: path based finder
476 605
477 As mentioned previously, Python comes with several default meta path finders. 606 As mentioned previously, Python comes with several default meta path finders.
478 One of these, called the :term:`path based finder`, searches an :term:`import 607 One of these, called the :term:`path based finder`, searches an :term:`import
(...skipping 45 matching lines...) Expand 10 before | Expand all | Expand 10 after
524 single: sys.path_hooks 653 single: sys.path_hooks
525 single: sys.path_importer_cache 654 single: sys.path_importer_cache
526 single: PYTHONPATH 655 single: PYTHONPATH
527 656
528 The :term:`path based finder` is responsible for finding and loading Python 657 The :term:`path based finder` is responsible for finding and loading Python
529 modules and packages whose location is specified with a string :term:`path 658 modules and packages whose location is specified with a string :term:`path
530 entry`. Most path entries name locations in the file system, but they need 659 entry`. Most path entries name locations in the file system, but they need
531 not be limited to this. 660 not be limited to this.
532 661
533 As a meta path finder, the :term:`path based finder` implements the 662 As a meta path finder, the :term:`path based finder` implements the
534 :meth:`find_module()` protocol previously described, however it exposes 663 :meth:`find_spec()` protocol previously described, however it exposes
535 additional hooks that can be used to customize how modules are found and 664 additional hooks that can be used to customize how modules are found and
536 loaded from the :term:`import path`. 665 loaded from the :term:`import path`.
537 666
538 Three variables are used by the :term:`path based finder`, :data:`sys.path`, 667 Three variables are used by the :term:`path based finder`, :data:`sys.path`,
539 :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__`` 668 :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__``
540 attributes on package objects are also used. These provide additional ways 669 attributes on package objects are also used. These provide additional ways
541 that the import machinery can be customized. 670 that the import machinery can be customized.
542 671
543 :data:`sys.path` contains a list of strings providing search locations for 672 :data:`sys.path` contains a list of strings providing search locations for
544 modules and packages. It is initialized from the :data:`PYTHONPATH` 673 modules and packages. It is initialized from the :data:`PYTHONPATH`
545 environment variable and various other installation- and 674 environment variable and various other installation- and
546 implementation-specific defaults. Entries in :data:`sys.path` can name 675 implementation-specific defaults. Entries in :data:`sys.path` can name
547 directories on the file system, zip files, and potentially other "locations" 676 directories on the file system, zip files, and potentially other "locations"
548 (see the :mod:`site` module) that should be searched for modules, such as 677 (see the :mod:`site` module) that should be searched for modules, such as
549 URLs, or database queries. Only strings and bytes should be present on 678 URLs, or database queries. Only strings and bytes should be present on
550 :data:`sys.path`; all other data types are ignored. The encoding of bytes 679 :data:`sys.path`; all other data types are ignored. The encoding of bytes
551 entries is determined by the individual :term:`path entry finders <path entry 680 entries is determined by the individual :term:`path entry finders <path entry
552 finder>`. 681 finder>`.
553 682
554 The :term:`path based finder` is a :term:`meta path finder`, so the import 683 The :term:`path based finder` is a :term:`meta path finder`, so the import
555 machinery begins the :term:`import path` search by calling the path 684 machinery begins the :term:`import path` search by calling the path
556 based finder's :meth:`find_module()` method as described previously. When 685 based finder's :meth:`find_spec()` method as described previously. When
557 the ``path`` argument to :meth:`find_module()` is given, it will be a 686 the ``path`` argument to :meth:`find_spec()` is given, it will be a
558 list of string paths to traverse - typically a package's ``__path__`` 687 list of string paths to traverse - typically a package's ``__path__``
559 attribute for an import within that package. If the ``path`` argument 688 attribute for an import within that package. If the ``path`` argument
560 is ``None``, this indicates a top level import and :data:`sys.path` is used. 689 is ``None``, this indicates a top level import and :data:`sys.path` is used.
561 690
562 The path based finder iterates over every entry in the search path, and 691 The path based finder iterates over every entry in the search path, and
563 for each of these, looks for an appropriate :term:`path entry finder` for the 692 for each of these, looks for an appropriate :term:`path entry finder` for the
564 path entry. Because this can be an expensive operation (e.g. there may be 693 path entry. Because this can be an expensive operation (e.g. there may be
565 `stat()` call overheads for this search), the path based finder maintains 694 `stat()` call overheads for this search), the path based finder maintains
566 a cache mapping path entries to path entry finders. This cache is maintained 695 a cache mapping path entries to path entry finders. This cache is maintained
567 in :data:`sys.path_importer_cache` (despite the name, this cache actually 696 in :data:`sys.path_importer_cache` (despite the name, this cache actually
(...skipping 10 matching lines...) Expand all
578 entry finder` that can handle the path entry, or it may raise 707 entry finder` that can handle the path entry, or it may raise
579 :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to 708 :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to
580 signal that the hook cannot find a :term:`path entry finder` for that 709 signal that the hook cannot find a :term:`path entry finder` for that
581 :term:`path entry`. The exception is ignored and :term:`import path` 710 :term:`path entry`. The exception is ignored and :term:`import path`
582 iteration continues. The hook should expect either a string or bytes object; 711 iteration continues. The hook should expect either a string or bytes object;
583 the encoding of bytes objects is up to the hook (e.g. it may be a file system 712 the encoding of bytes objects is up to the hook (e.g. it may be a file system
584 encoding, UTF-8, or something else), and if the hook cannot decode the 713 encoding, UTF-8, or something else), and if the hook cannot decode the
585 argument, it should raise :exc:`ImportError`. 714 argument, it should raise :exc:`ImportError`.
586 715
587 If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` 716 If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder`
588 being returned, then the path based finder's :meth:`find_module()` method 717 being returned, then the path based finder's :meth:`find_spec()` method
589 will store ``None`` in :data:`sys.path_importer_cache` (to indicate that 718 will store ``None`` in :data:`sys.path_importer_cache` (to indicate that
590 there is no finder for this path entry) and return ``None``, indicating that 719 there is no finder for this path entry) and return ``None``, indicating that
591 this :term:`meta path finder` could not find the module. 720 this :term:`meta path finder` could not find the module.
592 721
593 If a :term:`path entry finder` *is* returned by one of the :term:`path entry 722 If a :term:`path entry finder` *is* returned by one of the :term:`path entry
594 hook` callables on :data:`sys.path_hooks`, then the following protocol is used 723 hook` callables on :data:`sys.path_hooks`, then the following protocol is used
595 to ask the finder for a module loader, which is then used to load the module. 724 to ask the finder for a module spec, which is then used when loading the
596 725 module.
597 726
598 Path entry finder protocol 727 Path entry finder protocol
599 -------------------------- 728 --------------------------
600 729
601 In order to support imports of modules and initialized packages and also to 730 In order to support imports of modules and initialized packages and also to
602 contribute portions to namespace packages, path entry finders must implement 731 contribute portions to namespace packages, path entry finders must implement
603 the :meth:`find_loader()` method. 732 the :meth:`find_spec()` method.
733
734 :meth:`find_spec()` takes one argument, the fully qualified name of the
735 module being imported. :meth:`find_spec()` returns a fully populated
736 spec for the module. This spec will always have "loader" set (with one
737 exception).
738
739 To indicate to the import machinery that the spec represents a namespace
740 :term:`portion`. the path entry finder sets "loader" on the spec to
741 ``None`` and "submodule_search_locations" to a list containing the
742 portion.
743
744 Legacy path entry finder protocols
brett.cannon 2013/10/25 17:37:14 Make this a versionchanged comment
eric.snow 2013/10/26 23:35:03 Done.
745 ----------------------------------
746
747 Older path entry finders may implement one of two deprecated methods
748 instead of :meth:`find_spec()`: :meth:`find_loader()` and
749 :meth:`find_module()`. The methods are still respected for the sake of
750 backward compatibility. Howevever, if find_spec() is implemented on the
751 path entry finder, the legacy methods are ignored.
604 752
605 :meth:`find_loader()` takes one argument, the fully qualified name of the 753 :meth:`find_loader()` takes one argument, the fully qualified name of the
606 module being imported. :meth:`find_loader()` returns a 2-tuple where the 754 module being imported. :meth:`find_loader()` returns a 2-tuple where the
607 first item is the loader and the second item is a namespace :term:`portion`. 755 first item is the loader and the second item is a namespace :term:`portion`.
608 When the first item (i.e. the loader) is ``None``, this means that while the 756 When the first item (i.e. the loader) is ``None``, this means that while the
609 path entry finder does not have a loader for the named module, it knows that the 757 path entry finder does not have a loader for the named module, it knows that the
610 path entry contributes to a namespace portion for the named module. This will 758 path entry contributes to a namespace portion for the named module. This will
611 almost always be the case where Python is asked to import a namespace package 759 almost always be the case where Python is asked to import a namespace package
612 that has no physical presence on the file system. When a path entry finder 760 that has no physical presence on the file system. When a path entry finder
613 returns ``None`` for the loader, the second item of the 2-tuple return value 761 returns ``None`` for the loader, the second item of the 2-tuple return value
614 must be a sequence, although it can be empty. 762 must be a sequence, although it can be empty.
615 763
616 If :meth:`find_loader()` returns a non-``None`` loader value, the portion is 764 If :meth:`find_loader()` returns a non-``None`` loader value, the portion is
617 ignored and the loader is returned from the path based finder, terminating 765 ignored and the loader is returned from the path based finder, terminating
618 the search through the path entries. 766 the search through the path entries.
619 767
620 For backwards compatibility with other implementations of the import 768 For backwards compatibility with other implementations of the import
621 protocol, many path entry finders also support the same, 769 protocol, many path entry finders also support the same,
622 traditional :meth:`find_module()` method that meta path finders support. 770 traditional :meth:`find_module()` method that meta path finders support.
623 However path entry finder :meth:`find_module()` methods are never called 771 However path entry finder :meth:`find_module()` methods are never called
624 with a ``path`` argument (they are expected to record the appropriate 772 with a ``path`` argument (they are expected to record the appropriate
625 path information from the initial call to the path hook). 773 path information from the initial call to the path hook).
626 774
627 The :meth:`find_module()` method on path entry finders is deprecated, 775 The :meth:`find_module()` method on path entry finders is deprecated,
628 as it does not allow the path entry finder to contribute portions to 776 as it does not allow the path entry finder to contribute portions to
629 namespace packages. Instead path entry finders should implement the 777 namespace packages. If both :meth:`find_loader()` and :meth:`find_module()`
630 :meth:`find_loader()` method as described above. If it exists on the path 778 exist on a path entry finder, the import system will always call
631 entry finder, the import system will always call :meth:`find_loader()` 779 :meth:`find_loader()` in preference to :meth:`find_module()`.
632 in preference to :meth:`find_module()`.
633 780
634 781
635 Replacing the standard import system 782 Replacing the standard import system
636 ==================================== 783 ====================================
637 784
638 The most reliable mechanism for replacing the entire import system is to 785 The most reliable mechanism for replacing the entire import system is to
639 delete the default contents of :data:`sys.meta_path`, replacing them 786 delete the default contents of :data:`sys.meta_path`, replacing them
640 entirely with a custom meta path hook. 787 entirely with a custom meta path hook.
641 788
642 If it is acceptable to only alter the behaviour of import statements 789 If it is acceptable to only alter the behaviour of import statements
643 without affecting other APIs that access the import system, then replacing 790 without affecting other APIs that access the import system, then replacing
644 the builtin :func:`__import__` function may be sufficient. This technique 791 the builtin :func:`__import__` function may be sufficient. This technique
645 may also be employed at the module level to only alter the behaviour of 792 may also be employed at the module level to only alter the behaviour of
646 import statements within that module. 793 import statements within that module.
647 794
648 To selectively prevent import of some modules from a hook early on the 795 To selectively prevent import of some modules from a hook early on the
649 meta path (rather than disabling the standard import system entirely), 796 meta path (rather than disabling the standard import system entirely),
650 it is sufficient to raise :exc:`ImportError` directly from 797 it is sufficient to raise :exc:`ImportError` directly from
651 :meth:`find_module` instead of returning ``None``. The latter indicates 798 :meth:`find_spec` instead of returning ``None``. The latter indicates
652 that the meta path search should continue. while raising an exception 799 that the meta path search should continue. while raising an exception
653 terminates it immediately. 800 terminates it immediately.
654 801
655 802
656 Open issues 803 Open issues
657 =========== 804 ===========
658 805
659 XXX It would be really nice to have a diagram. 806 XXX It would be really nice to have a diagram.
660 807
661 XXX * (import_machinery.rst) how about a section devoted just to the 808 XXX * (import_machinery.rst) how about a section devoted just to the
(...skipping 21 matching lines...) Expand all
683 830
684 :pep:`366` describes the addition of the ``__package__`` attribute for 831 :pep:`366` describes the addition of the ``__package__`` attribute for
685 explicit relative imports in main modules. 832 explicit relative imports in main modules.
686 833
687 :pep:`328` introduced absolute and explicit relative imports and initially 834 :pep:`328` introduced absolute and explicit relative imports and initially
688 proposed ``__name__`` for semantics :pep:`366` would eventually specify for 835 proposed ``__name__`` for semantics :pep:`366` would eventually specify for
689 ``__package__``. 836 ``__package__``.
690 837
691 :pep:`338` defines executing modules as scripts. 838 :pep:`338` defines executing modules as scripts.
692 839
840 :pep:`451` adds the encapsulation of per-module import state in spec
841 objects. At also off-loads most of the boilerplate responsibilities of
842 loaders back onto the import machinery. These changes allow the
843 deprecation of several APIs in the import system and also addition of new
844 methods to finders and loaders.
693 845
694 .. rubric:: Footnotes 846 .. rubric:: Footnotes
695 847
696 .. [#fnmo] See :class:`types.ModuleType`. 848 .. [#fnmo] See :class:`types.ModuleType`.
697 849
698 .. [#fnlo] The importlib implementation avoids using the return value 850 .. [#fnlo] The importlib implementation avoids using the return value
699 directly. Instead, it gets the module object by looking the module name up 851 directly. Instead, it gets the module object by looking the module name up
700 in :data:`sys.modules`. The indirect effect of this is that an imported 852 in :data:`sys.modules`. The indirect effect of this is that an imported
701 module may replace itself in :data:`sys.modules`. This is 853 module may replace itself in :data:`sys.modules`. This is
702 implementation-specific behavior that is not guaranteed to work in other 854 implementation-specific behavior that is not guaranteed to work in other
703 Python implementations. 855 Python implementations.
704 856
705 .. [#fnpic] In legacy code, it is possible to find instances of 857 .. [#fnpic] In legacy code, it is possible to find instances of
706 :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It 858 :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It
707 is recommended that code be changed to use ``None`` instead. See 859 is recommended that code be changed to use ``None`` instead. See
708 :ref:`portingpythoncode` for more details. 860 :ref:`portingpythoncode` for more details.
OLDNEW
« no previous file with comments | « no previous file | no next file » | no next file with comments »

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+