Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(6737)

Side by Side Diff: Doc/reference/import.rst

Issue 18864: Implementation for PEP 451 (importlib.machinery.ModuleSpec)
Patch Set: Created 5 years, 6 months ago
Left:
Right:
Use n/p to move between diff chunks; N/P to move between comments. Please Sign in to add in-line comments.
Jump to:
View unified diff | Download patch
OLDNEW
1 1
2 .. _importsystem: 2 .. _importsystem:
3 3
4 ***************** 4 *****************
5 The import system 5 The import system
6 ***************** 6 *****************
7 7
8 .. index:: single: import machinery 8 .. index:: single: import machinery
9 9
10 Python code in one :term:`module` gains access to the code in another module 10 Python code in one :term:`module` gains access to the code in another module
(...skipping 192 matching lines...) Expand 10 before | Expand all | Expand 10 after
203 :func:`imp.reload` will reuse the *same* module object, and simply 203 :func:`imp.reload` will reuse the *same* module object, and simply
204 reinitialise the module contents by rerunning the module's code. 204 reinitialise the module contents by rerunning the module's code.
205 205
206 206
207 Finders and loaders 207 Finders and loaders
208 ------------------- 208 -------------------
209 209
210 .. index:: 210 .. index::
211 single: finder 211 single: finder
212 single: loader 212 single: loader
213 single: module spec
213 214
214 If the named module is not found in :data:`sys.modules`, then Python's import 215 If the named module is not found in :data:`sys.modules`, then Python's import
215 protocol is invoked to find and load the module. This protocol consists of 216 protocol is invoked to find and load the module. This protocol consists of
216 two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`. 217 two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`.
217 A finder's job is to determine whether it can find the named module using 218 A finder's job is to determine whether it can find the named module using
218 whatever strategy it knows about. Objects that implement both of these 219 whatever strategy it knows about. Objects that implement both of these
219 interfaces are referred to as :term:`importers <importer>` - they return 220 interfaces are referred to as :term:`importers <importer>` - they return
220 themselves when they find that they can load the requested module. 221 themselves when they find that they can load the requested module.
221 222
222 Python includes a number of default finders and importers. The first one 223 Python includes a number of default finders and importers. The first one
223 knows how to locate built-in modules, and the second knows how to locate 224 knows how to locate built-in modules, and the second knows how to locate
224 frozen modules. A third default finder searches an :term:`import path` 225 frozen modules. A third default finder searches an :term:`import path`
225 for modules. The :term:`import path` is a list of locations that may 226 for modules. The :term:`import path` is a list of locations that may
226 name file system paths or zip files. It can also be extended to search 227 name file system paths or zip files. It can also be extended to search
227 for any locatable resource, such as those identified by URLs. 228 for any locatable resource, such as those identified by URLs.
228 229
229 The import machinery is extensible, so new finders can be added to extend the 230 The import machinery is extensible, so new finders can be added to extend the
230 range and scope of module searching. 231 range and scope of module searching.
231 232
232 Finders do not actually load modules. If they can find the named module, they 233 Finders do not actually load modules. If they can find the named module, they
233 return a :term:`loader`, which the import machinery then invokes to load the 234 return a :term:`module spec`, an encapsulation of the module's import-related
234 module and create the corresponding module object. 235 information, which the import machinery then uses when loading the module.
235 236
236 The following sections describe the protocol for finders and loaders in more 237 The following sections describe the protocol for finders and loaders in more
237 detail, including how you can create and register new ones to extend the 238 detail, including how you can create and register new ones to extend the
238 import machinery. 239 import machinery.
239 240
241 .. versionchanged:: 3.4
242 In previous versions of Python, finders returned :term:`loaders <loader>`
243 directly, whereas now they return module specs which *contain* loaders.
244 Loaders are still used during import but have fewer responsibilities.
240 245
241 Import hooks 246 Import hooks
242 ------------ 247 ------------
243 248
244 .. index:: 249 .. index::
245 single: import hooks 250 single: import hooks
246 single: meta hooks 251 single: meta hooks
247 single: path hooks 252 single: path hooks
248 pair: hooks; import 253 pair: hooks; import
249 pair: hooks; meta 254 pair: hooks; meta
(...skipping 13 matching lines...) Expand all
263 ``package.__path__``) processing, at the point where their associated path 268 ``package.__path__``) processing, at the point where their associated path
264 item is encountered. Import path hooks are registered by adding new callables 269 item is encountered. Import path hooks are registered by adding new callables
265 to :data:`sys.path_hooks` as described below. 270 to :data:`sys.path_hooks` as described below.
266 271
267 272
268 The meta path 273 The meta path
269 ------------- 274 -------------
270 275
271 .. index:: 276 .. index::
272 single: sys.meta_path 277 single: sys.meta_path
273 pair: finder; find_module 278 pair: finder; find_spec
274 pair: finder; find_loader
275 279
276 When the named module is not found in :data:`sys.modules`, Python next 280 When the named module is not found in :data:`sys.modules`, Python next
277 searches :data:`sys.meta_path`, which contains a list of meta path finder 281 searches :data:`sys.meta_path`, which contains a list of meta path finder
278 objects. These finders are queried in order to see if they know how to handle 282 objects. These finders are queried in order to see if they know how to handle
279 the named module. Meta path finders must implement a method called 283 the named module. Meta path finders must implement a method called
280 :meth:`find_module()` which takes two arguments, a name and an import path. 284 :meth:`find_spec()` which takes two arguments, a name and an import path.
281 The meta path finder can use any strategy it wants to determine whether it can 285 The meta path finder can use any strategy it wants to determine whether it can
282 handle the named module or not. 286 handle the named module or not.
283 287
284 If the meta path finder knows how to handle the named module, it returns a 288 If the meta path finder knows how to handle the named module, it returns a
285 loader object. If it cannot handle the named module, it returns ``None``. If 289 spec object. If it cannot handle the named module, it returns ``None``. If
286 :data:`sys.meta_path` processing reaches the end of its list without returning 290 :data:`sys.meta_path` processing reaches the end of its list without returning
287 a loader, then an :exc:`ImportError` is raised. Any other exceptions raised 291 a spec, then an :exc:`ImportError` is raised. Any other exceptions raised
288 are simply propagated up, aborting the import process. 292 are simply propagated up, aborting the import process.
289 293
290 The :meth:`find_module()` method of meta path finders is called with two 294 The :meth:`find_spec()` method of meta path finders is called with two
291 arguments. The first is the fully qualified name of the module being 295 arguments. The first is the fully qualified name of the module being
292 imported, for example ``foo.bar.baz``. The second argument is the path 296 imported, for example ``foo.bar.baz``. The second argument is the path
293 entries to use for the module search. For top-level modules, the second 297 entries to use for the module search. For top-level modules, the second
294 argument is ``None``, but for submodules or subpackages, the second 298 argument is ``None``, but for submodules or subpackages, the second
295 argument is the value of the parent package's ``__path__`` attribute. If 299 argument is the value of the parent package's ``__path__`` attribute. If
296 the appropriate ``__path__`` attribute cannot be accessed, an 300 the appropriate ``__path__`` attribute cannot be accessed, an
297 :exc:`ImportError` is raised. 301 :exc:`ImportError` is raised.
298 302
299 The meta path may be traversed multiple times for a single import request. 303 The meta path may be traversed multiple times for a single import request.
300 For example, assuming none of the modules involved has already been cached, 304 For example, assuming none of the modules involved has already been cached,
301 importing ``foo.bar.baz`` will first perform a top level import, calling 305 importing ``foo.bar.baz`` will first perform a top level import, calling
302 ``mpf.find_module("foo", None)`` on each meta path finder (``mpf``). After 306 ``mpf.find_spec("foo", None)`` on each meta path finder (``mpf``). After
303 ``foo`` has been imported, ``foo.bar`` will be imported by traversing the 307 ``foo`` has been imported, ``foo.bar`` will be imported by traversing the
304 meta path a second time, calling 308 meta path a second time, calling
305 ``mpf.find_module("foo.bar", foo.__path__)``. Once ``foo.bar`` has been 309 ``mpf.find_spec("foo.bar", foo.__path__)``. Once ``foo.bar`` has been
306 imported, the final traversal will call 310 imported, the final traversal will call
307 ``mpf.find_module("foo.bar.baz", foo.bar.__path__)``. 311 ``mpf.find_spec("foo.bar.baz", foo.bar.__path__)``.
308 312
309 Some meta path finders only support top level imports. These importers will 313 Some meta path finders only support top level imports. These importers will
310 always return ``None`` when anything other than ``None`` is passed as the 314 always return ``None`` when anything other than ``None`` is passed as the
311 second argument. 315 second argument.
312 316
313 Python's default :data:`sys.meta_path` has three meta path finders, one that 317 Python's default :data:`sys.meta_path` has three meta path finders, one that
314 knows how to import built-in modules, one that knows how to import frozen 318 knows how to import built-in modules, one that knows how to import frozen
315 modules, and one that knows how to import modules from an :term:`import path` 319 modules, and one that knows how to import modules from an :term:`import path`
316 (i.e. the :term:`path based finder`). 320 (i.e. the :term:`path based finder`).
317 321
318 322 .. versionchanged:: 3.4
319 Loaders 323 The find_spec() method of meta path finders replaced :meth:`find_module()`.
324 which is now deprecated. While it will continue to work without change,
325 the import machinery will try it only if the finder does not implement
326 find_spec().
327
328
329 Loading
320 ======= 330 =======
321 331
322 If and when a module loader is found its 332 If and when a module spec is found, the import machinery will use it (and
323 :meth:`~importlib.abc.Loader.load_module` method is called, with a single 333 the loader it contains) when loading the module. Here is an approximation
324 argument, the fully qualified name of the module being imported. This method 334 of what happens during the loading portion of import::
325 has several responsibilities, and should return the module object it has 335
326 loaded [#fnlo]_. If it cannot load the module, it should raise an 336 module = None
327 :exc:`ImportError`, although any other exception raised during 337 if spec.loader is not None and hasattr(spec.loader, 'create_module'):
328 :meth:`load_module()` will be propagated. 338 module = spec.loader.create_module(spec)
329 339 if module is None:
330 In many cases, the finder and loader can be the same object; in such cases the 340 module = ModuleType(spec.name)
331 :meth:`finder.find_module()` would just return ``self``. 341 # The import-related module attributes get set here:
332 342 _init_module_attrs(spec, module)
333 Loaders must satisfy the following requirements: 343
344 if spec.loader is None and spec.submodule_search_locations is not None:
345 # namespace package
346 sys.modules[spec.name] = module
347 elif not hasattr(spec.loader, 'exec_module'):
Jim.J.Jewett 2013/10/30 03:32:01 So if spec.loader is None *but* spec.submodule_sea
eric.snow 2013/11/01 01:07:38 I'm glad you caught that. This happens in a coupl
348 module = spec.loader.load_module(spec.name)
349 else:
350 sys.modules[spec.name] = module
351 try:
352 spec.loader.exec_module(module)
353 except BaseException:
354 try:
355 del sys.modules[spec.name]
356 except KeyError:
357 pass
358 raise
359 module_to_return = sys.modules[spec.name]
360
361 Note the following details:
334 362
335 * If there is an existing module object with the given name in 363 * If there is an existing module object with the given name in
336 :data:`sys.modules`, the loader must use that existing module. (Otherwise, 364 :data:`sys.modules`, import will have already returned it.
337 :func:`imp.reload` will not work correctly.) If the named module does 365
338 not exist in :data:`sys.modules`, the loader must create a new module 366 * The module will exist in :data:`sys.modules` before the loader
339 object and add it to :data:`sys.modules`.
340
341 Note that the module *must* exist in :data:`sys.modules` before the loader
342 executes the module code. This is crucial because the module code may 367 executes the module code. This is crucial because the module code may
343 (directly or indirectly) import itself; adding it to :data:`sys.modules` 368 (directly or indirectly) import itself; adding it to :data:`sys.modules`
344 beforehand prevents unbounded recursion in the worst case and multiple 369 beforehand prevents unbounded recursion in the worst case and multiple
345 loading in the best. 370 loading in the best.
346 371
347 If loading fails, the loader must remove any modules it has inserted into 372 * If loading fails, the failing module -- and only the failing module --
348 :data:`sys.modules`, but it must remove **only** the failing module, and 373 gets removed from :data:`sys.modules`. Any module already in the
349 only if the loader itself has loaded it explicitly. Any module already in 374 :data:`sys.modules` cache, and any module that was successfully loaded
350 the :data:`sys.modules` cache, and any module that was successfully loaded 375 as a side-effect, must remain in the cache. This contrasts with
351 as a side-effect, must remain in the cache. 376 reloading where even the failing module is left in :data:`sys.modules`.
352 377
353 * The loader may set the ``__file__`` attribute of the module. If set, this 378 * After the module is created but before execution, the import machinery
354 attribute's value must be a string. The loader may opt to leave 379 sets the import-related module attributes ("init_module_attrs"), as
355 ``__file__`` unset if it has no semantic meaning (e.g. a module loaded from 380 summarized in a `later section <Import-related module attributes>`_.
356 a database). If ``__file__`` is set, it may also be appropriate to set the 381
357 ``__cached__`` attribute which is the path to any compiled version of the 382 * Module execution is the key moment of loading in which the module's
358 code (e.g. byte-compiled file). The file does not need to exist to set this 383 namespace gets populated. Execution is entirely delegated to the
359 attribute; the path can simply point to whether the compiled file would 384 loader, which gets to decide what gets populated and how.
360 exist (see :pep:`3147`). 385
361 386 * The module created during loading and passed to exec_module() may
362 * The loader may set the ``__name__`` attribute of the module. While not 387 not be the one returned at the end of import [#fnlo]_.
363 required, setting this attribute is highly recommended so that the 388
364 :meth:`repr()` of the module is more informative. 389 .. versionchanged:: 3.4
365 390 The import system has taken over the boilerplate responsibilities of
366 * If the module is a package (either regular or namespace), the loader must 391 loaders. These were previously performed by the :meth:`load_module()`
367 set the module object's ``__path__`` attribute. The value must be 392 method.
368 iterable, but may be empty if ``__path__`` has no further significance 393
369 to the loader. If ``__path__`` is not empty, it must produce strings 394 Loaders
370 when iterated over. More details on the semantics of ``__path__`` are 395 -------
371 given :ref:`below <package-path-rules>`. 396
372 397 Module loaders provide the critical function of loading: module execution.
373 * The ``__loader__`` attribute must be set to the loader object that loaded 398 The import machinery calls the :meth:`~importlib.abc.Loader.exec_module()`
374 the module. This is mostly for introspection and reloading, but can be 399 method with a single argument, the module object to execute. Any value
375 used for additional loader-specific functionality, for example getting 400 returned from exec_module() is ignored.
376 data associated with a loader. If the attribute is missing or set to ``None`` 401
377 then the import machinery will automatically set it **after** the module has 402 Loaders must satisfy the following requirements:
378 been imported.
379
380 * The module's ``__package__`` attribute must be set. Its value must be a
381 string, but it can be the same value as its ``__name__``. If the attribute
382 is set to ``None`` or is missing, the import system will fill it in with a
383 more appropriate value **after** the module has been imported.
384 When the module is a package, its ``__package__`` value should be set to its
385 ``__name__``. When the module is not a package, ``__package__`` should be
386 set to the empty string for top-level modules, or for submodules, to the
387 parent package's name. See :pep:`366` for further details.
388
389 This attribute is used instead of ``__name__`` to calculate explicit
390 relative imports for main modules, as defined in :pep:`366`.
391 403
392 * If the module is a Python module (as opposed to a built-in module or a 404 * If the module is a Python module (as opposed to a built-in module or a
393 dynamically loaded extension), the loader should execute the module's code 405 dynamically loaded extension), the loader should execute the module's code
394 in the module's global name space (``module.__dict__``). 406 in the module's global name space (``module.__dict__``).
395 407
396 408 * If loader cannot execute the module, it should raise an
397 Module reprs 409 :exc:`ImportError`, although any other exception raised during
398 ------------ 410 :meth:`exec_module()` will be propagated.
399 411
400 By default, all modules have a usable repr, however depending on the 412 In many cases, the finder and loader can be the same object; in such cases the
401 attributes set above, and hooks in the loader, you can more explicitly control 413 :meth:`finder.find_spec()` would just return a spec with the loader set
402 the repr of module objects. 414 to ``self``.
403 415
404 Loaders may implement a :meth:`module_repr()` method which takes a single 416 Module loaders may opt in to creating the module object during loading
405 argument, the module object. When ``repr(module)`` is called for a module 417 by implementing a :meth:`create_module()` method. It takes one argument,
406 with a loader supporting this protocol, whatever is returned from 418 the module spec, and returns the new module object to use during loading.
407 ``module.__loader__.module_repr(module)`` is returned as the module's repr 419 create_module() does not need to set any attributes on the module object.
408 without further processing. This return value must be a string. 420 If the loader does not define create_module(), the import machinery will
409 421 create the new module itself.
410 If the module has no ``__loader__`` attribute, or the loader has no 422
411 :meth:`module_repr()` method, then the module object implementation itself 423 .. versionadded:: 3.4
412 will craft a default repr using whatever information is available. It will 424 The create_module() method of loaders.
413 try to use the ``module.__name__``, ``module.__file__``, and 425
414 ``module.__loader__`` as input into the repr, with defaults for whatever 426 .. versionchanged:: 3.4
415 information is missing. 427 The load_module() method was replaced by exec_module() and the import
416 428 machinery assumed all the boilerplate responsibilities of loading.
417 Here are the exact rules used: 429
418 430 For compatibility with existing loaders, the import machinery will use
419 * If the module has a ``__loader__`` and that loader has a 431 the :meth:`~importlib.abc.Loader.load_module()` method of loaders if it
420 :meth:`module_repr()` method, call it with a single argument, which is the 432 exists and the loader does not also implement exec_module(). However,
421 module object. The value returned is used as the module's repr. 433 load_module() has been deprecated and loaders should implement
422 434 exec_module() instead.
423 * If an exception occurs in :meth:`module_repr()`, the exception is caught 435
424 and discarded, and the calculation of the module's repr continues as if 436 The load_module() method must implement all the boilerplate loading
425 :meth:`module_repr()` did not exist. 437 functionality described above in addition to executing the module. All
426 438 the same constraints apply, with some additional clarification:
427 * If the module has a ``__file__`` attribute, this is used as part of the 439
428 module's repr. 440 * If there is an existing module object with the given name in
429 441 :data:`sys.modules`, the loader must use that existing module.
430 * If the module has no ``__file__`` but does have a ``__loader__`` that is not 442 (Otherwise, :func:`imp.reload` will not work correctly.) If the
431 ``None``, then the loader's repr is used as part of the module's repr. 443 named module does not exist in :data:`sys.modules`, the loader
432 444 must create a new module object and add it to :data:`sys.modules`.
433 * Otherwise, just use the module's ``__name__`` in the repr. 445
434 446 * The module *must* exist in :data:`sys.modules` before the loader
435 This example, from :pep:`420` shows how a loader can craft its own module 447 executes the module code, to prevent unbounded recursion or multiple
436 repr:: 448 loading.
437 449
438 class NamespaceLoader: 450 * If loading fails, the loader must remove any modules it has inserted
439 @classmethod 451 into :data:`sys.modules`, but it must remove **only** the failing
440 def module_repr(cls, module): 452 module, and only if the loader itself has loaded it explicitly.
441 return "<module '{}' (namespace)>".format(module.__name__) 453
442 454 Module spec
455 -----------
456
457 The import machinery uses a variety of information about each module
458 during import, especially before loading. Most of the information is
459 common to all modules. The purpose of a module's spec is to encapsulate
460 this import-related information on a per-module basis.
461
462 Using a spec during import allows state to be transferred between import
463 system components, e.g. between the finder that creates the module spec
464 and the loader that executes it. Most importantly, it allows the
465 import machinery to perform the boilerplate operations of loading,
466 whereas without a module spec the loader had that responsibility.
467
468 See :class:`~importlib.machinery.ModuleSpec` for more specifics on what
469 information a module's spec may hold.
470
471 .. versionadded:: 3.4
472
473 Import-related module attributes
474 --------------------------------
475
476 The import machinery fills in these attributes on each module object
477 during loading, based on the module's spec, before the loader executes
478 the module.
479
480 .. attribute:: __name__
481
482 The ``__name__`` attribute must be set to the fully-qualified name of
483 the module. This name is used to uniquely identify the module in
484 the import system.
485
486 .. attribute:: __loader__
487
488 The ``__loader__`` attribute must be set to the loader object that
489 the import machinery used when loading the module. This is mostly
490 for introspection, but can be used for additional loader-specific
491 functionality, for example getting data associated with a loader.
492
493 .. attribute:: __package__
494
495 The module's ``__package__`` attribute must be set. Its value must
496 be a string, but it can be the same value as its ``__name__``. When
497 the module is a package, its ``__package__`` value should be set to
498 its ``__name__``. When the module is not a package, ``__package__``
499 should be set to the empty string for top-level modules, or for
500 submodules, to the parent package's name. See :pep:`366` for further
501 details.
502
503 This attribute is used instead of ``__name__`` to calculate explicit
504 relative imports for main modules, as defined in :pep:`366`.
505
506 .. attribute:: __spec__
507
508 The ``__spec__`` attribute must be set to the module spec that was
509 used when importing the module. This is used primarily for
510 introspection and during reloading.
511
512 .. attribute:: __path__
513
514 If the module is a package (either regular or namespace), the module
515 object's ``__path__`` attribute must be set. The value must be
516 iterable, but may be empty if ``__path__`` has no further significance.
517 If ``__path__`` is not empty, it must produce strings when iterated
518 over. More details on the semantics of ``__path__`` are given
519 :ref:`below <package-path-rules>`.
520
521 Non-package modules should not have a ``__path__`` attribute.
522
523 .. attribute:: __file__
524 .. attribute:: __cached__
525
526 ``__file__`` is optional. If set, this attribute's value must be a
527 string. The import system may opt to leave ``__file__`` unset if it
528 has no semantic meaning (e.g. a module loaded from a database). If
529 ``__file__`` is set, it may also be appropriate to set the
530 ``__cached__`` attribute which is the path to any compiled version of
531 the code (e.g. byte-compiled file). The file does not need to exist
532 to set this attribute; the path can simply point to where the
533 compiled file would exist (see :pep:`3147`).
Jim.J.Jewett 2013/10/30 03:32:01 This implies that the __cached__ attribute is unde
eric.snow 2013/11/01 01:07:38 I don't know of any reason why someone couldn't se
443 534
444 .. _package-path-rules: 535 .. _package-path-rules:
445 536
446 module.__path__ 537 module.__path__
447 --------------- 538 ---------------
448 539
449 By definition, if a module has an ``__path__`` attribute, it is a package, 540 By definition, if a module has an ``__path__`` attribute, it is a package,
450 regardless of its value. 541 regardless of its value.
451 542
452 A package's ``__path__`` attribute is used during imports of its subpackages. 543 A package's ``__path__`` attribute is used during imports of its subpackages.
453 Within the import machinery, it functions much the same as :data:`sys.path`, 544 Within the import machinery, it functions much the same as :data:`sys.path`,
454 i.e. providing a list of locations to search for modules during import. 545 i.e. providing a list of locations to search for modules during import.
455 However, ``__path__`` is typically much more constrained than 546 However, ``__path__`` is typically much more constrained than
456 :data:`sys.path`. 547 :data:`sys.path`.
457 548
458 ``__path__`` must be an iterable of strings, but it may be empty. 549 ``__path__`` must be an iterable of strings, but it may be empty.
459 The same rules used for :data:`sys.path` also apply to a package's 550 The same rules used for :data:`sys.path` also apply to a package's
460 ``__path__``, and :data:`sys.path_hooks` (described below) are 551 ``__path__``, and :data:`sys.path_hooks` (described below) are
461 consulted when traversing a package's ``__path__``. 552 consulted when traversing a package's ``__path__``.
462 553
463 A package's ``__init__.py`` file may set or alter the package's ``__path__`` 554 A package's ``__init__.py`` file may set or alter the package's ``__path__``
464 attribute, and this was typically the way namespace packages were implemented 555 attribute, and this was typically the way namespace packages were implemented
465 prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no 556 prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no
466 longer need to supply ``__init__.py`` files containing only ``__path__`` 557 longer need to supply ``__init__.py`` files containing only ``__path__``
467 manipulation code; the namespace loader automatically sets ``__path__`` 558 manipulation code; the import machinery automatically sets ``__path__``
468 correctly for the namespace package. 559 correctly for the namespace package.
560
561 Module reprs
562 ------------
563
564 By default, all modules have a usable repr, however depending on the
565 attributes set above, and in the module's spec, you can more explicitly
566 control the repr of module objects.
567
568 If the module has a spec (``__spec__``), the import machinery will try
569 to generate a repr from it. If that fails or there is no spec, the import
570 system will craft a default repr using whatever information is available
571 on the module. It will try to use the ``module.__name__``,
572 ``module.__file__``, and ``module.__loader__`` as input into the repr,
573 with defaults for whatever information is missing.
574
575 Here are the exact rules used:
576
577 * If the module has a ``__spec__`` attribute, the information in the spec
578 is used to generate the repr. The "name", "loader", "origin", and
579 "has_location" attributes are consulted.
580
581 * If the module has a ``__file__`` attribute, this is used as part of the
582 module's repr.
583
584 * If the module has no ``__file__`` but does have a ``__loader__`` that is not
585 ``None``, then the loader's repr is used as part of the module's repr.
586
587 * Otherwise, just use the module's ``__name__`` in the repr.
588
589 .. versionchanged:: 3.4
590 Use of loader.module_repr() has been deprecated and the module spec
591 is now used by the import machinery to generate a module repr.
592
593 For backward compatibility with Python 3.3, the module repr will be
594 generated by calling the loader's :meth:`module_repr()` method, if
595 defined, before trying either approach described above. However, the
596 method is deprecated.
469 597
470 598
471 The Path Based Finder 599 The Path Based Finder
472 ===================== 600 =====================
473 601
474 .. index:: 602 .. index::
475 single: path based finder 603 single: path based finder
476 604
477 As mentioned previously, Python comes with several default meta path finders. 605 As mentioned previously, Python comes with several default meta path finders.
478 One of these, called the :term:`path based finder`, searches an :term:`import 606 One of these, called the :term:`path based finder`, searches an :term:`import
(...skipping 45 matching lines...) Expand 10 before | Expand all | Expand 10 after
524 single: sys.path_hooks 652 single: sys.path_hooks
525 single: sys.path_importer_cache 653 single: sys.path_importer_cache
526 single: PYTHONPATH 654 single: PYTHONPATH
527 655
528 The :term:`path based finder` is responsible for finding and loading Python 656 The :term:`path based finder` is responsible for finding and loading Python
529 modules and packages whose location is specified with a string :term:`path 657 modules and packages whose location is specified with a string :term:`path
530 entry`. Most path entries name locations in the file system, but they need 658 entry`. Most path entries name locations in the file system, but they need
531 not be limited to this. 659 not be limited to this.
532 660
533 As a meta path finder, the :term:`path based finder` implements the 661 As a meta path finder, the :term:`path based finder` implements the
534 :meth:`find_module()` protocol previously described, however it exposes 662 :meth:`find_spec()` protocol previously described, however it exposes
535 additional hooks that can be used to customize how modules are found and 663 additional hooks that can be used to customize how modules are found and
536 loaded from the :term:`import path`. 664 loaded from the :term:`import path`.
537 665
538 Three variables are used by the :term:`path based finder`, :data:`sys.path`, 666 Three variables are used by the :term:`path based finder`, :data:`sys.path`,
539 :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__`` 667 :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__``
540 attributes on package objects are also used. These provide additional ways 668 attributes on package objects are also used. These provide additional ways
541 that the import machinery can be customized. 669 that the import machinery can be customized.
542 670
543 :data:`sys.path` contains a list of strings providing search locations for 671 :data:`sys.path` contains a list of strings providing search locations for
544 modules and packages. It is initialized from the :data:`PYTHONPATH` 672 modules and packages. It is initialized from the :data:`PYTHONPATH`
545 environment variable and various other installation- and 673 environment variable and various other installation- and
546 implementation-specific defaults. Entries in :data:`sys.path` can name 674 implementation-specific defaults. Entries in :data:`sys.path` can name
547 directories on the file system, zip files, and potentially other "locations" 675 directories on the file system, zip files, and potentially other "locations"
548 (see the :mod:`site` module) that should be searched for modules, such as 676 (see the :mod:`site` module) that should be searched for modules, such as
549 URLs, or database queries. Only strings and bytes should be present on 677 URLs, or database queries. Only strings and bytes should be present on
550 :data:`sys.path`; all other data types are ignored. The encoding of bytes 678 :data:`sys.path`; all other data types are ignored. The encoding of bytes
551 entries is determined by the individual :term:`path entry finders <path entry 679 entries is determined by the individual :term:`path entry finders <path entry
552 finder>`. 680 finder>`.
553 681
554 The :term:`path based finder` is a :term:`meta path finder`, so the import 682 The :term:`path based finder` is a :term:`meta path finder`, so the import
555 machinery begins the :term:`import path` search by calling the path 683 machinery begins the :term:`import path` search by calling the path
556 based finder's :meth:`find_module()` method as described previously. When 684 based finder's :meth:`find_spec()` method as described previously. When
557 the ``path`` argument to :meth:`find_module()` is given, it will be a 685 the ``path`` argument to :meth:`find_spec()` is given, it will be a
558 list of string paths to traverse - typically a package's ``__path__`` 686 list of string paths to traverse - typically a package's ``__path__``
559 attribute for an import within that package. If the ``path`` argument 687 attribute for an import within that package. If the ``path`` argument
560 is ``None``, this indicates a top level import and :data:`sys.path` is used. 688 is ``None``, this indicates a top level import and :data:`sys.path` is used.
561 689
562 The path based finder iterates over every entry in the search path, and 690 The path based finder iterates over every entry in the search path, and
563 for each of these, looks for an appropriate :term:`path entry finder` for the 691 for each of these, looks for an appropriate :term:`path entry finder` for the
564 path entry. Because this can be an expensive operation (e.g. there may be 692 path entry. Because this can be an expensive operation (e.g. there may be
565 `stat()` call overheads for this search), the path based finder maintains 693 `stat()` call overheads for this search), the path based finder maintains
566 a cache mapping path entries to path entry finders. This cache is maintained 694 a cache mapping path entries to path entry finders. This cache is maintained
567 in :data:`sys.path_importer_cache` (despite the name, this cache actually 695 in :data:`sys.path_importer_cache` (despite the name, this cache actually
(...skipping 10 matching lines...) Expand all
578 entry finder` that can handle the path entry, or it may raise 706 entry finder` that can handle the path entry, or it may raise
579 :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to 707 :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to
580 signal that the hook cannot find a :term:`path entry finder` for that 708 signal that the hook cannot find a :term:`path entry finder` for that
581 :term:`path entry`. The exception is ignored and :term:`import path` 709 :term:`path entry`. The exception is ignored and :term:`import path`
582 iteration continues. The hook should expect either a string or bytes object; 710 iteration continues. The hook should expect either a string or bytes object;
583 the encoding of bytes objects is up to the hook (e.g. it may be a file system 711 the encoding of bytes objects is up to the hook (e.g. it may be a file system
584 encoding, UTF-8, or something else), and if the hook cannot decode the 712 encoding, UTF-8, or something else), and if the hook cannot decode the
585 argument, it should raise :exc:`ImportError`. 713 argument, it should raise :exc:`ImportError`.
586 714
587 If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` 715 If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder`
588 being returned, then the path based finder's :meth:`find_module()` method 716 being returned, then the path based finder's :meth:`find_spec()` method
589 will store ``None`` in :data:`sys.path_importer_cache` (to indicate that 717 will store ``None`` in :data:`sys.path_importer_cache` (to indicate that
590 there is no finder for this path entry) and return ``None``, indicating that 718 there is no finder for this path entry) and return ``None``, indicating that
591 this :term:`meta path finder` could not find the module. 719 this :term:`meta path finder` could not find the module.
592 720
593 If a :term:`path entry finder` *is* returned by one of the :term:`path entry 721 If a :term:`path entry finder` *is* returned by one of the :term:`path entry
594 hook` callables on :data:`sys.path_hooks`, then the following protocol is used 722 hook` callables on :data:`sys.path_hooks`, then the following protocol is used
595 to ask the finder for a module loader, which is then used to load the module. 723 to ask the finder for a module spec, which is then used when loading the
596 724 module.
597 725
598 Path entry finder protocol 726 Path entry finder protocol
599 -------------------------- 727 --------------------------
600 728
601 In order to support imports of modules and initialized packages and also to 729 In order to support imports of modules and initialized packages and also to
602 contribute portions to namespace packages, path entry finders must implement 730 contribute portions to namespace packages, path entry finders must implement
603 the :meth:`find_loader()` method. 731 the :meth:`find_spec()` method.
604 732
605 :meth:`find_loader()` takes one argument, the fully qualified name of the 733 :meth:`find_spec()` takes one argument, the fully qualified name of the
606 module being imported. :meth:`find_loader()` returns a 2-tuple where the 734 module being imported. :meth:`find_spec()` returns a fully populated
607 first item is the loader and the second item is a namespace :term:`portion`. 735 spec for the module. This spec will always have "loader" set (with one
608 When the first item (i.e. the loader) is ``None``, this means that while the 736 exception).
609 path entry finder does not have a loader for the named module, it knows that the
610 path entry contributes to a namespace portion for the named module. This will
611 almost always be the case where Python is asked to import a namespace package
612 that has no physical presence on the file system. When a path entry finder
613 returns ``None`` for the loader, the second item of the 2-tuple return value
614 must be a sequence, although it can be empty.
615 737
616 If :meth:`find_loader()` returns a non-``None`` loader value, the portion is 738 To indicate to the import machinery that the spec represents a namespace
617 ignored and the loader is returned from the path based finder, terminating 739 :term:`portion`. the path entry finder sets "loader" on the spec to
618 the search through the path entries. 740 ``None`` and "submodule_search_locations" to a list containing the
741 portion.
619 742
620 For backwards compatibility with other implementations of the import 743 .. versionchanged:: 3.4
621 protocol, many path entry finders also support the same, 744 find_spec() replaced find_loader() and find_module(), but of which
622 traditional :meth:`find_module()` method that meta path finders support. 745 are now deprecated, but will be used if find_spec() is not defined.
623 However path entry finder :meth:`find_module()` methods are never called
624 with a ``path`` argument (they are expected to record the appropriate
625 path information from the initial call to the path hook).
626 746
627 The :meth:`find_module()` method on path entry finders is deprecated, 747 Older path entry finders may implement one of these two deprecated methods
628 as it does not allow the path entry finder to contribute portions to 748 instead of :meth:`find_spec()`. The methods are still respected for the
629 namespace packages. Instead path entry finders should implement the 749 sake of backward compatibility. Howevever, if find_spec() is implemented
630 :meth:`find_loader()` method as described above. If it exists on the path 750 on the path entry finder, the legacy methods are ignored.
631 entry finder, the import system will always call :meth:`find_loader()` 751
632 in preference to :meth:`find_module()`. 752 :meth:`find_loader()` takes one argument, the fully qualified name of the
753 module being imported. :meth:`find_loader()` returns a 2-tuple where the
754 first item is the loader and the second item is a namespace :term:`portion`.
755 When the first item (i.e. the loader) is ``None``, this means that while the
756 path entry finder does not have a loader for the named module, it knows that
757 the path entry contributes to a namespace portion for the named module.
758 This will almost always be the case where Python is asked to import a
759 namespace package that has no physical presence on the file system.
760 When a path entry finder returns ``None`` for the loader, the second
761 item of the 2-tuple return value must be a sequence, although it can be
762 empty.
763
764 If :meth:`find_loader()` returns a non-``None`` loader value, the portion is
765 ignored and the loader is returned from the path based finder, terminating
766 the search through the path entries.
767
768 For backwards compatibility with other implementations of the import
769 protocol, many path entry finders also support the same,
770 traditional :meth:`find_module()` method that meta path finders support.
771 However path entry finder :meth:`find_module()` methods are never called
772 with a ``path`` argument (they are expected to record the appropriate
773 path information from the initial call to the path hook).
774
775 The :meth:`find_module()` method on path entry finders is deprecated,
776 as it does not allow the path entry finder to contribute portions to
777 namespace packages. If both :meth:`find_loader()` and :meth:`find_module()`
778 exist on a path entry finder, the import system will always call
779 :meth:`find_loader()` in preference to :meth:`find_module()`.
633 780
634 781
635 Replacing the standard import system 782 Replacing the standard import system
636 ==================================== 783 ====================================
637 784
638 The most reliable mechanism for replacing the entire import system is to 785 The most reliable mechanism for replacing the entire import system is to
639 delete the default contents of :data:`sys.meta_path`, replacing them 786 delete the default contents of :data:`sys.meta_path`, replacing them
640 entirely with a custom meta path hook. 787 entirely with a custom meta path hook.
641 788
642 If it is acceptable to only alter the behaviour of import statements 789 If it is acceptable to only alter the behaviour of import statements
643 without affecting other APIs that access the import system, then replacing 790 without affecting other APIs that access the import system, then replacing
644 the builtin :func:`__import__` function may be sufficient. This technique 791 the builtin :func:`__import__` function may be sufficient. This technique
645 may also be employed at the module level to only alter the behaviour of 792 may also be employed at the module level to only alter the behaviour of
646 import statements within that module. 793 import statements within that module.
647 794
648 To selectively prevent import of some modules from a hook early on the 795 To selectively prevent import of some modules from a hook early on the
649 meta path (rather than disabling the standard import system entirely), 796 meta path (rather than disabling the standard import system entirely),
650 it is sufficient to raise :exc:`ImportError` directly from 797 it is sufficient to raise :exc:`ImportError` directly from
651 :meth:`find_module` instead of returning ``None``. The latter indicates 798 :meth:`find_spec` instead of returning ``None``. The latter indicates
652 that the meta path search should continue. while raising an exception 799 that the meta path search should continue. while raising an exception
653 terminates it immediately. 800 terminates it immediately.
654 801
655 802
656 Open issues 803 Open issues
657 =========== 804 ===========
658 805
659 XXX It would be really nice to have a diagram. 806 XXX It would be really nice to have a diagram.
660 807
661 XXX * (import_machinery.rst) how about a section devoted just to the 808 XXX * (import_machinery.rst) how about a section devoted just to the
(...skipping 21 matching lines...) Expand all
683 830
684 :pep:`366` describes the addition of the ``__package__`` attribute for 831 :pep:`366` describes the addition of the ``__package__`` attribute for
685 explicit relative imports in main modules. 832 explicit relative imports in main modules.
686 833
687 :pep:`328` introduced absolute and explicit relative imports and initially 834 :pep:`328` introduced absolute and explicit relative imports and initially
688 proposed ``__name__`` for semantics :pep:`366` would eventually specify for 835 proposed ``__name__`` for semantics :pep:`366` would eventually specify for
689 ``__package__``. 836 ``__package__``.
690 837
691 :pep:`338` defines executing modules as scripts. 838 :pep:`338` defines executing modules as scripts.
692 839
840 :pep:`451` adds the encapsulation of per-module import state in spec
841 objects. At also off-loads most of the boilerplate responsibilities of
Jim.J.Jewett 2013/10/30 03:32:01 "At also" -> "It also"
842 loaders back onto the import machinery. These changes allow the
843 deprecation of several APIs in the import system and also addition of new
844 methods to finders and loaders.
693 845
694 .. rubric:: Footnotes 846 .. rubric:: Footnotes
695 847
696 .. [#fnmo] See :class:`types.ModuleType`. 848 .. [#fnmo] See :class:`types.ModuleType`.
697 849
698 .. [#fnlo] The importlib implementation avoids using the return value 850 .. [#fnlo] The importlib implementation avoids using the return value
699 directly. Instead, it gets the module object by looking the module name up 851 directly. Instead, it gets the module object by looking the module name up
700 in :data:`sys.modules`. The indirect effect of this is that an imported 852 in :data:`sys.modules`. The indirect effect of this is that an imported
701 module may replace itself in :data:`sys.modules`. This is 853 module may replace itself in :data:`sys.modules`. This is
702 implementation-specific behavior that is not guaranteed to work in other 854 implementation-specific behavior that is not guaranteed to work in other
703 Python implementations. 855 Python implementations.
704 856
705 .. [#fnpic] In legacy code, it is possible to find instances of 857 .. [#fnpic] In legacy code, it is possible to find instances of
706 :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It 858 :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It
707 is recommended that code be changed to use ``None`` instead. See 859 is recommended that code be changed to use ``None`` instead. See
708 :ref:`portingpythoncode` for more details. 860 :ref:`portingpythoncode` for more details.
OLDNEW
« no previous file with comments | « Doc/library/importlib.rst ('k') | Lib/importlib/abc.py » ('j') | Lib/importlib/_bootstrap.py » ('J')

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+