Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(348)

Side by Side Diff: Doc/reference/import.rst

Issue 18864: Implementation for PEP 451 (importlib.machinery.ModuleSpec)
Patch Set: Created 5 years, 6 months ago
Left:
Right:
Use n/p to move between diff chunks; N/P to move between comments. Please Sign in to add in-line comments.
Jump to:
View unified diff | Download patch
OLDNEW
1 1
2 .. _importsystem: 2 .. _importsystem:
3 3
4 ***************** 4 *****************
5 The import system 5 The import system
6 ***************** 6 *****************
7 7
8 .. index:: single: import machinery 8 .. index:: single: import machinery
9 9
10 Python code in one :term:`module` gains access to the code in another module 10 Python code in one :term:`module` gains access to the code in another module
(...skipping 192 matching lines...) Expand 10 before | Expand all | Expand 10 after
203 :func:`imp.reload` will reuse the *same* module object, and simply 203 :func:`imp.reload` will reuse the *same* module object, and simply
204 reinitialise the module contents by rerunning the module's code. 204 reinitialise the module contents by rerunning the module's code.
205 205
206 206
207 Finders and loaders 207 Finders and loaders
208 ------------------- 208 -------------------
209 209
210 .. index:: 210 .. index::
211 single: finder 211 single: finder
212 single: loader 212 single: loader
213 single: module spec
213 214
214 If the named module is not found in :data:`sys.modules`, then Python's import 215 If the named module is not found in :data:`sys.modules`, then Python's import
215 protocol is invoked to find and load the module. This protocol consists of 216 protocol is invoked to find and load the module. This protocol consists of
216 two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`. 217 two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`.
217 A finder's job is to determine whether it can find the named module using 218 A finder's job is to determine whether it can find the named module using
218 whatever strategy it knows about. Objects that implement both of these 219 whatever strategy it knows about. Objects that implement both of these
219 interfaces are referred to as :term:`importers <importer>` - they return 220 interfaces are referred to as :term:`importers <importer>` - they return
220 themselves when they find that they can load the requested module. 221 themselves when they find that they can load the requested module.
221 222
222 Python includes a number of default finders and importers. The first one 223 Python includes a number of default finders and importers. The first one
223 knows how to locate built-in modules, and the second knows how to locate 224 knows how to locate built-in modules, and the second knows how to locate
224 frozen modules. A third default finder searches an :term:`import path` 225 frozen modules. A third default finder searches an :term:`import path`
225 for modules. The :term:`import path` is a list of locations that may 226 for modules. The :term:`import path` is a list of locations that may
226 name file system paths or zip files. It can also be extended to search 227 name file system paths or zip files. It can also be extended to search
227 for any locatable resource, such as those identified by URLs. 228 for any locatable resource, such as those identified by URLs.
228 229
229 The import machinery is extensible, so new finders can be added to extend the 230 The import machinery is extensible, so new finders can be added to extend the
230 range and scope of module searching. 231 range and scope of module searching.
231 232
232 Finders do not actually load modules. If they can find the named module, they 233 Finders do not actually load modules. If they can find the named module, they
233 return a :term:`loader`, which the import machinery then invokes to load the 234 return a :term:`module spec`, an encapsulation of the module's import-related
234 module and create the corresponding module object. 235 information, which the import machinery then uses when loading the module.
235 236
236 The following sections describe the protocol for finders and loaders in more 237 The following sections describe the protocol for finders and loaders in more
237 detail, including how you can create and register new ones to extend the 238 detail, including how you can create and register new ones to extend the
238 import machinery. 239 import machinery.
239 240
241 .. versionchanged:: 3.4
242 In previous versions of Python, finders returned :term:`loaders <loader>`
243 directly, whereas now they return module specs which *contain* loaders.
244 Loaders are still used during import but have fewer responsibilities.
240 245
241 Import hooks 246 Import hooks
242 ------------ 247 ------------
243 248
244 .. index:: 249 .. index::
245 single: import hooks 250 single: import hooks
246 single: meta hooks 251 single: meta hooks
247 single: path hooks 252 single: path hooks
248 pair: hooks; import 253 pair: hooks; import
249 pair: hooks; meta 254 pair: hooks; meta
(...skipping 13 matching lines...) Expand all
263 ``package.__path__``) processing, at the point where their associated path 268 ``package.__path__``) processing, at the point where their associated path
264 item is encountered. Import path hooks are registered by adding new callables 269 item is encountered. Import path hooks are registered by adding new callables
265 to :data:`sys.path_hooks` as described below. 270 to :data:`sys.path_hooks` as described below.
266 271
267 272
268 The meta path 273 The meta path
269 ------------- 274 -------------
270 275
271 .. index:: 276 .. index::
272 single: sys.meta_path 277 single: sys.meta_path
273 pair: finder; find_module 278 pair: finder; find_spec
274 pair: finder; find_loader
275 279
276 When the named module is not found in :data:`sys.modules`, Python next 280 When the named module is not found in :data:`sys.modules`, Python next
277 searches :data:`sys.meta_path`, which contains a list of meta path finder 281 searches :data:`sys.meta_path`, which contains a list of meta path finder
278 objects. These finders are queried in order to see if they know how to handle 282 objects. These finders are queried in order to see if they know how to handle
279 the named module. Meta path finders must implement a method called 283 the named module. Meta path finders must implement a method called
280 :meth:`find_module()` which takes two arguments, a name and an import path. 284 :meth:`find_spec()` which takes two arguments, a name and an import path.
brett.cannon 2013/11/08 21:26:09 Three now as there is also 'target'.
eric.snow 2013/12/17 06:44:46 Done.
281 The meta path finder can use any strategy it wants to determine whether it can 285 The meta path finder can use any strategy it wants to determine whether it can
282 handle the named module or not. 286 handle the named module or not.
283 287
284 If the meta path finder knows how to handle the named module, it returns a 288 If the meta path finder knows how to handle the named module, it returns a
285 loader object. If it cannot handle the named module, it returns ``None``. If 289 spec object. If it cannot handle the named module, it returns ``None``. If
286 :data:`sys.meta_path` processing reaches the end of its list without returning 290 :data:`sys.meta_path` processing reaches the end of its list without returning
287 a loader, then an :exc:`ImportError` is raised. Any other exceptions raised 291 a spec, then an :exc:`ImportError` is raised. Any other exceptions raised
288 are simply propagated up, aborting the import process. 292 are simply propagated up, aborting the import process.
289 293
290 The :meth:`find_module()` method of meta path finders is called with two 294 The :meth:`find_spec()` method of meta path finders is called with two
brett.cannon 2013/11/08 21:26:09 three arguments
eric.snow 2013/12/17 06:44:46 Done.
291 arguments. The first is the fully qualified name of the module being 295 arguments. The first is the fully qualified name of the module being
292 imported, for example ``foo.bar.baz``. The second argument is the path 296 imported, for example ``foo.bar.baz``. The second argument is the path
293 entries to use for the module search. For top-level modules, the second 297 entries to use for the module search. For top-level modules, the second
294 argument is ``None``, but for submodules or subpackages, the second 298 argument is ``None``, but for submodules or subpackages, the second
295 argument is the value of the parent package's ``__path__`` attribute. If 299 argument is the value of the parent package's ``__path__`` attribute. If
296 the appropriate ``__path__`` attribute cannot be accessed, an 300 the appropriate ``__path__`` attribute cannot be accessed, an
297 :exc:`ImportError` is raised. 301 :exc:`ImportError` is raised.
298 302
299 The meta path may be traversed multiple times for a single import request. 303 The meta path may be traversed multiple times for a single import request.
300 For example, assuming none of the modules involved has already been cached, 304 For example, assuming none of the modules involved has already been cached,
301 importing ``foo.bar.baz`` will first perform a top level import, calling 305 importing ``foo.bar.baz`` will first perform a top level import, calling
302 ``mpf.find_module("foo", None)`` on each meta path finder (``mpf``). After 306 ``mpf.find_spec("foo", None)`` on each meta path finder (``mpf``). After
brett.cannon 2013/11/08 21:26:09 Technically ('foo', None, None) now (and the other
eric.snow 2013/12/17 06:44:46 Done.
303 ``foo`` has been imported, ``foo.bar`` will be imported by traversing the 307 ``foo`` has been imported, ``foo.bar`` will be imported by traversing the
304 meta path a second time, calling 308 meta path a second time, calling
305 ``mpf.find_module("foo.bar", foo.__path__)``. Once ``foo.bar`` has been 309 ``mpf.find_spec("foo.bar", foo.__path__)``. Once ``foo.bar`` has been
306 imported, the final traversal will call 310 imported, the final traversal will call
307 ``mpf.find_module("foo.bar.baz", foo.bar.__path__)``. 311 ``mpf.find_spec("foo.bar.baz", foo.bar.__path__)``.
308 312
309 Some meta path finders only support top level imports. These importers will 313 Some meta path finders only support top level imports. These importers will
310 always return ``None`` when anything other than ``None`` is passed as the 314 always return ``None`` when anything other than ``None`` is passed as the
311 second argument. 315 second argument.
312 316
313 Python's default :data:`sys.meta_path` has three meta path finders, one that 317 Python's default :data:`sys.meta_path` has three meta path finders, one that
314 knows how to import built-in modules, one that knows how to import frozen 318 knows how to import built-in modules, one that knows how to import frozen
315 modules, and one that knows how to import modules from an :term:`import path` 319 modules, and one that knows how to import modules from an :term:`import path`
316 (i.e. the :term:`path based finder`). 320 (i.e. the :term:`path based finder`).
317 321
318 322 .. versionchanged:: 3.4
319 Loaders 323 The find_spec() method of meta path finders replaced :meth:`find_module()`.
324 which is now deprecated. While it will continue to work without change,
325 the import machinery will try it only if the finder does not implement
326 find_spec().
327
328
329 Loading
320 ======= 330 =======
321 331
322 If and when a module loader is found its 332 If and when a module spec is found, the import machinery will use it (and
323 :meth:`~importlib.abc.Loader.load_module` method is called, with a single 333 the loader it contains) when loading the module. Here is an approximation
324 argument, the fully qualified name of the module being imported. This method 334 of what happens during the loading portion of import::
325 has several responsibilities, and should return the module object it has 335
326 loaded [#fnlo]_. If it cannot load the module, it should raise an 336 module = None
327 :exc:`ImportError`, although any other exception raised during 337 if spec.loader is not None and hasattr(spec.loader, 'create_module'):
328 :meth:`load_module()` will be propagated. 338 module = spec.loader.create_module(spec)
329 339 if module is None:
330 In many cases, the finder and loader can be the same object; in such cases the 340 module = ModuleType(spec.name)
331 :meth:`finder.find_module()` would just return ``self``. 341 # The import-related module attributes get set here:
332 342 _init_module_attrs(spec, module)
333 Loaders must satisfy the following requirements: 343
344 if spec.loader is None:
345 if spec.submodule_search_locations is not None:
346 # namespace package
347 sys.modules[spec.name] = module
348 else:
349 # unsupported
350 raise ImportError
351 elif not hasattr(spec.loader, 'exec_module'):
352 module = spec.loader.load_module(spec.name)
brett.cannon 2013/11/08 21:26:09 # __loader__ and __package__ if missing.
eric.snow 2013/12/17 06:44:46 Done.
353 else:
354 sys.modules[spec.name] = module
355 try:
356 spec.loader.exec_module(module)
357 except BaseException:
358 try:
359 del sys.modules[spec.name]
360 except KeyError:
361 pass
362 raise
363 module_to_return = sys.modules[spec.name]
brett.cannon 2013/11/08 21:26:09 return sys.modules[spec.name]
eric.snow 2013/12/17 06:44:46 Done.
364
365 Note the following details:
334 366
335 * If there is an existing module object with the given name in 367 * If there is an existing module object with the given name in
336 :data:`sys.modules`, the loader must use that existing module. (Otherwise, 368 :data:`sys.modules`, import will have already returned it.
337 :func:`imp.reload` will not work correctly.) If the named module does 369
338 not exist in :data:`sys.modules`, the loader must create a new module 370 * The module will exist in :data:`sys.modules` before the loader
339 object and add it to :data:`sys.modules`.
340
341 Note that the module *must* exist in :data:`sys.modules` before the loader
342 executes the module code. This is crucial because the module code may 371 executes the module code. This is crucial because the module code may
343 (directly or indirectly) import itself; adding it to :data:`sys.modules` 372 (directly or indirectly) import itself; adding it to :data:`sys.modules`
344 beforehand prevents unbounded recursion in the worst case and multiple 373 beforehand prevents unbounded recursion in the worst case and multiple
345 loading in the best. 374 loading in the best.
346 375
347 If loading fails, the loader must remove any modules it has inserted into 376 * If loading fails, the failing module -- and only the failing module --
348 :data:`sys.modules`, but it must remove **only** the failing module, and 377 gets removed from :data:`sys.modules`. Any module already in the
349 only if the loader itself has loaded it explicitly. Any module already in 378 :data:`sys.modules` cache, and any module that was successfully loaded
350 the :data:`sys.modules` cache, and any module that was successfully loaded 379 as a side-effect, must remain in the cache. This contrasts with
351 as a side-effect, must remain in the cache. 380 reloading where even the failing module is left in :data:`sys.modules`.
352 381
353 * The loader may set the ``__file__`` attribute of the module. If set, this 382 * After the module is created but before execution, the import machinery
354 attribute's value must be a string. The loader may opt to leave 383 sets the import-related module attributes ("init_module_attrs"), as
brett.cannon 2013/11/08 21:26:09 "(_init_module_attrs() in the pseudocode example a
eric.snow 2013/12/17 06:44:46 Done.
355 ``__file__`` unset if it has no semantic meaning (e.g. a module loaded from 384 summarized in a `later section <Import-related module attributes>`_.
356 a database). If ``__file__`` is set, it may also be appropriate to set the 385
357 ``__cached__`` attribute which is the path to any compiled version of the 386 * Module execution is the key moment of loading in which the module's
358 code (e.g. byte-compiled file). The file does not need to exist to set this 387 namespace gets populated. Execution is entirely delegated to the
359 attribute; the path can simply point to whether the compiled file would 388 loader, which gets to decide what gets populated and how.
360 exist (see :pep:`3147`). 389
361 390 * The module created during loading and passed to exec_module() may
362 * The loader may set the ``__name__`` attribute of the module. While not 391 not be the one returned at the end of import [#fnlo]_.
363 required, setting this attribute is highly recommended so that the 392
364 :meth:`repr()` of the module is more informative. 393 .. versionchanged:: 3.4
365 394 The import system has taken over the boilerplate responsibilities of
366 * If the module is a package (either regular or namespace), the loader must 395 loaders. These were previously performed by the :meth:`load_module()`
brett.cannon 2013/11/08 21:26:09 importlib.abc.Loader.load_module()
eric.snow 2013/12/17 06:44:46 Done.
367 set the module object's ``__path__`` attribute. The value must be 396 method.
368 iterable, but may be empty if ``__path__`` has no further significance 397
369 to the loader. If ``__path__`` is not empty, it must produce strings 398 Loaders
370 when iterated over. More details on the semantics of ``__path__`` are 399 -------
371 given :ref:`below <package-path-rules>`. 400
372 401 Module loaders provide the critical function of loading: module execution.
373 * The ``__loader__`` attribute must be set to the loader object that loaded 402 The import machinery calls the :meth:`~importlib.abc.Loader.exec_module()`
brett.cannon 2013/11/08 21:26:09 What does the ~ do?
eric.snow 2013/12/17 06:44:46 It's equivalent to :meth:`exec_module() <importlib
374 the module. This is mostly for introspection and reloading, but can be 403 method with a single argument, the module object to execute. Any value
375 used for additional loader-specific functionality, for example getting 404 returned from exec_module() is ignored.
brett.cannon 2013/11/08 21:26:09 :meth:...
eric.snow 2013/12/17 06:44:46 Done.
376 data associated with a loader. If the attribute is missing or set to ``None`` 405
377 then the import machinery will automatically set it **after** the module has 406 Loaders must satisfy the following requirements:
378 been imported.
379
380 * The module's ``__package__`` attribute must be set. Its value must be a
381 string, but it can be the same value as its ``__name__``. If the attribute
382 is set to ``None`` or is missing, the import system will fill it in with a
383 more appropriate value **after** the module has been imported.
384 When the module is a package, its ``__package__`` value should be set to its
385 ``__name__``. When the module is not a package, ``__package__`` should be
386 set to the empty string for top-level modules, or for submodules, to the
387 parent package's name. See :pep:`366` for further details.
388
389 This attribute is used instead of ``__name__`` to calculate explicit
390 relative imports for main modules, as defined in :pep:`366`.
391 407
392 * If the module is a Python module (as opposed to a built-in module or a 408 * If the module is a Python module (as opposed to a built-in module or a
393 dynamically loaded extension), the loader should execute the module's code 409 dynamically loaded extension), the loader should execute the module's code
394 in the module's global name space (``module.__dict__``). 410 in the module's global name space (``module.__dict__``).
395 411
396 412 * If loader cannot execute the module, it should raise an
brett.cannon 2013/11/08 21:26:09 "If the loader"
eric.snow 2013/12/17 06:44:46 Done.
397 Module reprs 413 :exc:`ImportError`, although any other exception raised during
398 ------------ 414 :meth:`exec_module()` will be propagated.
399 415
400 By default, all modules have a usable repr, however depending on the 416 In many cases, the finder and loader can be the same object; in such cases the
401 attributes set above, and hooks in the loader, you can more explicitly control 417 :meth:`finder.find_spec()` would just return a spec with the loader set
402 the repr of module objects. 418 to ``self``.
403 419
404 Loaders may implement a :meth:`module_repr()` method which takes a single 420 Module loaders may opt in to creating the module object during loading
405 argument, the module object. When ``repr(module)`` is called for a module 421 by implementing a :meth:`create_module()` method. It takes one argument,
406 with a loader supporting this protocol, whatever is returned from 422 the module spec, and returns the new module object to use during loading.
407 ``module.__loader__.module_repr(module)`` is returned as the module's repr 423 create_module() does not need to set any attributes on the module object.
408 without further processing. This return value must be a string. 424 If the loader does not define create_module(), the import machinery will
409 425 create the new module itself.
410 If the module has no ``__loader__`` attribute, or the loader has no 426
411 :meth:`module_repr()` method, then the module object implementation itself 427 .. versionadded:: 3.4
412 will craft a default repr using whatever information is available. It will 428 The create_module() method of loaders.
413 try to use the ``module.__name__``, ``module.__file__``, and 429
414 ``module.__loader__`` as input into the repr, with defaults for whatever 430 .. versionchanged:: 3.4
415 information is missing. 431 The load_module() method was replaced by exec_module() and the import
416 432 machinery assumed all the boilerplate responsibilities of loading.
417 Here are the exact rules used: 433
418 434 For compatibility with existing loaders, the import machinery will use
419 * If the module has a ``__loader__`` and that loader has a 435 the :meth:`~importlib.abc.Loader.load_module()` method of loaders if it
420 :meth:`module_repr()` method, call it with a single argument, which is the 436 exists and the loader does not also implement exec_module(). However,
421 module object. The value returned is used as the module's repr. 437 load_module() has been deprecated and loaders should implement
422 438 exec_module() instead.
423 * If an exception occurs in :meth:`module_repr()`, the exception is caught 439
424 and discarded, and the calculation of the module's repr continues as if 440 The load_module() method must implement all the boilerplate loading
425 :meth:`module_repr()` did not exist. 441 functionality described above in addition to executing the module. All
426 442 the same constraints apply, with some additional clarification:
427 * If the module has a ``__file__`` attribute, this is used as part of the 443
428 module's repr. 444 * If there is an existing module object with the given name in
429 445 :data:`sys.modules`, the loader must use that existing module.
430 * If the module has no ``__file__`` but does have a ``__loader__`` that is not 446 (Otherwise, :func:`imp.reload` will not work correctly.) If the
431 ``None``, then the loader's repr is used as part of the module's repr. 447 named module does not exist in :data:`sys.modules`, the loader
432 448 must create a new module object and add it to :data:`sys.modules`.
433 * Otherwise, just use the module's ``__name__`` in the repr. 449
434 450 * The module *must* exist in :data:`sys.modules` before the loader
435 This example, from :pep:`420` shows how a loader can craft its own module 451 executes the module code, to prevent unbounded recursion or multiple
436 repr:: 452 loading.
437 453
438 class NamespaceLoader: 454 * If loading fails, the loader must remove any modules it has inserted
439 @classmethod 455 into :data:`sys.modules`, but it must remove **only** the failing
440 def module_repr(cls, module): 456 module, and only if the loader itself has loaded it explicitly.
441 return "<module '{}' (namespace)>".format(module.__name__) 457
442 458 Module spec
459 -----------
460
461 The import machinery uses a variety of information about each module
462 during import, especially before loading. Most of the information is
463 common to all modules. The purpose of a module's spec is to encapsulate
464 this import-related information on a per-module basis.
465
466 Using a spec during import allows state to be transferred between import
467 system components, e.g. between the finder that creates the module spec
468 and the loader that executes it. Most importantly, it allows the
469 import machinery to perform the boilerplate operations of loading,
470 whereas without a module spec the loader had that responsibility.
471
472 See :class:`~importlib.machinery.ModuleSpec` for more specifics on what
473 information a module's spec may hold.
474
475 .. versionadded:: 3.4
476
477 Import-related module attributes
478 --------------------------------
479
480 The import machinery fills in these attributes on each module object
481 during loading, based on the module's spec, before the loader executes
482 the module.
483
484 .. attribute:: __name__
485
486 The ``__name__`` attribute must be set to the fully-qualified name of
487 the module. This name is used to uniquely identify the module in
488 the import system.
489
490 .. attribute:: __loader__
491
492 The ``__loader__`` attribute must be set to the loader object that
493 the import machinery used when loading the module. This is mostly
494 for introspection, but can be used for additional loader-specific
495 functionality, for example getting data associated with a loader.
496
497 .. attribute:: __package__
498
499 The module's ``__package__`` attribute must be set. Its value must
500 be a string, but it can be the same value as its ``__name__``. When
501 the module is a package, its ``__package__`` value should be set to
502 its ``__name__``. When the module is not a package, ``__package__``
503 should be set to the empty string for top-level modules, or for
504 submodules, to the parent package's name. See :pep:`366` for further
505 details.
506
507 This attribute is used instead of ``__name__`` to calculate explicit
508 relative imports for main modules, as defined in :pep:`366`.
509
510 .. attribute:: __spec__
511
512 The ``__spec__`` attribute must be set to the module spec that was
513 used when importing the module. This is used primarily for
514 introspection and during reloading.
515
516 .. attribute:: __path__
517
518 If the module is a package (either regular or namespace), the module
519 object's ``__path__`` attribute must be set. The value must be
520 iterable, but may be empty if ``__path__`` has no further significance.
521 If ``__path__`` is not empty, it must produce strings when iterated
522 over. More details on the semantics of ``__path__`` are given
523 :ref:`below <package-path-rules>`.
524
525 Non-package modules should not have a ``__path__`` attribute.
526
527 .. attribute:: __file__
528 .. attribute:: __cached__
529
530 ``__file__`` is optional. If set, this attribute's value must be a
531 string. The import system may opt to leave ``__file__`` unset if it
532 has no semantic meaning (e.g. a module loaded from a database).
533
534 If ``__file__`` is set, it may also be appropriate to set the
535 ``__cached__`` attribute which is the path to any compiled version of
536 the code (e.g. byte-compiled file). The file does not need to exist
537 to set this attribute; the path can simply point to where the
538 compiled file would exist (see :pep:`3147`).
539
540 It is also appropriate to set ``__cached__`` when ``__file__`` is not
541 set. However, that scenario is quite atypical. Ultimately, the
542 loader is what makes use of ``__file__`` and/or ``__cached__``. So
543 if a loader can load from a cached module but otherwise does not load
544 from a file, that atypical scenario may be appropriate.
443 545
444 .. _package-path-rules: 546 .. _package-path-rules:
445 547
446 module.__path__ 548 module.__path__
447 --------------- 549 ---------------
448 550
449 By definition, if a module has an ``__path__`` attribute, it is a package, 551 By definition, if a module has an ``__path__`` attribute, it is a package,
450 regardless of its value. 552 regardless of its value.
451 553
452 A package's ``__path__`` attribute is used during imports of its subpackages. 554 A package's ``__path__`` attribute is used during imports of its subpackages.
453 Within the import machinery, it functions much the same as :data:`sys.path`, 555 Within the import machinery, it functions much the same as :data:`sys.path`,
454 i.e. providing a list of locations to search for modules during import. 556 i.e. providing a list of locations to search for modules during import.
455 However, ``__path__`` is typically much more constrained than 557 However, ``__path__`` is typically much more constrained than
456 :data:`sys.path`. 558 :data:`sys.path`.
457 559
458 ``__path__`` must be an iterable of strings, but it may be empty. 560 ``__path__`` must be an iterable of strings, but it may be empty.
459 The same rules used for :data:`sys.path` also apply to a package's 561 The same rules used for :data:`sys.path` also apply to a package's
460 ``__path__``, and :data:`sys.path_hooks` (described below) are 562 ``__path__``, and :data:`sys.path_hooks` (described below) are
461 consulted when traversing a package's ``__path__``. 563 consulted when traversing a package's ``__path__``.
462 564
463 A package's ``__init__.py`` file may set or alter the package's ``__path__`` 565 A package's ``__init__.py`` file may set or alter the package's ``__path__``
464 attribute, and this was typically the way namespace packages were implemented 566 attribute, and this was typically the way namespace packages were implemented
465 prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no 567 prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no
466 longer need to supply ``__init__.py`` files containing only ``__path__`` 568 longer need to supply ``__init__.py`` files containing only ``__path__``
467 manipulation code; the namespace loader automatically sets ``__path__`` 569 manipulation code; the import machinery automatically sets ``__path__``
468 correctly for the namespace package. 570 correctly for the namespace package.
571
572 Module reprs
573 ------------
574
575 By default, all modules have a usable repr, however depending on the
576 attributes set above, and in the module's spec, you can more explicitly
577 control the repr of module objects.
578
579 If the module has a spec (``__spec__``), the import machinery will try
580 to generate a repr from it. If that fails or there is no spec, the import
581 system will craft a default repr using whatever information is available
582 on the module. It will try to use the ``module.__name__``,
583 ``module.__file__``, and ``module.__loader__`` as input into the repr,
584 with defaults for whatever information is missing.
585
586 Here are the exact rules used:
587
588 * If the module has a ``__spec__`` attribute, the information in the spec
589 is used to generate the repr. The "name", "loader", "origin", and
590 "has_location" attributes are consulted.
591
592 * If the module has a ``__file__`` attribute, this is used as part of the
593 module's repr.
594
595 * If the module has no ``__file__`` but does have a ``__loader__`` that is not
596 ``None``, then the loader's repr is used as part of the module's repr.
597
598 * Otherwise, just use the module's ``__name__`` in the repr.
599
600 .. versionchanged:: 3.4
601 Use of loader.module_repr() has been deprecated and the module spec
602 is now used by the import machinery to generate a module repr.
603
604 For backward compatibility with Python 3.3, the module repr will be
605 generated by calling the loader's :meth:`module_repr()` method, if
606 defined, before trying either approach described above. However, the
607 method is deprecated.
469 608
470 609
471 The Path Based Finder 610 The Path Based Finder
472 ===================== 611 =====================
473 612
474 .. index:: 613 .. index::
475 single: path based finder 614 single: path based finder
476 615
477 As mentioned previously, Python comes with several default meta path finders. 616 As mentioned previously, Python comes with several default meta path finders.
478 One of these, called the :term:`path based finder`, searches an :term:`import 617 One of these, called the :term:`path based finder`, searches an :term:`import
(...skipping 45 matching lines...) Expand 10 before | Expand all | Expand 10 after
524 single: sys.path_hooks 663 single: sys.path_hooks
525 single: sys.path_importer_cache 664 single: sys.path_importer_cache
526 single: PYTHONPATH 665 single: PYTHONPATH
527 666
528 The :term:`path based finder` is responsible for finding and loading Python 667 The :term:`path based finder` is responsible for finding and loading Python
529 modules and packages whose location is specified with a string :term:`path 668 modules and packages whose location is specified with a string :term:`path
530 entry`. Most path entries name locations in the file system, but they need 669 entry`. Most path entries name locations in the file system, but they need
531 not be limited to this. 670 not be limited to this.
532 671
533 As a meta path finder, the :term:`path based finder` implements the 672 As a meta path finder, the :term:`path based finder` implements the
534 :meth:`find_module()` protocol previously described, however it exposes 673 :meth:`find_spec()` protocol previously described, however it exposes
535 additional hooks that can be used to customize how modules are found and 674 additional hooks that can be used to customize how modules are found and
536 loaded from the :term:`import path`. 675 loaded from the :term:`import path`.
537 676
538 Three variables are used by the :term:`path based finder`, :data:`sys.path`, 677 Three variables are used by the :term:`path based finder`, :data:`sys.path`,
539 :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__`` 678 :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__``
540 attributes on package objects are also used. These provide additional ways 679 attributes on package objects are also used. These provide additional ways
541 that the import machinery can be customized. 680 that the import machinery can be customized.
542 681
543 :data:`sys.path` contains a list of strings providing search locations for 682 :data:`sys.path` contains a list of strings providing search locations for
544 modules and packages. It is initialized from the :data:`PYTHONPATH` 683 modules and packages. It is initialized from the :data:`PYTHONPATH`
545 environment variable and various other installation- and 684 environment variable and various other installation- and
546 implementation-specific defaults. Entries in :data:`sys.path` can name 685 implementation-specific defaults. Entries in :data:`sys.path` can name
547 directories on the file system, zip files, and potentially other "locations" 686 directories on the file system, zip files, and potentially other "locations"
548 (see the :mod:`site` module) that should be searched for modules, such as 687 (see the :mod:`site` module) that should be searched for modules, such as
549 URLs, or database queries. Only strings and bytes should be present on 688 URLs, or database queries. Only strings and bytes should be present on
550 :data:`sys.path`; all other data types are ignored. The encoding of bytes 689 :data:`sys.path`; all other data types are ignored. The encoding of bytes
551 entries is determined by the individual :term:`path entry finders <path entry 690 entries is determined by the individual :term:`path entry finders <path entry
552 finder>`. 691 finder>`.
553 692
554 The :term:`path based finder` is a :term:`meta path finder`, so the import 693 The :term:`path based finder` is a :term:`meta path finder`, so the import
555 machinery begins the :term:`import path` search by calling the path 694 machinery begins the :term:`import path` search by calling the path
556 based finder's :meth:`find_module()` method as described previously. When 695 based finder's :meth:`find_spec()` method as described previously. When
557 the ``path`` argument to :meth:`find_module()` is given, it will be a 696 the ``path`` argument to :meth:`find_spec()` is given, it will be a
558 list of string paths to traverse - typically a package's ``__path__`` 697 list of string paths to traverse - typically a package's ``__path__``
559 attribute for an import within that package. If the ``path`` argument 698 attribute for an import within that package. If the ``path`` argument
560 is ``None``, this indicates a top level import and :data:`sys.path` is used. 699 is ``None``, this indicates a top level import and :data:`sys.path` is used.
561 700
562 The path based finder iterates over every entry in the search path, and 701 The path based finder iterates over every entry in the search path, and
563 for each of these, looks for an appropriate :term:`path entry finder` for the 702 for each of these, looks for an appropriate :term:`path entry finder` for the
564 path entry. Because this can be an expensive operation (e.g. there may be 703 path entry. Because this can be an expensive operation (e.g. there may be
565 `stat()` call overheads for this search), the path based finder maintains 704 `stat()` call overheads for this search), the path based finder maintains
566 a cache mapping path entries to path entry finders. This cache is maintained 705 a cache mapping path entries to path entry finders. This cache is maintained
567 in :data:`sys.path_importer_cache` (despite the name, this cache actually 706 in :data:`sys.path_importer_cache` (despite the name, this cache actually
(...skipping 10 matching lines...) Expand all
578 entry finder` that can handle the path entry, or it may raise 717 entry finder` that can handle the path entry, or it may raise
579 :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to 718 :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to
580 signal that the hook cannot find a :term:`path entry finder` for that 719 signal that the hook cannot find a :term:`path entry finder` for that
581 :term:`path entry`. The exception is ignored and :term:`import path` 720 :term:`path entry`. The exception is ignored and :term:`import path`
582 iteration continues. The hook should expect either a string or bytes object; 721 iteration continues. The hook should expect either a string or bytes object;
583 the encoding of bytes objects is up to the hook (e.g. it may be a file system 722 the encoding of bytes objects is up to the hook (e.g. it may be a file system
584 encoding, UTF-8, or something else), and if the hook cannot decode the 723 encoding, UTF-8, or something else), and if the hook cannot decode the
585 argument, it should raise :exc:`ImportError`. 724 argument, it should raise :exc:`ImportError`.
586 725
587 If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` 726 If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder`
588 being returned, then the path based finder's :meth:`find_module()` method 727 being returned, then the path based finder's :meth:`find_spec()` method
589 will store ``None`` in :data:`sys.path_importer_cache` (to indicate that 728 will store ``None`` in :data:`sys.path_importer_cache` (to indicate that
590 there is no finder for this path entry) and return ``None``, indicating that 729 there is no finder for this path entry) and return ``None``, indicating that
591 this :term:`meta path finder` could not find the module. 730 this :term:`meta path finder` could not find the module.
592 731
593 If a :term:`path entry finder` *is* returned by one of the :term:`path entry 732 If a :term:`path entry finder` *is* returned by one of the :term:`path entry
594 hook` callables on :data:`sys.path_hooks`, then the following protocol is used 733 hook` callables on :data:`sys.path_hooks`, then the following protocol is used
595 to ask the finder for a module loader, which is then used to load the module. 734 to ask the finder for a module spec, which is then used when loading the
596 735 module.
597 736
598 Path entry finder protocol 737 Path entry finder protocol
599 -------------------------- 738 --------------------------
600 739
601 In order to support imports of modules and initialized packages and also to 740 In order to support imports of modules and initialized packages and also to
602 contribute portions to namespace packages, path entry finders must implement 741 contribute portions to namespace packages, path entry finders must implement
603 the :meth:`find_loader()` method. 742 the :meth:`find_spec()` method.
604 743
605 :meth:`find_loader()` takes one argument, the fully qualified name of the 744 :meth:`find_spec()` takes one argument, the fully qualified name of the
brett.cannon 2013/11/08 21:26:09 Takes two arguments now thanks to 'target'
eric.snow 2013/12/17 06:44:46 Done.
606 module being imported. :meth:`find_loader()` returns a 2-tuple where the 745 module being imported. :meth:`find_spec()` returns a fully populated
607 first item is the loader and the second item is a namespace :term:`portion`. 746 spec for the module. This spec will always have "loader" set (with one
608 When the first item (i.e. the loader) is ``None``, this means that while the 747 exception).
609 path entry finder does not have a loader for the named module, it knows that the
610 path entry contributes to a namespace portion for the named module. This will
611 almost always be the case where Python is asked to import a namespace package
612 that has no physical presence on the file system. When a path entry finder
613 returns ``None`` for the loader, the second item of the 2-tuple return value
614 must be a sequence, although it can be empty.
615 748
616 If :meth:`find_loader()` returns a non-``None`` loader value, the portion is 749 To indicate to the import machinery that the spec represents a namespace
617 ignored and the loader is returned from the path based finder, terminating 750 :term:`portion`. the path entry finder sets "loader" on the spec to
618 the search through the path entries. 751 ``None`` and "submodule_search_locations" to a list containing the
752 portion.
619 753
620 For backwards compatibility with other implementations of the import 754 .. versionchanged:: 3.4
621 protocol, many path entry finders also support the same, 755 find_spec() replaced find_loader() and find_module(), but of which
brett.cannon 2013/11/08 21:26:09 "but" -> "both"
eric.snow 2013/12/17 06:44:46 Done.
622 traditional :meth:`find_module()` method that meta path finders support. 756 are now deprecated, but will be used if find_spec() is not defined.
623 However path entry finder :meth:`find_module()` methods are never called
624 with a ``path`` argument (they are expected to record the appropriate
625 path information from the initial call to the path hook).
626 757
627 The :meth:`find_module()` method on path entry finders is deprecated, 758 Older path entry finders may implement one of these two deprecated methods
628 as it does not allow the path entry finder to contribute portions to 759 instead of :meth:`find_spec()`. The methods are still respected for the
629 namespace packages. Instead path entry finders should implement the 760 sake of backward compatibility. Howevever, if find_spec() is implemented
630 :meth:`find_loader()` method as described above. If it exists on the path 761 on the path entry finder, the legacy methods are ignored.
631 entry finder, the import system will always call :meth:`find_loader()` 762
632 in preference to :meth:`find_module()`. 763 :meth:`find_loader()` takes one argument, the fully qualified name of the
764 module being imported. :meth:`find_loader()` returns a 2-tuple where the
765 first item is the loader and the second item is a namespace :term:`portion`.
766 When the first item (i.e. the loader) is ``None``, this means that while the
767 path entry finder does not have a loader for the named module, it knows that
768 the path entry contributes to a namespace portion for the named module.
769 This will almost always be the case where Python is asked to import a
770 namespace package that has no physical presence on the file system.
771 When a path entry finder returns ``None`` for the loader, the second
772 item of the 2-tuple return value must be a sequence, although it can be
773 empty.
774
775 If :meth:`find_loader()` returns a non-``None`` loader value, the portion is
776 ignored and the loader is returned from the path based finder, terminating
777 the search through the path entries.
778
779 For backwards compatibility with other implementations of the import
780 protocol, many path entry finders also support the same,
781 traditional :meth:`find_module()` method that meta path finders support.
782 However path entry finder :meth:`find_module()` methods are never called
783 with a ``path`` argument (they are expected to record the appropriate
784 path information from the initial call to the path hook).
785
786 The :meth:`find_module()` method on path entry finders is deprecated,
787 as it does not allow the path entry finder to contribute portions to
788 namespace packages. If both :meth:`find_loader()` and :meth:`find_module()`
789 exist on a path entry finder, the import system will always call
790 :meth:`find_loader()` in preference to :meth:`find_module()`.
633 791
634 792
635 Replacing the standard import system 793 Replacing the standard import system
636 ==================================== 794 ====================================
637 795
638 The most reliable mechanism for replacing the entire import system is to 796 The most reliable mechanism for replacing the entire import system is to
639 delete the default contents of :data:`sys.meta_path`, replacing them 797 delete the default contents of :data:`sys.meta_path`, replacing them
640 entirely with a custom meta path hook. 798 entirely with a custom meta path hook.
641 799
642 If it is acceptable to only alter the behaviour of import statements 800 If it is acceptable to only alter the behaviour of import statements
643 without affecting other APIs that access the import system, then replacing 801 without affecting other APIs that access the import system, then replacing
644 the builtin :func:`__import__` function may be sufficient. This technique 802 the builtin :func:`__import__` function may be sufficient. This technique
645 may also be employed at the module level to only alter the behaviour of 803 may also be employed at the module level to only alter the behaviour of
646 import statements within that module. 804 import statements within that module.
647 805
648 To selectively prevent import of some modules from a hook early on the 806 To selectively prevent import of some modules from a hook early on the
649 meta path (rather than disabling the standard import system entirely), 807 meta path (rather than disabling the standard import system entirely),
650 it is sufficient to raise :exc:`ImportError` directly from 808 it is sufficient to raise :exc:`ImportError` directly from
651 :meth:`find_module` instead of returning ``None``. The latter indicates 809 :meth:`find_spec` instead of returning ``None``. The latter indicates
652 that the meta path search should continue. while raising an exception 810 that the meta path search should continue. while raising an exception
653 terminates it immediately. 811 terminates it immediately.
654 812
655 813
656 Open issues 814 Open issues
657 =========== 815 ===========
658 816
659 XXX It would be really nice to have a diagram. 817 XXX It would be really nice to have a diagram.
660 818
661 XXX * (import_machinery.rst) how about a section devoted just to the 819 XXX * (import_machinery.rst) how about a section devoted just to the
(...skipping 21 matching lines...) Expand all
683 841
684 :pep:`366` describes the addition of the ``__package__`` attribute for 842 :pep:`366` describes the addition of the ``__package__`` attribute for
685 explicit relative imports in main modules. 843 explicit relative imports in main modules.
686 844
687 :pep:`328` introduced absolute and explicit relative imports and initially 845 :pep:`328` introduced absolute and explicit relative imports and initially
688 proposed ``__name__`` for semantics :pep:`366` would eventually specify for 846 proposed ``__name__`` for semantics :pep:`366` would eventually specify for
689 ``__package__``. 847 ``__package__``.
690 848
691 :pep:`338` defines executing modules as scripts. 849 :pep:`338` defines executing modules as scripts.
692 850
851 :pep:`451` adds the encapsulation of per-module import state in spec
852 objects. It also off-loads most of the boilerplate responsibilities of
853 loaders back onto the import machinery. These changes allow the
854 deprecation of several APIs in the import system and also addition of new
855 methods to finders and loaders.
693 856
694 .. rubric:: Footnotes 857 .. rubric:: Footnotes
695 858
696 .. [#fnmo] See :class:`types.ModuleType`. 859 .. [#fnmo] See :class:`types.ModuleType`.
697 860
698 .. [#fnlo] The importlib implementation avoids using the return value 861 .. [#fnlo] The importlib implementation avoids using the return value
699 directly. Instead, it gets the module object by looking the module name up 862 directly. Instead, it gets the module object by looking the module name up
700 in :data:`sys.modules`. The indirect effect of this is that an imported 863 in :data:`sys.modules`. The indirect effect of this is that an imported
701 module may replace itself in :data:`sys.modules`. This is 864 module may replace itself in :data:`sys.modules`. This is
702 implementation-specific behavior that is not guaranteed to work in other 865 implementation-specific behavior that is not guaranteed to work in other
703 Python implementations. 866 Python implementations.
704 867
705 .. [#fnpic] In legacy code, it is possible to find instances of 868 .. [#fnpic] In legacy code, it is possible to find instances of
706 :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It 869 :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It
707 is recommended that code be changed to use ``None`` instead. See 870 is recommended that code be changed to use ``None`` instead. See
708 :ref:`portingpythoncode` for more details. 871 :ref:`portingpythoncode` for more details.
OLDNEW
« Doc/library/importlib.rst ('K') | « Doc/library/importlib.rst ('k') | Lib/importlib/abc.py » ('j') | no next file with comments »

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+