classification
Title: Add inspect.locate and inspect.resolve
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Michael.Felt, Trundle, berker.peksag, daniel.urban, eric.araujo, eric.snow, ezio.melotti, meador.inge, ncoghlan, ned.deily, python-dev, terry.reedy, vinay.sajip, yselivanov
Priority: normal Keywords: patch

Created on 2011-09-06 16:30 by eric.araujo, last changed 2020-06-29 10:35 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
issue12915.diff berker.peksag, 2012-12-10 13:48 review
issue12915_v2.diff berker.peksag, 2012-12-30 23:05 review
Pull Requests
URL Status Linked Edit
PR 18310 merged vinay.sajip, 2020-02-02 10:46
PR 18517 merged vinay.sajip, 2020-02-15 21:48
PR 18720 merged Michael.Felt, 2020-03-01 14:25
Messages (32)
msg143626 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-06 16:30
The need to resolve a dotted name to a Python object is spreading in the stdlib: pydoc has locate and resolve, packaging has util.resolve_name, unittest has something else, etc.  For the benefit of stdlib maintainers as well as the community, I think such functionality should be exposed publicly by the inspect module.
msg143796 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-09-09 19:41
To be a little clearer, this is about dotted import names, not regular dotted names.
pydoc.locate(path, forceload=0):
  """Locate an object by name or dotted path, importing as necessary"""
pydoc.resolve(thing, forceload=0):
  """Given an object or a path to an object, get the object and its name.""" 

It is not completely clear to me how this is different from using __import__ but I will believe it is. It something like this is in at least 3 places, +1 on factoring it out to one place.
msg143911 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-12 15:50
Sorry if I was unclear; the functions work with a dotted name as a string and resolve it to an object.  __import__ works with module, whereas this kind of functions work with mod.name, pkg.mod.name.attr, etc., that is, they combine import and getattr.
msg143960 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-13 12:26
In addition, error handling/reporting is not trivial to get right.  We’ve had to fix the code in distutils2 and it’s still not quite right (#12703).

I opened this report because I’d like to see all stdlib modules use the same functions and I’d prefer people to copy-paste the same robust code for backports.
msg145941 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-10-19 19:32
New changeset 1405df4a1535 by Éric Araujo in branch 'default':
Expand tests and fix bugs in packaging.util.resolve_name.
http://hg.python.org/cpython/rev/1405df4a1535
msg146020 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-20 15:10
The version in logging.config appears to be doing the same job, but is shorter:

def _resolve(name):
    """Resolve a dotted name to a global object."""
    name = name.split('.')
    used = name.pop(0)
    found = __import__(used)
    for n in name:
        used = used + '.' + n
        try:
            found = getattr(found, n)
        except AttributeError:
            __import__(used)
            found = getattr(found, n)
    return found

The line "used = used + '.' + n" could of course be improved.
msg147597 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-11-14 14:24
New changeset 5df1065ddb8b by Éric Araujo in branch 'default':
Expand tests and fix bugs in util.resolve_name.
http://hg.python.org/distutils2/rev/5df1065ddb8b
msg178625 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2012-12-30 23:05
Attaching new patch to address Vinay Sajip's suggestions on Rietveld. Thank you, Vinay.
msg182938 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-02-25 14:01
Something to consider with these functions: it is probably desirable to also support the popular alternate notation which uses an explicit ":" to separate the module name from the reference within the module.

For example, setuptools entry points and nose test references both use that format.

The alternate notation should be fairly easy to both detect and handle through "name.partition(':')"
msg183109 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-02-27 03:58
Can you provide a couple of examples of that notation?
msg183115 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2013-02-27 06:36
> Can you provide a couple of examples of that notation?

entry_points = """
[console_scripts]
pybabel = babel.messages.frontend:main

[distutils.commands]
compile_catalog = babel.messages.frontend:compile_catalog
extract_messages = babel.messages.frontend:extract_messages
init_catalog = babel.messages.frontend:init_catalog
update_catalog = babel.messages.frontend:update_catalog

[distutils.setup_keywords]
message_extractors = babel.messages.frontend:check_message_extractors

[babel.checkers]
num_plurals = babel.messages.checkers:num_plurals
python_format = babel.messages.checkers:python_format

[babel.extractors]
ignore = babel.messages.extract:extract_nothing
python = babel.messages.extract:extract_python
javascript = babel.messages.extract:extract_javascript
"""

Source:

http://babel.edgewall.org/browser/trunk/setup.py#L38
msg209878 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-02-01 08:31
Another question to consider: is inspect the best place for this? I don't think it is, because

(a) It's not really an inspection facility
(b) Importing inspect to get this functionality would pull in lots
    of stuff which wouldn't be used in the typical use case.

I think it makes more sense for it to be in importlib. Accordingly adding Brett to nosy, for his thoughts.
msg209893 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-02-01 15:45
importlib.util.resolve_name() already exists for resolving an explicit relative import name to an absolute one, so closing this as out of date.
msg209895 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-02-01 16:19
> importlib.util.resolve_name() already exists

But that's not what the proposed functionality is for, is it? This covers finding values inside an imported module which can be accessed via a dotted path from the module globals.
msg209902 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-02-01 17:32
Importlib already has importlib.import_module() (since Python 2.7) and that's as far as I'm willing to go for finding a module by name. Anything past that is a getarr() call on the resulting module and thus not worth adding to importlib.
msg209903 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-02-01 17:58
> and thus not worth adding to importlib.

Okay, fair enough. It's not purely an import function, though partly related to imports.
msg360664 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2020-01-25 00:14
With the lack of support, I suggest closing this.
msg360681 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-01-25 11:48
> With the lack of support, I suggest closing this.

Well, the lack of support is just for adding to importlib. It still seems useful to refactor it out ... just not clear where the best place is. Perhaps pkgutil? It's definitely useful to have in the stdlib in one place. I'll come up with a PR soon.
msg360710 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2020-01-25 20:48
FTR, Django has a similar helper:

    from django.utils.module_loading import import_string

    ValidationError = import_string('django.core.exceptions.ValidationError')

https://docs.djangoproject.com/en/3.0/ref/utils/#django.utils.module_loading.import_string
msg362000 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-02-14 22:02
New changeset 1ed61617a4a6632905ad6a0b440cd2cafb8b6414 by Vinay Sajip in branch 'master':
bpo-12915: Add pkgutil.resolve_name (GH-18310)
https://github.com/python/cpython/commit/1ed61617a4a6632905ad6a0b440cd2cafb8b6414
msg362007 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-15 10:02
The regex looks too strict. Python 3 accepts non-ASCII module names.

Should existing functions be modified to reuse the new function?
msg362022 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-02-15 17:23
> The regex looks too strict. Python 3 accepts non-ASCII module names.

Would adding the re.U flag be enough, or are you thinking of other possibilities?
msg362023 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-02-15 17:33
To be more precise, one could use

^(?!\d)(\w+)(\.(?!\d)(\w+))* for _DOTTED_WORDS and use the re.U flag when compiling the pattern. That should match any packages where segments start with a letter and continue with alphanumerics. Do you agree?
msg362132 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-17 09:46
I reopen the issue to discuss the non-ASCII identifiers.

Currently, the code uses [a-z_]\w*, but "é" is a valid Python module name for example:

$ echo 'print("here")' > é.py
$ python3 -c 'import é'
here

I'm not sure how to design the regex. I just reported a potential issue ;-) See the PEP 3131 and str.isidentifier() method:
https://docs.python.org/dev/library/stdtypes.html#str.isidentifier
msg362161 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-02-17 23:05
> I'm not sure how to design the regex.

Did you not look at the PR I added to address this (Unicode chars), with a regex change? I added you as a reviewer.
msg362881 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-02-28 14:26
New changeset 4f17c5cd9a1ec50fe8de7ef68c39220a01a862cb by Vinay Sajip in branch 'master':
bpo-12915: Improve Unicode support for package names and attributes. (GH-18517)
https://github.com/python/cpython/commit/4f17c5cd9a1ec50fe8de7ef68c39220a01a862cb
msg362987 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-02-29 16:27
PR18517 has, likely, a utf-8 dependency. AIX, default latin-1 does not accept the new test.

Starting with this merge AIX bot fails with:
======================================================================
ERROR: test_name_resolution (test.test_pkgutil.PkgutilTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_pkgutil.py", line 249, in test_name_resolution
    os.makedirs(d, exist_ok=True)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/os.py", line 223, in makedirs
    mkdir(name, mode)
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 17-19: ordinal not in range(256)

Not completely resolved.
msg363023 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-29 22:35
> File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_pkgutil.py", line 249, in test_name_resolution
>    os.makedirs(d, exist_ok=True)
(...)
> UnicodeEncodeError: 'latin-1' codec can't encode characters in position 17-19: ordinal not in range(256)

The test should be modified to skip the current uw value in the loop ("continue") if the filesystem encoding cannot encode the Unicode string (catch UnicodeEncodeError on the makedirs() call).
msg363044 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-03-01 07:12
I am very busy with normal work this week. However I’ll attempt to add a pr with your (Victor”s)  suggestion. 

Sent from my iPhone

> On 29 Feb 2020, at 23:36, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> STINNER Victor <vstinner@python.org> added the comment:
> 
>> File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_pkgutil.py", line 249, in test_name_resolution
>>   os.makedirs(d, exist_ok=True)
> (...)
>> UnicodeEncodeError: 'latin-1' codec can't encode characters in position 17-19: ordinal not in range(256)
> 
> The test should be modified to skip the current uw value in the loop ("continue") if the filesystem encoding cannot encode the Unicode string (catch UnicodeEncodeError on the makedirs() call).
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue12915>
> _______________________________________
>
msg363250 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-03 10:11
New changeset e0acec15412203359d59db33e447698ee51e07fc by Michael Felt in branch 'master':
bpo-12915: Skip test_pkgutil.test_name_resolution() non-encodable filenames (GH-18720)
https://github.com/python/cpython/commit/e0acec15412203359d59db33e447698ee51e07fc
msg363251 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-03 10:12
I understand that all known issues are now fixed, so I close the issue.
msg372552 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-29 07:26
See bpo-41154 for possible regression introduced by these changes.
History
Date User Action Args
2020-06-29 10:35:17vstinnersetnosy: - vstinner
2020-06-29 07:26:34ned.deilysetnosy: + ned.deily
messages: + msg372552
2020-03-03 10:12:12vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg363251

stage: patch review -> resolved
2020-03-03 10:11:21vstinnersetmessages: + msg363250
2020-03-01 14:25:35Michael.Feltsetstage: resolved -> patch review
pull_requests: + pull_request18078
2020-03-01 07:12:09Michael.Feltsetmessages: + msg363044
2020-02-29 22:35:39vstinnersetmessages: + msg363023
2020-02-29 16:27:53Michael.Feltsetnosy: + Michael.Felt
messages: + msg362987
2020-02-28 14:26:49vinay.sajipsetmessages: + msg362881
2020-02-17 23:05:18vinay.sajipsetmessages: + msg362161
2020-02-17 09:46:33vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg362132
2020-02-15 21:48:15vinay.sajipsetpull_requests: + pull_request17895
2020-02-15 17:33:13vinay.sajipsetmessages: + msg362023
2020-02-15 17:23:21vinay.sajipsetmessages: + msg362022
2020-02-15 10:02:24vstinnersetnosy: + vstinner
messages: + msg362007
2020-02-14 22:07:09vinay.sajipsetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.9, - Python 3.5
2020-02-14 22:02:21vinay.sajipsetmessages: + msg362000
2020-02-02 10:46:34vinay.sajipsetpull_requests: + pull_request17686
2020-01-25 20:48:50berker.peksagsetmessages: + msg360710
2020-01-25 11:48:44vinay.sajipsetmessages: + msg360681
2020-01-25 00:14:07eric.araujosetmessages: + msg360664
2020-01-24 23:32:27brett.cannonsetnosy: - brett.cannon
2014-02-01 17:58:05vinay.sajipsetmessages: + msg209903
2014-02-01 17:32:30brett.cannonsetnosy: brett.cannon, terry.reedy, vinay.sajip, ncoghlan, ezio.melotti, eric.araujo, Trundle, meador.inge, daniel.urban, python-dev, eric.snow, berker.peksag, yselivanov
messages: + msg209902
2014-02-01 16:19:23vinay.sajipsetstatus: closed -> open
resolution: out of date -> (no value)
messages: + msg209895
2014-02-01 15:45:57brett.cannonsetstatus: open -> closed
resolution: out of date
messages: + msg209893
2014-02-01 08:31:17vinay.sajipsetnosy: + brett.cannon
messages: + msg209878
2014-01-31 23:07:01yselivanovsetnosy: + yselivanov

versions: + Python 3.5, - Python 3.4
2013-02-27 06:36:21vinay.sajipsetmessages: + msg183115
2013-02-27 03:58:42ezio.melottisetmessages: + msg183109
2013-02-25 14:01:54ncoghlansetnosy: + ncoghlan
messages: + msg182938
2013-01-01 03:28:22meador.ingesetnosy: + meador.inge
2012-12-30 23:05:28berker.peksagsetfiles: + issue12915_v2.diff

messages: + msg178625
2012-12-10 14:48:57daniel.urbansetstage: test needed -> patch review
2012-12-10 13:48:06berker.peksagsetfiles: + issue12915.diff
nosy: + berker.peksag

keywords: + patch
versions: + Python 3.4, - Python 3.3
2011-11-14 14:24:20python-devsetmessages: + msg147597
2011-10-20 15:10:16vinay.sajipsetnosy: + vinay.sajip
messages: + msg146020
2011-10-19 19:41:48eric.snowsetnosy: + eric.snow
2011-10-19 19:32:55python-devsetnosy: + python-dev
messages: + msg145941
2011-09-13 12:26:01eric.araujosetmessages: + msg143960
2011-09-12 15:50:00eric.araujosetmessages: + msg143911
2011-09-09 19:41:57terry.reedysetnosy: + terry.reedy

messages: + msg143796
stage: test needed
2011-09-09 16:32:42daniel.urbansetnosy: + daniel.urban
2011-09-08 16:22:03Trundlesetnosy: + Trundle
2011-09-08 10:25:32ezio.melottisetnosy: + ezio.melotti
2011-09-06 16:30:32eric.araujocreate