3.10b2 -> 3.10b3 regression in importlib.metadata for rinoh #88625

asottile · 2021-06-19T06:22:26Z

BPO	44459
Nosy	@jaraco, @asottile, @pablogsal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jaraco'
closed_at = <Date 2021-06-19.23:43:36.535>
created_at = <Date 2021-06-19.06:22:26.381>
labels = ['deferred-blocker', 'library', '3.10', 'type-crash']
title = '3.10b2 -> 3.10b3 regression in importlib.metadata for rinoh'
updated_at = <Date 2021-06-19.23:43:36.532>
user = 'https://github.com/asottile'

bugs.python.org fields:

activity = <Date 2021-06-19.23:43:36.532>
actor = 'jaraco'
assignee = 'jaraco'
closed = True
closed_date = <Date 2021-06-19.23:43:36.535>
closer = 'jaraco'
components = ['Library (Lib)']
creation = <Date 2021-06-19.06:22:26.381>
creator = 'Anthony Sottile'
dependencies = []
files = []
hgrepos = []
issue_num = 44459
keywords = ['3.10regression']
message_count = 6.0
messages = ['396117', '396137', '396139', '396141', '396146', '396154']
nosy_count = 3.0
nosy_names = ['jaraco', 'Anthony Sottile', 'pablogsal']
pr_nums = []
priority = 'deferred blocker'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue44459'
versions = ['Python 3.10']

asottile · 2021-06-19T06:22:26Z

installed from git:

$ git remote -v
origin	https://github.com/brechtm/rinohtype.git (fetch)
origin	https://github.com/brechtm/rinohtype.git (push)
$ git rev-parse HEAD
4054539bae53eaddd10291c8429a1a32aeeb4786

working in 3.10 b2:

$ ./venv310/bin/python --version --version
Python 3.10.0b2 (default, Jun  2 2021, 00:22:18) [GCC 9.3.0]
$ ./venv310/bin/python -m rinoh
usage: rinoh [-h] [-f FORMAT] [-o OPTION=VALUE] [-t NAME or FILENAME] [-s NAME or FILENAME]
             [-O FILENAME or DIRECTORY] [-p PAPER] [-i] [--list-templates] [--list-stylesheets]
             [--list-fonts [FILENAME]] [--list-formats] [--list-options FRONTEND] [--version] [--docs]
             [input]

Render a structured document to PDF.

positional arguments:
  input                 the document to render

options:
  -h, --help            show this help message and exit
  -f FORMAT, --format FORMAT
                        the format of the input file (default: autodetect)
  -o OPTION=VALUE, --option OPTION=VALUE
                        options to be passed to the input file reader
  -t NAME or FILENAME, --template NAME or FILENAME
                        the document template or template configuration file to use (default: article)
  -s NAME or FILENAME, --stylesheet NAME or FILENAME
                        the style sheet used to style the document elements (default: the template's default)
  -O FILENAME or DIRECTORY, --output FILENAME or DIRECTORY
                        write the PDF output to FILENAME or to an existing DIRECTORY with a filename derived
                        from the input filename (default: the current working directory)
  -p PAPER, --paper PAPER
                        the paper size to render to (default: the template's default)
  -i, --install-resources
                        automatically install missing resources (fonts, templates, style sheets) from PyPI
  --list-templates      list the installed document templates and exit
  --list-stylesheets    list the installed style sheets and exit
  --list-fonts [FILENAME]
                        list the installed fonts or, if FILENAME is given, write a PDF file displaying all
                        the fonts
  --list-formats        list the supported input formats and exit
  --list-options FRONTEND
                        list the options supported by the given frontend and exit
  --version             show program's version number and exit
  --docs                open the online documentation in the default browser

broken in 3.10 b3:

$ ./venv/bin/python --version --version
Python 3.10.0b3+ (heads/3.10:1b4addf3cb, Jun 18 2021, 17:21:48) [GCC 9.3.0]
$ ./venv/bin/python -m rinoh
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/__init__.py", line 41, in <module>
    from . import resource
  File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/resource.py", line 205, in <module>
    from .template import DocumentTemplate
  File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/template.py", line 42, in <module>
    from .stylesheets import sphinx
  File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/stylesheets/__init__.py", line 42, in <module>
    .format(stylesheet.description, stylesheet))
  File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/style.py", line 670, in __str__
    for name, entry_point in self.installed_resources:
  File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/resource.py", line 54, in installed_resources
    for entry_point in ilm.entry_points()[cls.entry_point_group]:
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line 979, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line 437, in load
    ordered = sorted(eps, key=by_group)
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line -1, in <genexpr>
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line 600, in _normalized_name
    return Prepared.normalize(self.name)
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line 841, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/home/asottile/workspace/cpython/prefix/lib/python3.10/re.py", line 187, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

jaraco · 2021-06-19T15:38:01Z

I suspect the performance enhancements to distribution deduplication are relevant here. Previously, distributions were deduplicated based on the canonical distribution name. Now they're deduplicated on the normalized name. And it seems that when attempting to calculate the normalized name in the indicated environment that the Distribution.name isn't a text string. That's a little surprising, but I'll investigate to see how that condition comes about.

jaraco · 2021-06-19T16:00:32Z

Running the debugger, I can confirm that Distribution.name is None in this case:

~ $ python3.10 -m pip-run -q rinohtype -- -m pdb -m rinoh --help
WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.
You should consider upgrading via the '/usr/local/bin/python3.10 -m pip install --upgrade pip' command.
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/re.py(187)sub()
-> return _compile(pattern, flags).sub(repl, string, count)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pdb.py", line 1709, in main
    pdb._runmodule(mainpyfile)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pdb.py", line 1541, in _runmodule
    mod_name, mod_spec, code = runpy._get_module_details(module_name)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-w0xovt3h/rinoh/__init__.py", line 41, in <module>
    from . import resource
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-w0xovt3h/rinoh/resource.py", line 205, in <module>
    from .template import DocumentTemplate
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-w0xovt3h/rinoh/template.py", line 42, in <module>
    from .stylesheets import sphinx
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-w0xovt3h/rinoh/stylesheets/__init__.py", line 42, in <module>
    .format(stylesheet.description, stylesheet))
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-w0xovt3h/rinoh/style.py", line 670, in __str__
    for name, entry_point in self.installed_resources:
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-w0xovt3h/rinoh/resource.py", line 54, in installed_resources
    for entry_point in ilm.entry_points()[cls.entry_point_group]:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 979, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 437, in load
    ordered = sorted(eps, key=by_group)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line -1, in <genexpr>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 600, in _normalized_name
    return Prepared.normalize(self.name)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 841, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/re.py", line 187, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

(Pdb) string is None
True
(Pdb) u
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py(841)normalize()
-> return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
(Pdb) u
> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py(600)_normalized_name()
-> return Prepared.normalize(self.name)
(Pdb) self.name is None
True

jaraco · 2021-06-19T17:03:51Z

It seems that even on Python 3.9 or 3.10b2, the DynamicRinohDistribution would return None for the name property (despite the attempt to return the empty string). The Python object model doesn't allow overriding descriptor (@Property) in a superclass with a static class property in the subclass.

I'm struggling to understand how the DynamicRinohDistribution works at all. I think I see. importlib.metadata.Distribution, while it declares a couple of abstract methods, doesn't actually declare itself an abstract base class, so someone can subclass it and get the degenerate behavior for the abstract methods. That's why a DynamicRinohDistribution.metadata gets None for self.read_text and then email.message_from_string(None) returns an empty Message object.

My instinct here is that the Distribution.name was always expected to return a string and the fact that it's returning None is a symptom of the subclass not implementing the abstract methods. To be sure, type annotations might have helped here (if the name property declared that the return type is str, then None would be an invalid return value).

The importlib distribution discovery mechanism was designed for other package distribution providers to make their distributions visible. It was not designed with the Rinoh use-case in mind (where a distribution already installed using the standard finders would present other "dynamic" distributions).

Given that Rinoh is using the distribution discovery mechanism in a way that it was not designed to support and is unlikely to be a widespread (or even repeated) use-case, I'm leaning toward a recommendation that the issue be fixed in Rinoh by defining a name property that returns text (and preferably something other than the empty string).

jaraco · 2021-06-19T17:25:18Z

A small patch to rinoh seems to work around the issue:

diff --git a/src/rinoh/resource.py b/src/rinoh/resource.py
index 57143f9d..586a4bb7 100644
--- a/src/rinoh/resource.py
+++ b/src/rinoh/resource.py
@@ -132,7 +132,9 @@ class DynamicEntryPoint(DynamicEntryPointBase):
 class DynamicRinohDistribution(ilm.Distribution):
     """Distribution for registering resource entry points to at runtime"""
 
-    name = ''
+    @property
+    def name(self):
+        return 'rinoh-dynamic-distro'
 
     def __init__(self):
         self._templates = {}

rinohtype master $ python3.10 -m pip-run --use-feature=in-tree-build -q . -- -m rinoh --help
usage: rinoh [-h] [-f FORMAT] [-o OPTION=VALUE] [-t NAME or FILENAME] [-s NAME or FILENAME] [-O FILENAME or DIRECTORY] [-p PAPER] [-i] [--list-templates]
             [--list-stylesheets] [--list-fonts [FILENAME]] [--list-formats] [--list-options FRONTEND] [--version] [--docs]
             [input]

Render a structured document to PDF.

positional arguments:
  input                 the document to render

options:
  -h, --help            show this help message and exit
  -f FORMAT, --format FORMAT
                        the format of the input file (default: autodetect)
  -o OPTION=VALUE, --option OPTION=VALUE
                        options to be passed to the input file reader
  -t NAME or FILENAME, --template NAME or FILENAME
                        the document template or template configuration file to use (default: article)
  -s NAME or FILENAME, --stylesheet NAME or FILENAME
                        the style sheet used to style the document elements (default: the template's default)
  -O FILENAME or DIRECTORY, --output FILENAME or DIRECTORY
                        write the PDF output to FILENAME or to an existing DIRECTORY with a filename derived from the input filename (default: the current
                        working directory)
  -p PAPER, --paper PAPER
                        the paper size to render to (default: the template's default)
  -i, --install-resources
                        automatically install missing resources (fonts, templates, style sheets) from PyPI
  --list-templates      list the installed document templates and exit
  --list-stylesheets    list the installed style sheets and exit
  --list-fonts [FILENAME]
                        list the installed fonts or, if FILENAME is given, write a PDF file displaying all the fonts
  --list-formats        list the supported input formats and exit
  --list-options FRONTEND
                        list the options supported by the given frontend and exit
  --version             show program's version number and exit
  --docs                open the online documentation in the default browser

Anthony, does that recommendation address the concern? Do you have any reason to think other users may be doing something similarly? Do you think it's worth adding a compatibility shim to support this improper usage?

jaraco · 2021-06-19T23:43:36Z

I submitted brechtm/rinohtype#270 to address the issue in rinohtype. Please feel free to reopen and address the questions posed above if appropriate.

BobDenny · 2022-04-16T13:03:39Z

@jaraco -- I hand-applied this tiny patch to my resource.py from rinoh 0.5.3 (2021-06-16) and I now have a functional rinoh. Thank you so much. This exact issue "out of the box:" got me too and I'm using 3.7.

Python 3.7.8 (tags/v3.7.8:4b47a5b6ba, Jun 28 2020, 08:53:46) [MSC v.1916 64 bit (AMD64)] on win32

asottile mannequin added 3.10 only security fixes stdlib Python modules in the Lib dir type-crash A hard crash of the interpreter, possibly with a core dump labels Jun 19, 2021

jaraco assigned jaraco Jun 19, 2021

pablogsal added deferred-blocker labels Jun 19, 2021

jaraco added deferred-blocker and removed deferred-blocker labels Jun 19, 2021

jaraco closed this as completed Jun 19, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3.10b2 -> 3.10b3 regression in importlib.metadata for rinoh #88625

3.10b2 -> 3.10b3 regression in importlib.metadata for rinoh #88625

asottile mannequin commented Jun 19, 2021

asottile mannequin commented Jun 19, 2021

jaraco commented Jun 19, 2021

jaraco commented Jun 19, 2021

jaraco commented Jun 19, 2021

jaraco commented Jun 19, 2021

jaraco commented Jun 19, 2021

BobDenny commented Apr 16, 2022

3.10b2 -> 3.10b3 regression in importlib.metadata for rinoh #88625

3.10b2 -> 3.10b3 regression in importlib.metadata for rinoh #88625

Comments

asottile mannequin commented Jun 19, 2021

asottile mannequin commented Jun 19, 2021

jaraco commented Jun 19, 2021

jaraco commented Jun 19, 2021

jaraco commented Jun 19, 2021

jaraco commented Jun 19, 2021

jaraco commented Jun 19, 2021

BobDenny commented Apr 16, 2022