classification
Title: Immutable module type can't be used as package in custom loader
Type: behavior Stage: resolved
Components: Interpreter Core Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: dino.viehland Nosy List: brett.cannon, corona10, dino.viehland, vstinner
Priority: normal Keywords: patch

Created on 2020-01-14 21:33 by dino.viehland, last changed 2020-01-28 02:21 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18006 merged dino.viehland, 2020-01-14 23:51
Messages (7)
msg360000 - (view) Author: Dino Viehland (dino.viehland) * (Python committer) Date: 2020-01-14 21:33
I'm trying to create a custom module type for a custom loader where the returned modules are immutable.  But I'm running into an issue where the immutable module type can't be used as a module for a package.  That's because the import machinery calls setattr to set the module as an attribute on it's parent in _boostrap.py

        # Set the module as an attribute on its parent.
        parent_module = sys.modules[parent]
        setattr(parent_module, name.rpartition('.')[2], module)

I'd be okay if for these immutable module types they simply didn't have their children packages published on them.

A simple simulation of this is a package which replaces its self with an object which doesn't support adding arbitrary attributes:

x/__init__.py:
import sys

class MyMod(object):
    __slots__ = ['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
    def __init__(self):
        for attr in self.__slots__:
            setattr(self, attr, globals()[attr])


sys.modules['x'] = MyMod()

x/y.py:
# Empty file

>>> from x import y
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
AttributeError: 'MyMod' object has no attribute 'y'

There's a few different options I could see on how this could be supported:
    1) Simply handle the attribute error and allow things to continue
    2) Add ability for the modules loader to perform the set, and fallback to setattr if one isn't available.  Such as:
         getattr(parent_module, 'add_child_module', setattr)(parent_module, name.rpartition('.')[2], module)
    3) Add the ability for the module type to handle the setattr:
         getattr(type(parent_module), 'add_child_module', fallback)(parent_module, 
, name.rpartition('.')[2], module)
msg360006 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-01-14 22:16
So I think this is way too marginal a use-case to expand the API to let loaders inject themselves into the semantics of it.

I assume going with option 1 but raising an ImportWarning would be too noisy for your use-case? If not I'm totally fine with that solution.
msg360014 - (view) Author: Dino Viehland (dino.viehland) * (Python committer) Date: 2020-01-14 23:55
I think the warning shouldn't be too bad.  It looks like ImportWarnings are filtered by default already, and the extra overhead of raising a warning in this case probably is nothing compared to the actual work in loading the module.
msg360164 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-01-17 09:19
I apologize for the noise caused by the wrong PR connection.
msg360525 - (view) Author: Dino Viehland (dino.viehland) * (Python committer) Date: 2020-01-23 00:42
New changeset 9b6fec46513006d7b06fcb645cca6e4f5bf7c7b8 by Dino Viehland in branch 'master':
bpo-39336: Allow packages to not let their child modules be set on them (#18006)
https://github.com/python/cpython/commit/9b6fec46513006d7b06fcb645cca6e4f5bf7c7b8
msg360745 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-27 09:20
test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x: bpo-39459.
msg360832 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-28 02:21
commit 2528a6c3d0660c03ae43d796628462ccf8e58190
Author: Dino Viehland <dinoviehland@gmail.com>
Date:   Mon Jan 27 14:04:56 2020 -0800

    Add test.test_import.data.unwritable package to makefile (#18211)
History
Date User Action Args
2020-01-28 02:21:01vstinnersetmessages: + msg360832
2020-01-27 09:20:03vstinnersetnosy: + vstinner
messages: + msg360745
2020-01-23 00:57:27dino.viehlandsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-01-23 00:42:44dino.viehlandsetmessages: + msg360525
2020-01-17 09:19:58corona10setnosy: + corona10
messages: + msg360164
2020-01-17 09:18:35corona10setpull_requests: - pull_request17433
2020-01-17 09:17:08corona10setpull_requests: + pull_request17433
2020-01-17 09:16:53corona10setpull_requests: - pull_request17431
2020-01-17 09:15:57corona10setpull_requests: + pull_request17431
2020-01-14 23:55:02dino.viehlandsetmessages: + msg360014
2020-01-14 23:51:07dino.viehlandsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request17406
2020-01-14 22:16:53brett.cannonsetmessages: + msg360006
2020-01-14 21:40:20dino.viehlandsetnosy: + brett.cannon
2020-01-14 21:33:31dino.viehlandcreate