Issue43460
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2021-03-10 11:30 by douglas-raillard-arm, last changed 2022-04-11 14:59 by admin.
Messages (3) | |||
---|---|---|---|
msg388427 - (view) | Author: Douglas Raillard (douglas-raillard-arm) | Date: 2021-03-10 11:30 | |
Instances of subclasses of BaseException created with keyword argument fail to copy properly as demonstrated by: import copy class E(BaseException): def __init__(self, x): self.x=x # works fine e = E(None) copy.copy(e) # raises e = E(x=None) copy.copy(e) This seems to affect all Python versions I've tested (3.6 <= Python <= 3.9). I've currently partially worked around the issue with a custom pickler that just restores __dict__, but: * "args" is not part of __dict__, and setting "args" key in __dict__ does not create a "working object" (i.e. the key is set, but is ignored for all intents and purposes except direct lookup in __dict__) * pickle is friendly: you can provide a custom pickler that chooses the reduce function for each single class. copy module is much less friendly: copyreg.pickle() only allow registering custom functions for specific classes. That means there is no way (that I know) to make copy.copy() select a custom reduce for a whole subclass tree. One the root of the issue: * exception from the standard library prevent keyword arguments (maybe because of that issue ?), but there is no such restriction on user-defined classes. * the culprit is BaseException_reduce() (in Objects/exceptions.c) [1] It seems that the current behavior is a consequence of the __dict__ being created lazily, I assume for speed and memory efficiency There seems to be a few approaches that would solve the issue: * keyword arguments passed to the constructor could be fused with the positional arguments in BaseException_new (using the signature, but signature might be not be available for extension types I suppose) * keyword arguments could just be stored like "args" in a "kwargs" attribute in PyException_HEAD, so they are preserved and passed again to __new__ when the instance is restored upon copying/pickling. * the fact that keyword arguments were used could be saved as a bool in PyException_HEAD. When set, this flag would make BaseException_reduce() only use __dict__ and not "args". This would technically probably be a breaking change, but the only cases I can think of where this would be observable are a bit far fetched (if __new__ or __init__ have side effects beyond storing attributes in __dict__). [1] https://github.com/python/cpython/blob/master/Objects/exceptions.c#L134 |
|||
msg388428 - (view) | Author: Douglas Raillard (douglas-raillard-arm) | Date: 2021-03-10 11:47 | |
The solution based on the signature is something along those lines: class E(BaseException): def __new__(cls, *args, **kwargs): """ Fix exception copying. Turn all the keyword arguments into positional arguments, so that the :exc:`BaseException` machinery has all the parameters for a valid call to ``__new__``, instead of missing all the keyword arguments. """ sig = inspect.signature(cls.__init__) bound_args = sig.bind_partial(*args, **kwargs) bound_args.apply_defaults() args = tuple(bound_args.arguments.values()) return super().__new__(cls, *args) def __init__(self, x): self.x=x But there are a many shortcomings to that approach: * What if super().__new__() consumes arguments before passing the rest to __init__() ? This fix is blind to that since it only cares about __init__ signature * What if inspect.signature() does not provide a signature (extension modules) ? * Defaults are "hardcoded" in the args, so the object will always be restored with the defaults of the time it was created. This is a breaking change, as currently the defaults used when restoring the instance are the current ones. * Also uses more memory for args (and for pickle files), since it contains all the defaults |
|||
msg396600 - (view) | Author: Irit Katriel (iritkatriel) * | Date: 2021-06-27 21:31 | |
See issue32696, issue30005, issue29466 |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:42 | admin | set | github: 87626 |
2021-06-27 21:31:34 | iritkatriel | set | versions:
+ Python 3.11 nosy: + iritkatriel messages: + msg396600 components: + Library (Lib) |
2021-03-10 11:47:51 | douglas-raillard-arm | set | messages: + msg388428 |
2021-03-10 11:30:36 | douglas-raillard-arm | create |