This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: functools.update_wrapper breaks the signature of functools.partial objects
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8, Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, grahamd, larry, ncoghlan, ofey404, ping, yselivanov
Priority: normal Keywords: patch

Created on 2022-02-15 18:06 by larry, last changed 2022-04-11 14:59 by admin.

File name Uploaded Description Edit larry, 2022-02-15 18:06 ofey404, 2022-02-23 15:58
Pull Requests
URL Status Linked Edit
PR 31529 open ofey404, 2022-02-23 15:02
Messages (21)
msg413299 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-02-15 18:06
It's considered good hygiene to use functools.update_wrapper() to make your wrapped functions look like the original.  However, when using functools.partial() to pre-supply arguments to a function, if you then call functools.update_wrapper() to update that partial object, inspect.signature() returns the *original* function's signature, not the *wrapped* function's signature.

To be precise: if you wrap a function with functools.partial() to pre-provide arguments, then immediately call inspect.signature() on that partial object, it returns the correct signature with the pre-filled parameters removed.  If you then call functools.update_wrapper() to update the partial from the original function, inspect.signature() now returns the *wrong* signature. 

I looked into it a little.  The specific thing changing inspect.signature()'s behavior is the '__wrapped__' attribute added by functools.update_wrapper().  By default inspect.signature() will unwrap partial objects, but only if it has a '__wrapped__' attribute.

This all looks pretty deliberate.  And it seems like there was some thought given to this wrinkle; inspect.signature() takes a "follow_wrapper_chains" parameter the user can supply to control this behavior.  But the default is True, meaning that by default it unwraps partial objects if they have a '__wrapped__'.

I admit I don't have any context for this.  Why do we want inspect.signature() to return the wrong signature by default?
msg413557 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-02-19 18:34
Yury, Ka-Ping, can you guys shed any light on this?  Your names are still on
msg413688 - (view) Author: Ka-Ping Yee (ping) * (Python committer) Date: 2022-02-22 01:05
Hmm, interesting.  I wasn't involved in writing the `follow_wrapper_chains` feature, so I don't know why it's there.  I wonder if some digging through the revision history of `` and `` would yield insight.
msg413778 - (view) Author: Ofey Chan (ofey404) * Date: 2022-02-23 08:53
Hello, I am new to cpython project and want to help.

I dig into `follow_wrapper_chains` feature and found it really interesting.

In `inspect.signature()`, the conversion of `functools.partial` object's signature is made when going down the unwrap chain.

Relevant code:

So, there is an inconsistent assumption which cause the problem:

- `inspect.signature()` handle `functools.partial` object it met specially.
- `functools.update_wrapper()` just treat `functools.partial` object as a normal decorator and ignore it.
After calling `functools.update_wrapper()`, a new (wrong) signature is constructed, and it covers the original (right) process.

That's why `inspect.signature()` returns the *original* function's signature, not the *wrapped* function's signature.

In my humble opinion, A sane solution might be that: let the `functools.update_wrapper` respect the `functools.partial` object in the similar way of `inspect.signature()`.

I'm working on a pull request to express my idea more clearly, any help is welcome!
msg413814 - (view) Author: Ofey Chan (ofey404) * Date: 2022-02-23 15:57
I fix the problem.

But sorry for my last message, because it totally miss the point ;)

In `inspect.signature()`, the order of handle `follow_wrapper_chains` and `functools.partial` cause the problem.

Relevant code:

The original order is:

1. `follow_wrapper_chains` unwrap decorators.
   - It would check `__wrapped__` attribute in `unwrap()`.
   - `functools.update_wrapper()` would set `__wrapped__`.
2. Then handle `functools.partial`, construct new signature with `_signature_get_partial()`

So the original `functools.partial` object would skip (1), goto (2) and would be correctly processed.

But after calling `functools.update_wrapper()`, the `functools.partial` object has a `__wrapped__` attribute, so it directly handled by (1) and will never reach (2). That's why `inspect.signature()` return the original function's signature.

`` shows the `__wrapped__` attribute.

My solution is simple: swap the order of (1) and (2).

`functools.partial` is a special type of wrapper, handle it before going down the wrapper chain is sane.

And I have written test case to ensure it's correct, hope it works.
msg413833 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-02-23 17:49
Ofey, I appreciate your enthusiasm, but you should probably slow down.  Fixing the bug is probably going to be the easy part here.  But we're not to that stage quite yet.  First, we need to determine

* why the code behaves like this--is this behavior a genuine bug, or is it actually a bugfix for some worse behavior?
* will fixing the bug cause problems for Python users? and if so, can we still fix the bug while mitigating the damage to people who are unfortunately depending on the bug?

The next step is not to write a bugfix for this exact behavior, it's to determine why the code is the way it is.  If it was genuinely just a mistake, and we can simply fix it and people will thank us, then we may have a use for your patch.  But, generally, people who work on Python are smart, and they don't tend to commit dumb mistakes, so we can't simply assume it's a simple bug and fix it.
msg413876 - (view) Author: Ofey Chan (ofey404) * Date: 2022-02-24 02:50
Thank you Larry.  It can never be too careful to deal with language issues!

> why the code behaves like this--is this behavior a genuine bug, or is it actually a bugfix for some worse behavior?

In my view, there's rather an concept needing clarify, than a genuine bug.  I look into the file blame, and it shows the processing order around `follow_wrapper_chains` might be carefully arranged...

- The `follow_wrapper_chains` functionality is added in issue 13266:
- When `signature()` first implemented, `__wrapped__` is handled then `partial`, this order never changed.
   - Issue #15008: Implement PEP 362 "Signature Objects":

> will fixing the bug cause problems for Python users? and if so, can we still fix the bug while mitigating the damage to people who are unfortunately depending on the bug?

It's really a heavy responsibility!  Slow down is right... A clear explaination and plan should be constructed before any action is taken.

So I may study PEP 362 to get some context, about why the code is this way.  And I'm wondering what else can I do?
msg413896 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-02-24 09:21
Okay, so, I considered the problem for a while, and I have a reasonable theory about what follow_wrapper_chains was for in the first place.

If you have a generic decorator, like functools.cache(), it usually looks like this:

  def my_decorator(fn):
    def generic_version(*args, **kwargs):
      args, kwargs = do_something(args, kwargs)
      return fn(*args, **kwargs)
    return generic_version

  def add_five(i):
    return i+5

If you take the signature of add_five, you'd get (*args, **kwargs), because that's the signature of the wrapper function returned by the decorator.  The decorator doesn't change the parameters of the function, but because of how decorators work it can occlude the proper function signature.  In that instance, follow_wrapper_chains does the right thing, and as a result you get a precise function signature.

(Of course, it would do the wrong thing if your hand-written decorator *also* behaved like a partial application, adding in its own hard-coded arguments, so that the resulting function signature changed.)

Still, obviously it's doing the wrong thing when it comes to functools.partial() functions.

My suspicion is that I'm the rare individual who actually uses update_wrapper on a functools.partial object.  So maybe we have the situation here where, yeah, it's a bug, and we can fix it without causing further breakage.

Maybe we can loop in someone who works on a popular runtime function introspection library (FastAPI, Pydantic) to see if they have any take on it.
msg414175 - (view) Author: Ofey Chan (ofey404) * Date: 2022-02-28 06:07
> Maybe we can loop in someone who works on a popular runtime function introspection library (FastAPI, Pydantic) to see if they have any take on it.

I've checked issues of FastAPI and Pydantic. There is only one issue about `update_wrapper()`, and it's about documentation generation:

Would it be proper to open an issue under FastAPI and Pydantic, to describe the situation, and collect their feedbacks?
msg414176 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-02-28 06:21
I emailed the Pydantic and FastAPI guys and didn't hear back.  Given what you found on their issue trackers, I think it's unlikely that they care a lot about this issue (but were too busy to reply).  It's far more likely that they don't care.

Doing a little research (git blame), it looks like the "follow the wrapped chain to find the original signature" work was done by Nick Coghlan about nine years ago; he touched both functools.update_wrapper and the inspect module.

The only other people to touch the code recently are Yuri and Batuhan.  I've nosied Nick and Batuhan (already looped in Yuri), just to ping them and see if they have any strong opinions.

If nobody has anything remarkable to say about it, honestly we probably *can* just merge your patch, Ofey.  I see your name is now marked with a star; are you now authorized to contribute to CPython?

(Note that I only glanced at your patch so far; if we were going to merge this bugfix I would of course first do a full review.)
msg414638 - (view) Author: Ofey Chan (ofey404) * Date: 2022-03-07 06:33
I updated NEWS and all checks have passed!
msg414890 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-03-11 06:21
Nobody I've nosied on this issue recently has expressed any opinion on the matter.  I'm gonna try one more person: Graham Dumpleton, the maintainer of "wrapt", Python's premier function-wrapping.

Graham, care to express any opinions about this issue?  Can we fix it without causing widespread wailing and gnashing of teeth?  Do you think people are depending on the current how-can-you-describe-it-as-anything-but-broken behavior?
msg414937 - (view) Author: Graham Dumpleton (grahamd) Date: 2022-03-11 21:20
My vague recollection was that I identified some time back that partial() didn't behave correctly regards introspection for some use case I was trying to apply it to in the wrapt implementation. As a result I ended up creating my own PartialCallableObjectProxy implementation based around wrapt's own transparent object proxy object so that introspection worked properly and went with that where I needed it. I don't remember the exact details at the moment and don't think commit comments in code are likely to help. Even so, will try and spend some time this weekend looking more at the issue and see what I can remember about it and see if there is anything more I can comment on that may help.
msg415024 - (view) Author: Graham Dumpleton (grahamd) Date: 2022-03-13 04:35
I am still working through this and thinking about implications, but my first impression is that the functools.partial object should provide an attribute (property) __signature__ which yields the correct result.

When you think about it, any user who wants to implement a function wrapper using a class to do so rather than using functools.update_wrapper(), has to implement __signature__ if the wrapper is a signature changing decorator. So why shouldn't Python itself follow the same mechanism that is forced on users in their own wrappers.

If functools.partial were to implement __signature__, then the part of PEP 362 where it says:

> If the object is an instance of functools.partial, construct a new Signature from its partial.func attribute, and account for already bound partial.args and partial.kwargs

becomes redundant as the code to deal with it is localised within the functools.partial implementation by virtue of __signature__ on that type rather than having a special case in inspect.signature().

If this was seen as making more sense, one might even argue that FunctionType and the bound variant could implement __signature__ and so localise things to those implementations as well, which would further simplify inspect.signature().

This would set a good precedent going forward that if any special callable wrapper objects are added to the Python core in the future, that they implement __signature__, rather than someone thinking that further special cases could be added to inspect.signature() to deal with them.

I have yet to do some actual code experiments so might have more thoughts on the matter later.
msg415025 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-03-13 04:38
You make a good point.  I filed a separate bug (#46846) suggesting that partial objects should set their own annotations and signature.  I agree that objects performing such magic should take care of these details themselves, rather than requiring the inspect module to have workarounds based on deep knowledge of these other modules' inner workings.
msg416023 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-03-25 22:39
I heard back from both Samuel Colvin (Pydantic) and Sebastián Ramírez (FastAPI).  They said neither of them use update_wrapper with partial objects.  They also took a peek in a bunch of other projects (FastAPI, Typer, SQLModel, Asyncer, SQLAlchemy, Trio, and AnyIO) and nobody was doing it.

So honestly it seems like nobody (but me!) calls update_wrapper on partial objects, and we can just fix it.

Graham, any final thoughts before we start pulling levers and merging PRs?  For now I just want to fix this bug.  I'm in favor of re-engineering the relevant objects so they write their own __signature__ objects, so inspect.Signature doesn't have to understand the internals of objects from other modules.  But maybe we save that for another day.
msg416024 - (view) Author: Graham Dumpleton (grahamd) Date: 2022-03-25 23:18
It is Django I would worry about and look at closely as they do stuff with decorators on instance methods that uses partials.

    def _wrapper(self, *args, **kwargs):
        # bound_method has the signature that 'decorator' expects i.e. no
        # 'self' argument, but it's a closure over self so it can call
        # 'func'. Also, wrap method.__get__() in a function because new
        # attributes can't be set on bound method objects, only on functions.
        bound_method = wraps(method)(partial(method.__get__(self, type(self))))
        for dec in decorators:
            bound_method = dec(bound_method)
        return bound_method(*args, **kwargs)
msg416025 - (view) Author: Graham Dumpleton (grahamd) Date: 2022-03-25 23:28
Another example in Django, albeit in a test harness.

msg416028 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2022-03-26 00:22
Ooh, good one.  I don't know anybody in the Django project to contact though.  Anybody have any leads?
msg416031 - (view) Author: Graham Dumpleton (grahamd) Date: 2022-03-26 00:31
These days I have no idea who is active on Django.
msg416060 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2022-03-26 12:36
On the historical front, wraps & update_wrapper were only designed to handle true wrapper functions (i.e. those that don't change the calling signature).

For anything else (including partial), I considered it unlikely that the doc string would still be accurate, let alone any of the other metadata, so I didn't worry about supporting them.

That sais, if it's practical to make the results of combining the two less broken then I agree it would make sense to do so.
Date User Action Args
2022-04-11 14:59:56adminsetgithub: 90917
2022-03-26 12:36:12ncoghlansetmessages: + msg416060
2022-03-26 00:31:38grahamdsetmessages: + msg416031
2022-03-26 00:22:29larrysetmessages: + msg416028
2022-03-25 23:28:39grahamdsetmessages: + msg416025
2022-03-25 23:18:35grahamdsetmessages: + msg416024
2022-03-25 22:39:02larrysetmessages: + msg416023
2022-03-13 04:38:15larrysetmessages: + msg415025
2022-03-13 04:35:30grahamdsetmessages: + msg415024
2022-03-11 21:20:40grahamdsetmessages: + msg414937
2022-03-11 06:21:01larrysetnosy: + grahamd
messages: + msg414890
2022-03-07 06:33:35ofey404setmessages: + msg414638
2022-02-28 06:21:48larrysetnosy: + ncoghlan, BTaskaya
messages: + msg414176
2022-02-28 06:07:59ofey404setmessages: + msg414175
2022-02-24 09:21:14larrysetmessages: + msg413896
2022-02-24 02:50:47ofey404setmessages: + msg413876
2022-02-23 17:49:02larrysetmessages: + msg413833
2022-02-23 15:58:22ofey404setfiles: +
2022-02-23 15:57:53ofey404setmessages: + msg413814
2022-02-23 15:02:28ofey404setkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request29653
2022-02-23 08:53:26ofey404setnosy: + ofey404
messages: + msg413778
2022-02-22 01:05:52pingsetmessages: + msg413688
2022-02-19 18:34:32larrysetnosy: + ping, yselivanov
messages: + msg413557
2022-02-15 18:06:14larrycreate