classification
Title: Make static methods created by @staticmethod callable
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, mark.dickinson, methane, rhettinger, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2021-03-31 14:31 by vstinner, last changed 2021-04-14 00:48 by methane. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25117 merged vstinner, 2021-03-31 14:37
PR 25268 merged vstinner, 2021-04-07 23:11
Messages (22)
msg389905 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-31 14:31
Currently, static methods created by the @staticmethod decorator are not callable as regular function. Example:
---
@staticmethod
def func():
    print("my func")

class MyClass:
    method = func

func() # A: regular function
MyClass.method() # B: class method
MyClass().method() # C: instance method
---

The func() call raises TypeError('staticmethod' object is not callable) exception.

I propose to make staticmethod objects callable to get a similar to built-in function:
---
func = len

class MyClass:
    method = func

func("abc") # A: regular function
MyClass.method("abc") # B: class method
MyClass().method("abc") # C: instance method
---

The 3 variants (A, B, C) to call the built-in len() function work just as expected.

If static method objects become callable, the 3 variants (A, B, C) will just work.

It would avoid the hack like _pyio.Wrapper:
---
class DocDescriptor:
    """Helper for builtins.open.__doc__
    """
    def __get__(self, obj, typ=None):
        return (
            "open(file, mode='r', buffering=-1, encoding=None, "
                 "errors=None, newline=None, closefd=True)\n\n" +
            open.__doc__)

class OpenWrapper:
    """Wrapper for builtins.open

    Trick so that open won't become a bound method when stored
    as a class variable (as dbm.dumb does).

    See initstdio() in Python/pylifecycle.c.
    """
    __doc__ = DocDescriptor()

    def __new__(cls, *args, **kwargs):
        return open(*args, **kwargs)
---

Currently, it's not possible possible to use directly _pyio.open as a method:
---
class MyClass:
    method = _pyio.open
---

whereas "method = io.open" just works because io.open() is a built-in function.


See also bpo-43680 "Remove undocumented io.OpenWrapper and _pyio.OpenWrapper" and my thread on python-dev:

"Weird io.OpenWrapper hack to use a function as method"
https://mail.python.org/archives/list/python-dev@python.org/thread/QZ7SFW3IW3S2C5RMRJZOOUFSHHUINNME/
msg389906 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-03-31 14:35
Seems like a duplicate of #20309.
msg389907 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-31 14:44
> Seems like a duplicate of #20309.

My usecase is to avoid any behavior difference between io.open and _pyio.open functions: PEP 399 "Pure Python/C Accelerator Module Compatibility Requirements". Currently, this is a very subtle difference when it's used to define a method.

I dislike the current _pyio.OpenWrapper "hack". I would prefer that _pyio.open would be directly usable to define a method. I propose to use @staticmethod, but I am open to other ideas. It could be a new decorator: @staticmethod_or_function.

Is it worth it to introduce a new @staticmethod_or_function decorator just to leave @staticmethod unchanged?

Note: The PEP 570 "Python Positional-Only Parameters" (implemented in Python 3.8) removed another subtle difference between functions implemented in C and functions implemented in Python. Now functions implemented in Python can only have positional only parameters.
msg389909 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-03-31 14:46
I don't understand what the problem is. _pyio.open is a function not a static method.

>>> import _pyio
>>> _pyio.open
<function open at 0x7f184cf33a10>
msg389910 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-31 14:50
> I don't understand what the problem is. _pyio.open is a function not a static method.

The problem is that _pyio.open doesn't behave exactly as io.open when it's used to define a method:
---
#from io import open
from _pyio import open

class MyClass:
   my_open = open

MyClass().my_open("document.txt", "w")
---

This code currently fails with a TypeError, whereas it works with io.open.

The problem is that I failed to find a way to create a function in Python which behaves exactly as built-in functions like len() or io.open().
msg389914 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-03-31 15:56
Isn't the problem that Python functions are (non-overriding) descriptors, but builtin-functions are not descriptors?
Changing static methods is not going to fix that.

How about adding wrappers to make Python functions behave like builtin functions and vice versa?
msg389916 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-31 16:33
> Changing static methods is not going to fix that.

My plan for the _pyio module is:

(1) Make static methods callable
(2) Decorate _pyio.open() with @staticmethod

That would only fix the very specific case of _pyio.open(). But open() use case seems to be common enough to became the example in the @staticmethod documentation!
https://docs.python.org/dev/library/functions.html#staticmethod

Example added in bpo-31567 "Inconsistent documentation around decorators" by:

commit 03b9537dc515d10528f83c920d38910b95755aff
Author: √Čric Araujo <merwok@users.noreply.github.com>
Date:   Thu Oct 12 12:28:55 2017 -0400

    bpo-31567: more decorator markup fixes in docs (GH-3959) (#3966)
msg389917 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-31 16:42
> Isn't the problem that Python functions are (non-overriding) descriptors, but builtin-functions are not descriptors?
> Changing static methods is not going to fix that.
> How about adding wrappers to make Python functions behave like builtin functions and vice versa?

I would love consistency, but is that possible without breaking almost all Python projects?

Honestly, I'm annoying by the having to use staticmethod(), or at least the fact that built-in functions and functions implemented in Python don't behave the same. It's hard to remind if a stdlib function requires staticmethod() or not. Moreover, maybe staticmethod() is not needed today, but it would become required tomorrow if the built-in function becomes a Python function somehow.

So yeah, I would prefer consistency. But backward compatibility may enter into the game as usual. PR 25117 tries to minimize the risk of backward compatibility issues.

For example, if we add __get__() to built-in methods and a bound method is created on the following example, it means that all code relying on the current behavior of built-in functions (don't use staticmethod) would break :-(
---
class MyClass:
    # built-in function currently converted to a method
    # magically without having to use staticmethod()
    method = len
---

Would it be possible to remove __get__() from FunctionType to allow using a Python function as a method? How much code would it break? :-) What would create the bound method on a method call?
---
def func():
    ...

class MyClass:
    method = func

# magic happens here!
bound_method = MyClass().method
---
msg389963 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-04-01 09:33
If make staticmethod a calllable and always wrap open, we need to change also its repr and add the __doc__ attribute (and perhaps other attributes to make it more interchangeable with the original function).

Alternate option: make staticmethod(func) returning func if it is not a descriptor.
msg390496 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-07 23:16
Serhiy Storchaka:
> If make staticmethod a calllable and always wrap open, we need to change also its repr and add the __doc__ attribute (and perhaps other attributes to make it more interchangeable with the original function).

You right and I like this idea! I created PR 25268 to inherit the function attributes (__name__, __doc__, etc.) in @staticmethod and @classmethod wrappers.
msg390525 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-08 11:34
There is a nice side effect of PR 25268 + PR 25117: pydoc provides better self documentation for the following code:

class X:
    @staticmethod
    def sm(x, y):
        '''A static method'''
        ...

pydoc on X.sm:
---
sm(x, y)
    A static method
---

instead of:
---
<staticmethod object>
---
msg390594 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-04-09 07:49
Currently pydoc on X.sm gives:
---
sm(x, y)
    A static method
---


I concur with Mark Shannon. The root problem is that Python functions and built-in functions have different behavior when assigned as class attribute. The former became an instance method, but the latter is not.

If wrap builtin open with statickmethod, the repr of open will be something like "staticmethod(<function open at 0x7f03031681b0>)" instead of just "<function open at 0x7f03031681b0>". It is confusing. It will produce a lot of questions why open (and only open) is so special.
msg390607 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-09 12:02
Serhiy:
> I concur with Mark Shannon. The root problem is that Python functions and built-in functions have different behavior when assigned as class attribute. The former became an instance method, but the latter is not.

Do you see a way to make C functions and Python functions behave the same?
msg390639 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-09 15:51
New changeset 507a574de31a1bd7fed8ba4f04afa285d985109b by Victor Stinner in branch 'master':
bpo-43682: @staticmethod inherits attributes (GH-25268)
https://github.com/python/cpython/commit/507a574de31a1bd7fed8ba4f04afa285d985109b
msg390704 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-04-10 08:39
> Do you see a way to make C functions and Python functions behave the same?

Implement __get__ for C functions.

Of course it is breaking change so we should first emit a warning. It will force all users to use staticmethod explicitly if they set a C function as a class attribute. We can also start emitting warnings for all callable non-descriptor class attributes.
msg390738 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-10 20:02
> Implement __get__ for C functions. Of course it is breaking change so we should first emit a warning. It will force all users to use staticmethod explicitly if they set a C function as a class attribute. We can also start emitting warnings for all callable non-descriptor class attributes.

Well... such change would impact way more code and sounds to require a painful migration plan.

Also, it doesn't prevent to make static methods callable, no?
msg390802 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-11 22:21
New changeset 553ee2781a37ac9d2068da3e1325a780ca79e21e by Victor Stinner in branch 'master':
bpo-43682: Make staticmethod objects callable (GH-25117)
https://github.com/python/cpython/commit/553ee2781a37ac9d2068da3e1325a780ca79e21e
msg390823 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-12 08:07
Ok, static methods are now callable in Python 3.10. Moreover, @staticmethod and @classmethod copy attributes from the callable object, same as functools.wraps().

Thanks to this change, I was able to propose to PR 25354 "bpo-43680: _pyio.open() becomes a static method".

Serhiy: if you want to "Implement __get__ for C functions", I suggest you opening a new issue for that. To be honest, I'm a little bit scared by the migration path, I expect that it will require to fix *many* projects.
msg390829 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-04-12 09:39
This is a significant change to the language.
There should be a PEP, or at the very least a discussion on Python Dev.

There may well be a very good reason why static methods have not been made callable before that you have overlooked.

Changing static methods to be callable will break backwards compatibility for any code that tests `callable(x)` where `x` is a static method.

I'm not saying that making staticmethods callable is a bad idea, just that it needs proper discussion.

https://bugs.python.org/issue20309 was closed as "won't fix". What has changed?
msg390841 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-12 12:09
Mark Shannon:
> Changing static methods to be callable will break backwards compatibility for any code that tests `callable(x)` where `x` is a static method.

Can you please elaborate on why this is an issue?

In the pydoc case, it sounds like an enhancement:
https://bugs.python.org/issue43682#msg390525
msg390848 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-04-12 13:10
Are you asking why breaking backwards compatibility is an issue?
Or how it breaks backwards compatibility?

pydoc could be changed to produce the proposed output, it doesn't need this change.

We don't know what this change will break, but we do know that it is a potentially breaking change.
`callable(staticmethod(f))` will change from `False` to `True`.

I don't think you should be making changes like this unilaterally.
msg391024 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-04-14 00:48
Strictly speaking, adding any method is "potential" breaking change because hasattr(obj, "new_method") become from False to True. And since Python is dynamic language, any change is "potential" breaking change.

But we don't treat such change as breaking change. Practical beats purity.
We can use beta period to see is this change breaks real world application.

In case of staticmethod, I think creating a new thread in python-dev is ideal because it is language core feature. I will post a thread.
History
Date User Action Args
2021-04-14 00:48:59methanesetmessages: + msg391024
2021-04-12 13:10:58Mark.Shannonsetmessages: + msg390848
2021-04-12 12:09:51vstinnersetmessages: + msg390841
2021-04-12 09:39:25Mark.Shannonsetmessages: + msg390829
2021-04-12 08:07:43vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg390823

stage: patch review -> resolved
2021-04-11 22:21:28vstinnersetmessages: + msg390802
2021-04-10 20:02:12vstinnersetmessages: + msg390738
2021-04-10 08:39:05serhiy.storchakasetmessages: + msg390704
2021-04-09 15:51:29vstinnersetmessages: + msg390639
2021-04-09 12:02:37vstinnersetmessages: + msg390607
2021-04-09 07:49:13serhiy.storchakasetmessages: + msg390594
2021-04-09 03:56:55methanesetnosy: + methane
2021-04-08 11:34:47vstinnersetmessages: + msg390525
2021-04-07 23:16:04vstinnersetmessages: + msg390496
2021-04-07 23:11:45vstinnersetpull_requests: + pull_request24003
2021-04-01 09:33:13serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg389963
2021-03-31 16:42:18vstinnersetmessages: + msg389917
2021-03-31 16:33:04vstinnersetmessages: + msg389916
2021-03-31 16:12:26mark.dickinsonsetnosy: + rhettinger
2021-03-31 15:56:11Mark.Shannonsetmessages: + msg389914
2021-03-31 14:51:06vstinnersettitle: Make function wrapped by staticmethod callable -> Make static methods created by @staticmethod callable
2021-03-31 14:50:45vstinnersetmessages: + msg389910
2021-03-31 14:46:24Mark.Shannonsetnosy: + Mark.Shannon

messages: + msg389909
title: Make static methods created by @staticmethod callable -> Make function wrapped by staticmethod callable
2021-03-31 14:44:37vstinnersettitle: Make function wrapped by staticmethod callable -> Make static methods created by @staticmethod callable
2021-03-31 14:44:15vstinnersetmessages: + msg389907
2021-03-31 14:37:54vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request23860
2021-03-31 14:35:50mark.dickinsonsetnosy: + mark.dickinson
messages: + msg389906
2021-03-31 14:31:21vstinnercreate