typing.NewType doesn't support PEP-604.

>>> import typing
>>> A = typing.NewType("A", int)
>>> B = typing.NewType("B", str)
>>> A | B
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'function' and 'function'

It is possible to implement NewType to return a class rather than a function as it does currently. However, the class version uses substantially more memory (1072 bytes vs 144 bytes with sys.getsizeof) and NewType is supposed to have almost no runtime overhead.

What about to return callable object instead of function from a typing.NewType?

It will look something like this:

class _NewType:
    __slots__ = ('__name__', '__supertype__')

    def __init__(self, name, supertype):
        self.__name__ = name
        self.__supertype__ = supertype

    def __call__(self, val):
        return val

    def __or__(self, other):
        return Union[self, other]

    def __ror__(self, other):
        return Union[other, self]



def NewType(name, tp):
    """NewType creates simple unique types with almost zero
    runtime overhead. NewType(name, tp) is considered a subtype of tp
    by static type checkers. At runtime, NewType(name, tp) returns
    a dummy callable object that simply returns its argument. Usage::

        UserId = NewType('UserId', int)

        def name_by_id(user_id: UserId) -> str:
            ...

        UserId('user')          # Fails type check

        name_by_id(42)          # Fails type check
        name_by_id(UserId(42))  # OK

        num = UserId(5) + 1     # type: int
    """
    return _NewType(name, tp)

With such implementation __or__ will be available for a NewType and actually because of slots size of object will be 48 bytes (with sys.getsizeof) which is less than current 144 bytes (if memory is important).

python/typing#746 has some previous discussion of implementing NewType as a class (motivated by __repr__), including benchmarks.

Jelle I can try to implement it in C to see if it will give better performance.

@yurii, I would like to caution against a C accelerator for the typing module. My reasoning follows:

1. The maintenance burden is higher. typing is already somewhat complex in Python (lots of MRO/metaclass wizardry). A C module would require knowledge of the C API.

2. Following from 1., I fear it may raise the barrier for contributions. This is purely anecdotal but I think there are more contributors who know Python than Python + C API.

3. C API code is much more error-prone than Python code.

4. It's very hard to find available reviewers for typing-related C changes.

5. Backports become harder due to point 3. and 4. Also C code needs much more scrutiny. If we cause some obscure bug in Python, it raises an exception; in C it potentially segfaults and kills the interpreter.

6. Third-party monkey-patching becomes harder.

Unfortunately, I can't offer a good solution to this issue at the moment either.

@ken thanks for pointing this out.

The only solution which I see is to convert NewType into callable class but it will lead to performance degradation.

@ken what do you think about that?

Do we have to solve it? Using from __future__ import annotations it should work, at least when used as an annotation. Or is that too inconvenient or inconsistent?

Using from __future__ import annotations it should work

Not in type aliases or generic bases. (And if Larry's co_annotations PEP is accepted, it would break in normal annotations too.) I'd prefer to get rid of this sort of subtle difference where the obvious way to write a type only works in some contexts, though I realize that's not possible in all cases.

That means I think we should move forward with making NewTypes into instances with __call__ instead of functions, though according to my benchmarks in the typing issue that will incur a small performance penalty.

I opened PR with refactored typing.NewType into callable class.

Also I have added __module__ attr to typing.NewType so it will become pickable.

Jelle, can you post more details about your benchmark?

This is what I got last year (copied from the typing issues):

In [83]: from typing import NewType

In [84]: nt = NewType("nt", int)

In [85]: class NewTypeClass:
...: def __init__(self, name, supertype):
...: self.name = name
...: self.supertype = supertype
...: def __call__(self, obj):
...: return obj
...: def __repr__(self):
...: return f"<NewType: {self.name}>"
...:

In [86]: ntc = NewTypeClass("ntc", int)

In [87]: ntc
Out[87]: <NewType: ntc>

In [88]: %timeit nt(3)
211 ns ± 2.27 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [89]: %timeit ntc(3)
253 ns ± 5.35 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Not sure what version of Python I used at the time. I just repeated it on 3.9.4 MacOS and got much faster time but a similar ~20% slowdown:

In [6]: %timeit nt(3)
112 ns ± 5.11 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [7]: %timeit ntc(3)
136 ns ± 2.36 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Hm, 20% isn't so bad, but one of the arguments for NewType was that it "disappears" at runtime -- which the current version does, but the class-based version doesn't quite.

Right now I don't have the cycles to think about this deeply. Maybe a few weeks after I'm back from vacation (~August 8) I will have more time.

This issue is now out of date on. After Serhiy's refactoring, any function types can be unioned.

>>> NewType('x', int)
>>> int | NewType('x', int)
int | typing.NewType.<locals>.new_type

The only problem now is that the repr is weird, but Serhiy also has a fix for that in bpo-34963 (PR 9951) without converting to a class.

Thus, I am closing the issue. Please go to bpo-44642 for a general discussion on whether union should support arbitrary functions at all, or bpo-34963 for the repr problem. Thanks all!

Does that work if you try to union two NewTypes?

Does that work if you try to union two NewTypes?

Oh woops I was too eager in closing this issue -- it doesn't work. Sorry, please disregard my previous message. I'm reopening this. Thanks for the reminder Jelle.

New changeset 965dd76 by Yurii Karabas in branch 'main':
bpo-44353: Refactor typing.NewType into callable class (GH-27250)
965dd76

New changeset 4868b94 by Yurii Karabas in branch 'main':
bpo-44353: Add test to cover __or__ of two NewType (bpo-27259)
4868b94

Hm, 20% isn't so bad, but one of the arguments for NewType was that it "disappears" at runtime -- which the current version does, but the class-based version doesn't quite.

NewType was described originally in python/typing#189 and the resulting PEP text landed in https://github.com/python/typing/pull/226/files.

I find the 20% performance impact non-ideal but ultimately worth it because:

• it solves the issue described in this bug;
• it allows us to implement issubclass/isinstance in the future if we so choose;
• it also potentially allows us to chain NewType.

Since we can't fully rely on from __future__ import annotations to side-step the performance cost, we don't really have a different option than to change it. The alternative is to leave it as is which makes it a non-composable pseudo-type unlike the others.

Currently adoption of NewType is relatively low, in part due to the feature's obscurity and partially because of its limits. Slower instantiation performance will at worst keep the adoption low, but it can potentially bring more users to NewType since it becomes less hacky.

I'm +1 to converting to a class as done by the PR.

Right now I don't have the cycles to think about this deeply. Maybe a few weeks after I'm back from vacation (~August 8) I will have more time.

Sorry, I overeagerly already merged the change to main. Let's hold off with the 3.10 backport until you had time to think about it. If we decide against the change after all, reverting on main should have no negative impact since we're pre-alpha 1.

OTOH, for 3.10 the clock is ticking. Let's go for it. (If it's deemed too slow, we could eventually add an accelerator just for this.)

I found that replacing __call__ on the NewType class with an identity function written in C makes things faster instead:

In [54]: %timeit ntc2(1)
79 ns ± 0.37 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [55]: %timeit ntc(1)
126 ns ± 0.315 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [56]: %timeit nt(1)
103 ns ± 4.23 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Here ntc2 has __call__ implemented in C, ntc is the previous class-based version and nt is the current function-based version.

So perhaps we could just stick a private ._idfunc in some C-implemented module (functools? types?) and use it as the call for our NewType class.

Jelle thanks for pointing this out, I will implement helper function in C and use it as a __call__ method for NewType

New changeset c2f33df by Miss Islington (bot) in branch '3.10':
bpo-44353: Refactor typing.NewType into callable class (GH-27250) (bpo-27258)
c2f33df

New changeset 9ae5ba7 by Miss Islington (bot) in branch '3.10':
bpo-44353: Add test to cover __or__ of two NewType (GH-27259) (bpo-27261)
9ae5ba7

New changeset 96c4cbd by Yurii Karabas in branch 'main':
bpo-44353: Implement typing.NewType __call__ method in C (bpo-27262)
96c4cbd

Note: we won't be backporting _typing to Python 3.10 as it is much too late for a new module at this point in the life cycle. Consequently, 3.10 will be a (temporary) performance regression in terms of typing.NewType.

Thanks everyone, most of all Josepth for reporting and Yurii for implementing the required changes.

PR27262 introduced reference leaks and currently all refleak buildbots are red:

Example:

https://buildbot.python.org/all/#/builders/205/builds/102/steps/5/logs/stdio

Ran 367 tests in 0.198s
OK (skipped=1)
.
test_typing leaked [220, 220, 220] references, sum=660
test_typing leaked [69, 68, 68] memory blocks, sum=205
1 test failed again:
test_typing
== Tests result: FAILURE then FAILURE ==
Per our buildbot policy, if a fix is not performed in 24 h we will need to revert

New changeset 8f42106 by Yurii Karabas in branch 'main':
bpo-44353: Fix memory leak introduced by #71449 (GH-27305)
8f42106

PR 27311 copies tests from PR 9951, adds support of nested NewTypes (with __qualname__ containing multiple components) an makes them pickleable by name as functions and classes.

New changeset e89ef0a by Serhiy Storchaka in branch 'main':
bpo-44353: Expand NewType tests for complex __qualname__. (bpo-27311)
e89ef0a

New changeset 7aac3f6 by Ken Jin in branch 'main':
bpo-44353: Document that typing.NewType is now a class (bpo-27319)
7aac3f6

New changeset d15949a by Miss Islington (bot) in branch '3.10':
bpo-44353: Document that typing.NewType is now a class (GH-27319) (GH-27321)
d15949a

New changeset a22b05d by Yurii Karabas in branch 'main':
bpo-44353: Improve tests covering typing.NewType pickling (GH-27302)
a22b05d

New changeset 05f5d8e by Łukasz Langa in branch '3.10':
[3.10] bpo-44353: Expand NewType tests for complex __qualname__ (GH-27311) (GH-27326)
05f5d8e

New changeset e8c0174 by Miss Islington (bot) in branch '3.10':
bpo-44353: Improve tests covering typing.NewType pickling (GH-27302) (GH-27328)
e8c0174

The buildbots are still not green. Unfortunately we will need to revert all PRs if this is not fixed soon as this is already making other issues

@pablo, I don't think this change is causing the buildbots to fail. The test failure on all the currently failing buildbots is:

======================================================================
ERROR: test_absolute_circular_submodule (test.test_import.CircularImportTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.11/test/test_import/__init__.py", line 1355, in test_absolute_circular_submodule
    import test.test_import.data.circular_imports.subpkg2.parent
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'test.test_import.data.circular_imports.subpkg2'

Which seems to be #71486.

Pick any of the failing refleak buildbots. For example:

https://buildbot.python.org/all/#/builders/280/builds/99

You will see:

----------------------------------------------------------------------
Ran 106 tests in 0.074s
OK
......
test_types leaked [2, 2, 2] references, sum=6
test_types leaked [1, 1, 1] memory blocks, sum=3
1 test failed again:
test_types

types is not typing. This is a different failure from what we've seen before. I'll fix that today.

Bisecting points to:

fe13f0b is the first bad commit
commit fe13f0b
Author: Yurii Karabas <1998uriyyo@gmail.com>
Date: Fri Jul 23 12:47:00 2021 +0300

bpo-44676: Add ability to serialize types.Union (GH-27244)

Lib/test/test_types.py | 35 +++++++++++++++
Lib/typing.py | 4 +-
.../2021-07-19-19-53-46.bpo-44676.WgIMvh.rst | 2 +
Objects/unionobject.c | 51 ++++++++++++++++++++++
4 files changed, 90 insertions(+), 2 deletions(-)
create mode 100644 Misc/NEWS.d/next/Core and Builtins/2021-07-19-19-53-46.bpo-44676.WgIMvh.rst
bisect run success

I am closing this one again, apologies for the confusion.

Thanks a bunch Yurii, Serhiy, Jelle, Łukasz and Pablo for working on this! I'm re-closing this issue. *Fingers-crossed* we won't have to open this again ;-).

New changeset 93c65df by Alex Waygood in branch 'main':
bpo-44353: Correct docstring for NewType (bpo-29785)
93c65df

New changeset 3f024e2 by Miss Islington (bot) in branch '3.10':
bpo-44353: Correct docstring for NewType (GH-29785)
3f024e2