`typing.NewType` doesn't support PEP 604.

```
>>> import typing
>>> A = typing.NewType("A", int)
>>> B = typing.NewType("B", str)
>>> A | B
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'function' and 'function'
```

It is possible to implement NewType to return a class rather than a function as it does currently. However, the class version uses substantially more memory (1072 bytes vs 144 bytes with sys.getsizeof) and NewType is supposed to have almost no runtime overhead.

What about to return callable object instead of function from a `typing.NewType`?

It will look something like this:
```
class _NewType:
    __slots__ = ('__name__', '__supertype__')

    def __init__(self, name, supertype):
        self.__name__ = name
        self.__supertype__ = supertype

    def __call__(self, val):
        return val

    def __or__(self, other):
        return Union[self, other]

    def __ror__(self, other):
        return Union[other, self]



def NewType(name, tp):
    """NewType creates simple unique types with almost zero
    runtime overhead. NewType(name, tp) is considered a subtype of tp
    by static type checkers. At runtime, NewType(name, tp) returns
    a dummy callable object that simply returns its argument. Usage::

        UserId = NewType('UserId', int)

        def name_by_id(user_id: UserId) -> str:
            ...

        UserId('user')          # Fails type check

        name_by_id(42)          # Fails type check
        name_by_id(UserId(42))  # OK

        num = UserId(5) + 1     # type: int
    """
    return _NewType(name, tp)
``` 

With such implementation `__or__` will be available for a NewType and actually because of __slots__ size of object will be 48 bytes (with sys.getsizeof) which is less than current 144 bytes (if memory is important).

https://github.com/python/typing/issues/746 has some previous discussion of implementing NewType as a class (motivated by __repr__), including benchmarks.

Jelle I can try to implement it in C to see if it will give better performance.

@Yurii, I would like to caution against a C accelerator for the typing module. My reasoning follows:

1. The maintenance burden is higher. typing is already somewhat complex in Python (lots of MRO/metaclass wizardry). A C module would require knowledge of the C API.

2. Following from 1., I fear it may raise the barrier for contributions. This is purely anecdotal but I think there are more contributors who know Python than Python + C API.

3. C API code is much more error-prone than Python code.

4. It's very hard to find available reviewers for typing-related C changes.

5. Backports become harder due to point 3. and 4. Also C code needs much more scrutiny. If we cause some obscure bug in Python, it raises an exception; in C it potentially segfaults and kills the interpreter.

6. Third-party monkey-patching becomes harder.

Unfortunately, I can't offer a good solution to this issue at the moment either.

@Ken thanks for pointing this out.

The only solution which I see is to convert NewType into callable class but it will lead to performance degradation.

@Ken what do you think about that?

Do we have to solve it? Using `from __future__ import annotations` it should work, at least when used as an annotation. Or is that too inconvenient or inconsistent?

> Using `from __future__ import annotations` it should work

Not in type aliases or generic bases. (And if Larry's co_annotations PEP is accepted, it would break in normal annotations too.) I'd prefer to get rid of this sort of subtle difference where the obvious way to write a type only works in some contexts, though I realize that's not possible in all cases.

That means I think we should move forward with making NewTypes into instances with __call__ instead of functions, though according to my benchmarks in the typing issue that will incur a small performance penalty.

I opened PR with refactored `typing.NewType` into callable class.

Also I have added `__module__` attr to typing.NewType so it will become pickable.

Jelle, can you post more details about your benchmark?

This is what I got last year (copied from the typing issues):

In [83]: from typing import NewType

In [84]: nt = NewType("nt", int)

In [85]: class NewTypeClass:
    ...:     def __init__(self, name, supertype):
    ...:         self.name = name
    ...:         self.supertype = supertype
    ...:     def __call__(self, obj):
    ...:         return obj
    ...:     def __repr__(self):
    ...:         return f"<NewType: {self.name}>"
    ...:     

In [86]: ntc = NewTypeClass("ntc", int)

In [87]: ntc
Out[87]: <NewType: ntc>

In [88]: %timeit nt(3)
211 ns ± 2.27 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [89]: %timeit ntc(3)
253 ns ± 5.35 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)


Not sure what version of Python I used at the time. I just repeated it on 3.9.4 MacOS and got much faster time but a similar ~20% slowdown:

In [6]: %timeit nt(3)
112 ns ± 5.11 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [7]: %timeit ntc(3)
136 ns ± 2.36 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Hm, 20% isn't so bad, but one of the arguments for NewType was that it "disappears" at runtime -- which the current version does, but the class-based version doesn't quite.

Right now I don't have the cycles to think about this deeply. Maybe a few weeks after I'm back from vacation (~August 8) I will have more time.

This issue is now out of date on. After Serhiy's refactoring, any function types can be unioned.

>>> NewType('x', int)
>>> int | NewType('x', int)
int | typing.NewType.<locals>.new_type

The only problem now is that the repr is weird, but Serhiy also has a fix for that in issue34963 (PR 9951) without converting to a class.

Thus, I am closing the issue. Please go to issue44642 for a general discussion on whether union should support arbitrary functions at all, or issue34963 for the repr problem. Thanks all!

Does that work if you try to union two NewTypes?

> Does that work if you try to union two NewTypes?

Oh woops I was too eager in closing this issue -- it doesn't work. Sorry, please disregard my previous message. I'm reopening this. Thanks for the reminder Jelle.

New changeset 965dd76e9060e27e2253ba8c8d21a142b178720d by Yurii Karabas in branch 'main':
bpo-44353: Refactor typing.NewType into callable class (GH-27250)
https://github.com/python/cpython/commit/965dd76e9060e27e2253ba8c8d21a142b178720d

New changeset 4868b94c6089d457673b1ba5b5b64c2f38c435af by Yurii Karabas in branch 'main':
bpo-44353: Add test to cover __or__ of two NewType (#27259)
https://github.com/python/cpython/commit/4868b94c6089d457673b1ba5b5b64c2f38c435af

> Hm, 20% isn't so bad, but one of the arguments for NewType was that it "disappears" at runtime -- which the current version does, but the class-based version doesn't quite.

NewType was described originally in https://github.com/python/typing/issues/189 and the resulting PEP text landed in https://github.com/python/typing/pull/226/files.

I find the 20% performance impact non-ideal but ultimately worth it because:
- it solves the issue described in this bug;
- it allows us to implement issubclass/isinstance in the future if we so choose;
- it also potentially allows us to chain NewType.

Since we can't fully rely on `from __future__ import annotations` to side-step the performance cost, we don't really have a different option than to change it. The alternative is to leave it as is which makes it a non-composable pseudo-type unlike the others.

Currently adoption of NewType is relatively low, in part due to the feature's obscurity and partially because of its limits. Slower instantiation performance will at worst keep the adoption low, but it can potentially bring more users to NewType since it becomes less hacky.

I'm +1 to converting to a class as done by the PR.


> Right now I don't have the cycles to think about this deeply. Maybe a few weeks after I'm back from vacation (~August 8) I will have more time.

Sorry, I overeagerly already merged the change to `main`. Let's hold off with the 3.10 backport until you had time to think about it. If we decide against the change after all, reverting on `main` should have no negative impact since we're pre-alpha 1.

OTOH, for 3.10 the clock is ticking. Let's go for it. (If it's deemed too slow, we could eventually add an accelerator just for this.)

I found that replacing __call__ on the NewType class with an identity function written in C makes things faster instead:

In [54]: %timeit ntc2(1)
79 ns ± 0.37 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [55]: %timeit ntc(1)
126 ns ± 0.315 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [56]: %timeit nt(1)
103 ns ± 4.23 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Here ntc2 has __call__ implemented in C, ntc is the previous class-based version and nt is the current function-based version.

So perhaps we could just stick a private `._idfunc` in some C-implemented module (functools? types?) and use it as the __call__ for our NewType class.

Jelle thanks for pointing this out, I will implement helper function in C and use it as a __call__ method for NewType

New changeset c2f33dfc83ab270412bf243fb21f724037effa1a by Miss Islington (bot) in branch '3.10':
bpo-44353: Refactor typing.NewType into callable class (GH-27250) (#27258)
https://github.com/python/cpython/commit/c2f33dfc83ab270412bf243fb21f724037effa1a

New changeset 9ae5ba7dbf08d56a1b30d67fcde75532fe136d77 by Miss Islington (bot) in branch '3.10':
bpo-44353: Add test to cover __or__ of two NewType (GH-27259) (#27261)
https://github.com/python/cpython/commit/9ae5ba7dbf08d56a1b30d67fcde75532fe136d77

New changeset 96c4cbd96c769e92869c62ba898dd9eb670baa81 by Yurii Karabas in branch 'main':
bpo-44353: Implement typing.NewType __call__ method in C (#27262)
https://github.com/python/cpython/commit/96c4cbd96c769e92869c62ba898dd9eb670baa81

Note: we won't be backporting _typing to Python 3.10 as it is much too late for a new module at this point in the life cycle. Consequently, 3.10 will be a (temporary) performance regression in terms of typing.NewType.

Thanks everyone, most of all Josepth for reporting and Yurii for implementing the required changes.

PR27262 introduced reference leaks and currently all refleak buildbots are red:

Example:

https://buildbot.python.org/all/#/builders/205/builds/102/steps/5/logs/stdio

Ran 367 tests in 0.198s
OK (skipped=1)
.
test_typing leaked [220, 220, 220] references, sum=660
test_typing leaked [69, 68, 68] memory blocks, sum=205
1 test failed again:
    test_typing
== Tests result: FAILURE then FAILURE ==
Per our buildbot policy, if a fix is not performed in 24 h we will need to revert

New changeset 8f42106b5c362495f72c6ca2fa3884538e4023db by Yurii Karabas in branch 'main':
bpo-44353: Fix memory leak introduced by GH-27262 (GH-27305)
https://github.com/python/cpython/commit/8f42106b5c362495f72c6ca2fa3884538e4023db

PR 27311 copies tests from PR 9951, adds support of nested NewTypes (with __qualname__ containing multiple components) an makes them pickleable by name as functions and classes.

New changeset e89ef0ad2a299770a88ece8f7a316f7d3eb65c9f by Serhiy Storchaka in branch 'main':
bpo-44353: Expand NewType tests for complex __qualname__. (#27311)
https://github.com/python/cpython/commit/e89ef0ad2a299770a88ece8f7a316f7d3eb65c9f

New changeset 7aac3f623610cf3dffbf548a5be5bfd4fa6790a0 by Ken Jin in branch 'main':
bpo-44353: Document that typing.NewType is now a class (#27319)
https://github.com/python/cpython/commit/7aac3f623610cf3dffbf548a5be5bfd4fa6790a0

New changeset d15949a845a6db66675bca7105ad508ba9e79639 by Miss Islington (bot) in branch '3.10':
bpo-44353: Document that typing.NewType is now a class (GH-27319) (GH-27321)
https://github.com/python/cpython/commit/d15949a845a6db66675bca7105ad508ba9e79639

New changeset a22b05da87bdfb081d6aaecfce136ae8dbb8680c by Yurii Karabas in branch 'main':
bpo-44353: Improve tests covering typing.NewType pickling (GH-27302)
https://github.com/python/cpython/commit/a22b05da87bdfb081d6aaecfce136ae8dbb8680c

New changeset 05f5d8e48c86c8025d3f82285666d93e52e360f9 by Łukasz Langa in branch '3.10':
[3.10] bpo-44353: Expand NewType tests for complex __qualname__ (GH-27311) (GH-27326)
https://github.com/python/cpython/commit/05f5d8e48c86c8025d3f82285666d93e52e360f9

New changeset e8c01749c02a97a2f5b01086951bb564121c5113 by Miss Islington (bot) in branch '3.10':
bpo-44353: Improve tests covering typing.NewType pickling (GH-27302) (GH-27328)
https://github.com/python/cpython/commit/e8c01749c02a97a2f5b01086951bb564121c5113

The buildbots are still not green. Unfortunately we will need to revert all PRs if this is not fixed soon as this is already making other issues

@Pablo, I don't think this change is causing the buildbots to fail. The test failure on all the currently failing buildbots is:

======================================================================
ERROR: test_absolute_circular_submodule (test.test_import.CircularImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.11/test/test_import/__init__.py", line 1355, in test_absolute_circular_submodule
    import test.test_import.data.circular_imports.subpkg2.parent
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'test.test_import.data.circular_imports.subpkg2'
----------------------------------------------------------------------

Which seems to be GH-27299.

Pick any of the failing refleak buildbots. For example:

https://buildbot.python.org/all/#/builders/280/builds/99

You will see:

----------------------------------------------------------------------
Ran 106 tests in 0.074s
OK
......
test_types leaked [2, 2, 2] references, sum=6
test_types leaked [1, 1, 1] memory blocks, sum=3
1 test failed again:
    test_types

types is not typing. This is a different failure from what we've seen before. I'll fix that today.

Bisecting points to:

fe13f0b0f696464dd6f283576668dbf57cb11399 is the first bad commit
commit fe13f0b0f696464dd6f283576668dbf57cb11399
Author: Yurii Karabas <1998uriyyo@gmail.com>
Date:   Fri Jul 23 12:47:00 2021 +0300

    bpo-44676: Add ability to serialize types.Union (GH-27244)

 Lib/test/test_types.py                             | 35 +++++++++++++++
 Lib/typing.py                                      |  4 +-
 .../2021-07-19-19-53-46.bpo-44676.WgIMvh.rst       |  2 +
 Objects/unionobject.c                              | 51 ++++++++++++++++++++++
 4 files changed, 90 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2021-07-19-19-53-46.bpo-44676.WgIMvh.rst
bisect run success

I am closing this one again, apologies for the confusion.

Thanks a bunch Yurii, Serhiy, Jelle, Łukasz and Pablo for working on this! I'm re-closing this issue. *Fingers-crossed* we won't have to open this again ;-).

New changeset 93c65df83cef71a4bc77d71afecdec8744c4f73a by Alex Waygood in branch 'main':
bpo-44353: Correct docstring for `NewType` (#29785)
https://github.com/python/cpython/commit/93c65df83cef71a4bc77d71afecdec8744c4f73a

New changeset 3f024e27c29a57dd4f805aa2431d713ed0fe57b2 by Miss Islington (bot) in branch '3.10':
bpo-44353: Correct docstring for `NewType` (GH-29785)
https://github.com/python/cpython/commit/3f024e27c29a57dd4f805aa2431d713ed0fe57b2