As discussed before, there will be two PRs. One for the core components, and the second one (large) for typing updates. I will open the first PR shortly.

As for __class_getitem__, why not implement type.__getitem__ instead of hacking PyObject_GetItem()?

For reference: PEP-560.

As for __class_getitem__, why not implement type.__getitem__ instead of hacking PyObject_GetItem()?

This question was raised by Mark Shannon on python-dev. Actually, my initial implementation did exactly this, but I didn't like it for two reasons:

1. dir(type) is already very long, I don't want to make it even longer.
2. Some users may be surprised that although type.__getitem__ is defined, int[something] will rise "TypeError: type object is not subscriptable"

Mark however disappeared since then, so I don't know what is his current opinion. I am however not as sure about this now. If there are some arguments for this, then I can revert to the older approach (implementing type.__getitem__).

Guido, what is your preference for the implementation of __class_getitem__: PyObject_GetItem (current one) or type.__getitem__ (old one)? Can we just go ahead with the version you like and then re-consider if some objections will appear? I am asking because you are going on vacation soon and there is a second part of implementation -- changes in typing (and someone needs to review them :-)

I like the current approach (PyObject_GetItem). I agree with both of your reasons to prefer it.

This adds an overhead to every indexing.

$ ./python -m perf timeit --compare-to=./python0 -s 'a = [1]' --duplicate=10000 'a[0]'
python0: ..................... 15.3 ns +- 0.5 ns
python: ..................... 16.1 ns +- 0.3 ns

Mean +- std dev: [python0] 15.3 ns +- 0.5 ns -> [python] 16.1 ns +- 0.3 ns: 1.05x slower (+5%)

$ ./python -m perf timeit --compare-to=./python0 -s 'd = {1: 2}' --duplicate=10000 'd[1]'
python0: ..................... 17.6 ns +- 0.6 ns
python: ..................... 18.4 ns +- 0.5 ns

Mean +- std dev: [python0] 17.6 ns +- 0.6 ns -> [python] 18.4 ns +- 0.5 ns: 1.05x slower (+5%)

This adds an overhead to every indexing.

I'm not doubting your results, but I'm curious how that happened. The extra
code in PyObject_GetItem is right before the point where it would otherwise
raise TypeError -- I presume you benchmark never reached that point. Could
it be that the compiler ends up rearranging the code in a way that is less
optimal for the instruction cache?

I think this is a cause. The difference may be dependent on the compiler and platform.

New changeset 2b5fd1e by Ivan Levkivskyi in branch 'master':
bpo-32226: Implementation of PEP-560 (core components) (bpo-4732)
2b5fd1e

Interesting. I have noticed similar when I tried to add a fast loop proposed by Serhiy. It should save at least one pointer comparison for base class, but sometimes it actually led to slow-downs (although very small). How can one fight this kind of problems?

I think the best thing to do is not to panic! Also maybe PGO could help?

On Dec 14, 2017 15:43, "Ivan Levkivskyi" <report@bugs.python.org> wrote:

Ivan Levkivskyi <levkivskyi@gmail.com> added the comment:

Interesting. I have noticed similar when I tried to add a fast loop
proposed by Serhiy. It should save at least one pointer comparison for base
class, but sometimes it actually led to slow-downs (although very small).
How can one fight this kind of problems?

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue32226\>

---

It is very to get stable benchmarks with timings shorter than 1 ms, and even harder for timings shorter than 1 us.

I wrote documentation to implement how to get more stable results:
http://perf.readthedocs.io/en/latest/run_benchmark.html#how-to-get-reproductible-benchmark-results

On Linux, using CPU isolation helps a lot, and perf uses automatically isolated CPUs.

New changeset ce5b0e9 by Serhiy Storchaka in branch 'master':
bpo-32226: Make __class_getitem__ an automatic class method. (bpo-5098)
ce5b0e9

Please don't forgot to document this feature.

Can I help?

I'm implementing the feature related to pickling classes, and want to implement pickling generic types as an example. But since PEP-560 will change all here, I need to wait this change.

Serhiy,

I am sorry for a delay, I have recently moved to another country, this is why I have not much time. I will try to work on this weekend. Here are some points where you can be helpful:

• Make a short documentation PR for __mro_entry__ and __base_subclass__ methods.
• Document the __class_getitem__ C API calling convention (with motivation, many classes are implemented in C but are generic in nature, like numpy.ndarray) in PEP-560.

(I know this might be not very interesting, but this will save time for me to focus only on typing part.)

Sorry, I know very small about this module to writing the documentation.

Since the commit 45700fb, test_genericclass leaks references:
---
vstinner@apu$ ./python -m test -R 3:3 test_genericclass -m test.test_genericclass.CAPITest.test_c_class
Run tests sequentially
0:00:00 load avg: 1.27 [1/1] test_genericclass
beginning 6 repetitions
123456
......
test_genericclass leaked [2, 3, 2] memory blocks, sum=7
test_genericclass failed

1 test failed:
test_genericclass

Total duration: 107 ms
Tests result: FAILURE
---

The leak comes from _testcapi.Generic[int]. Maybe from generic_alias_new() of Modules/_testcapi.c.

Oh ok, I think that I found the bug: PR 5212.

New changeset e860089 by Victor Stinner in branch 'master':
bpo-32226: Fix memory leak in generic_alias_dealloc() (bpo-5212)
e860089

New changeset d911e40 by Ivan Levkivskyi in branch 'master':
bpo-32226: PEP-560: improve typing module (bpo-4906)
d911e40

What's the status of this? The issue is still open with "release blocker" status.

I think all the critical things have been implemented/fixed. There are unfortunately no docs yet (my fault), also there are two bugs related to PEP-560, but they are not new, they also exist in Python 3.6.

I am however marking this as high priority, since these two bugs prevent progress on other projects (pickling generic classes), so it would be good to fix them before beta-2.

Are there more specific descriptions of those bugs? Links to a tracker?
(Even if it's the typing tracker.)

python/typing#512 and python/typing#511 are first issue (they are closely related to each other). This is not directly related to PEP-560, but changes in typing because of it will allow to fix them in a simple manner.

The second one is Union simplification (was discussed in few PRs on typing tracker). This is not really a bug, but something we should decide on before 3.7 final. It will be difficult to change it later. Again, with PEP-560 in place the corresponding refactoring for removal should be straightforward.

In addition, there is python/typing#392, this is already almost fixed by the typing PR implementing PEP-560, but it doesn't work for dunder attributes (I think we should not relay only specific set of reserved names, rather than all dunders).

ISTM that those ought to be separate issues since PEP-560 has been
implemented fully here.

Also I am getting cold feet about Union simplification (esp. this late in
the release cycle). We should probably retreat on the typing tracker and
discuss why we're considering it -- is there evidence (even anecdotal) that
it's harmful?

OK, I will close this issue, and open separate issues for documentation, Union, etc.