Consider this reproducer.py:

import sys
LEN = int(sys.argv[1])

with open('big_dict.py', 'w') as f:
    print('INTS = {', file=f)
    for i in range(LEN):
        print(f'    {i}: None,', file=f)
    print('}', file=f)


import big_dict
assert len(big_dict.INTS) == LEN, len(big_dict.INTS)

And run it with any number > 65535:

$ python3.9 reproducer.py 65536
Traceback (most recent call last):
  File "/tmp/reproducer.py", line 12, in <module>
    assert len(big_dict.INTS) == LEN, len(big_dict.INTS)
AssertionError: 65535

This has not happened on python 3.8. This also happens with PYTHONOLDPARSER=1.

Also reproduces with today's git.

It appears that the 65535 key is missing regardless of the LEN value.

Bisect says 8a4cd70 is the first bad commit. Considering that 0xFFFF appears a few times in that patch, that seems plausible ;)

I think what's happening is that in 'compiler_dict' (Python/compile.c), it's checking whether 'elements' has reached a maximum (0xFFFF). However, it's not doing this after incrementing; instead, it's checking before incrementing and resetting 'elements' to 0 when it should be resetting to 1. The 65535th element isn't counted.

New changeset c51db0e by Pablo Galindo in branch 'master':
bpo-41531: Fix compilation of dict literals with more than 0xFFFF elements (GH-21850)
c51db0e

@hroncok,

How did you discover this issue?

I'd like to clean up the code for creating dictionary literals and it might be helpful to know where such huge dictionary literals exist.
I'm guessing that they are used as lookup tables for things like Unicode code-point tables, and that they would only include constants.

@hroncok said on Twitter it was reported at Storyyeller/enjarify#17

New changeset d64d78b by Miss Islington (bot) in branch '3.9':
bpo-41531: Fix compilation of dict literals with more than 0xFFFF elements (GH-21850) (GH-22107)
d64d78b