I'm fine with limiting MAX_NFRAME to USHRT_MAX and change traceback_t.nframe type to unsigned short (16 bits). Storing 65536 should be way enough. It would be great if you manage to add your new field without making traceback_t larger.

In fact, I didn't check: maybe traceback_t size is not change by your PR, because of memory aligment and padding.

By the way, first I taught that your modified trace_t: no, you modified traceback_t. _tracemalloc ensures that a traceback_t instance is allocated exactly once in the heap memory, to reduce the memory footprint. So making traceback_t larger should impact less the memory than making trace_t larger.