I’m a bit worried about flex_complete() covering up errors. If I add calls to PyErr_WriteUnraisable(), I can see both the s1 and s2 decodes failing. Input the following line:

>>> "©"; import x;

and then move the cursor back one place, so it is directly after the “x”, but not after the semicolon (;). Then press Tab. Both errors are:

ValueError: embedded null byte

It looks like the reason is that PyUnicode_DecodeLocaleAndSize() requires that str[len] is a null character (the error message is misleading). It seems the len parameter is mainly there to verify that there are no embedded null characters, i.e. you cannot use it to give a truncated string.

It looks like Py_DecodeLocale() is used underneath; maybe it is simpler to call that directly. But it does not solve the string truncating problem.

A test case (perhaps using a pseudo-terminal) might also help pick this kind of thing up, if we can’t report errors any other way.