A correction for the problem found by GvR in change 58692:

> There's one mystery: if I remove ob_sstate from the PyStringObject struct,
> some (unicode) string literals are mutilated, e.g. ('\\1', '\1') prints
> ('\\1', '\t').  This must be an out of bounds write or something that I
> can't track down.  (It doesn't help that it doesn't occur in debug mode.
> And no, make clean + recompilation doesn't help either.)
> 
> So, in the mean time, I just keep the field, renamed to 'ob_placeholder'.

I think I found the problem. It reproduces on Windows, with a slightly
different input
    >>> ('\\2','\1')
    ('\\2', '\n')
(the win32 release build is of the kind "optimized with debug info", so
using the debugger is possible)

The problem is in unicodeobject.c::PyUnicode_DecodeUnicodeEscape:
- the input buffer is not null-terminated
- when decoding octal escape, we increment s without checking if it is
still in the limits.
In my case, the "\1" was followed by a "2" in memory, hence the bogus
chr(0o12) == '\n'.

Also corrected a potential problem when the string ends with a \:
PyUnicode_DecodeUnicodeEscape("\\t", 1) should return an error.

Thanks!!

The patch as given doesn't quite work -- it gives an error message on
string literals like '\s'.  By reverting some of your code and only
keeping the bounds checks for the octal escape code I got it to work
though.  See my checkin.

Committed revision 58707.

This needs to be backported too; it seems a pure coincidence it didn't
trigger before!

(Hold on, the assert I added triggers in debug mode.)

I don't like this assert either: this function is part of the public
API, and should not crash the interpreter just because of a trailing \.

To test easily:

import ctypes
decode = ctypes.pythonapi.PyUnicodeUCS2_DecodeUnicodeEscape
decode.restype = ctypes.py_object
decode(b'\\1', 1, None)

This should gently raise a UnicodeDecodeError IMO.

Better fix:
Committed revision 58708.