The curses module (only since Python 3.3), locale.strcoll(), locale.strxfrm(), time.strftime() and imp.NullImporter() (only on Windows) accept embedded null characters, whereas they convert the Unicode string to a wide character (wchar_t*) string.

The problem is that the null character truncates the string. Example:

>>> locale.strxfrm('a')
'a'
>>> locale.strxfrm('a\0b')
'a'

Attached patch fixes these functions. I wrote the patch for Python 3.3.

PyUnicode_AsWideCharString() documentation should also warn about this issue.

Here is a patch for the documentation.

I added warnings for, PyUnicode_AsWideChar*, PyUnicode_EncodeFSDefault and PyUnicode_AsUnicode*, since they're all concerned by this issue.

I removed the hints "using wcslen on the result of PyUnicode_AsWideChar*", since the resulting wchar_t strings may not be null-terminated

New changeset fa5c8cf29963 by Victor Stinner in branch '3.2':
Issue bpo-13617: Document that the result of the conversion of a Unicode object to
http://hg.python.org/cpython/rev/fa5c8cf29963

New changeset f30ac7729f2b by Victor Stinner in branch 'default':
Issue bpo-13617: Document that the result of the conversion of a Unicode object to
http://hg.python.org/cpython/rev/f30ac7729f2b

New changeset 1c4d9534263e by Victor Stinner in branch '2.7':
Issue bpo-13617: Document that the result PyUnicode_AsUnicode() and
http://hg.python.org/cpython/rev/1c4d9534263e

embedded_nul-2.patch: a more complete patch check also null byte in functions calling PyUnicode_EncodeFSDefault().

I added some comments in Rietveld.

I see other instances of the use of non-checked PyUnicode_AsWideCharString() and PyUnicode_AsUnicode().

@victor can you pick this up again please.

Could you please answer my comments Victor?

Note that this (or a very similar issue) also affects os.listdir() on Windows: os.listdir(bytes_path_with_nul) raises ValueError as expected, but os.listdir(unicode_path_with_nul) does not. Test case:

>>> import os
>>> os.mkdir('foo')
>>> os.listdir(b'foo\x00zzz')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: listdir: embedded null character in path
>>> os.listdir('foo\x00zzz')
[]

However, this is not the case on Linux, as there both calls raise an appropriate ValueError.

This needs to be fixed in posixmodule.c's path_converter() function.

I'm in the middle of implementing PEP-471 (os.scandir), so don't want to create a proper patch right now, but the fix is to add these lines in posixmodule.c path_converter() after the if (length > 32767) {...} block:

        if ((size_t)length != wcslen(wide)) {
            FORMAT_EXCEPTION(PyExc_ValueError, "embedded null character in %s");
            Py_DECREF(unicode);
            return 0;
        }

We should also add test to test_os.py like the following:

    def test_listdir_nul_in_path(self):
        self.assertRaises(ValueError, os.listdir, 'y\x00z')
        self.assertRaises(ValueError, os.listdir, b'y\x00z')

Could you update your patch Victor?

Sorry, I lost track of this issue. Feel free to update and complete my patch :-)

PR 2302 doesn't fix all issues with PyUnicode_AsWideCharString(). bpo-30708 should fix them.

While working on this issue I found a way to inject environment variables for a subprocess on Windows. Reclassified this issue as a security issue. PR 2302 fixes this. May be there are other security vulnerabilities fixed by it.

Wow, it's nice to see activity on this issue that I opened 6 years ago :-)

Sorry Serhiy, I don't have the bandwidth right now to review your change :-( In lack of review, I suggest you to just push it.

New changeset f7eae0a by Serhiy Storchaka in branch 'master':
[security] bpo-13617: Reject embedded null characters in wchar* strings. (bpo-2302)
f7eae0a

New changeset 0834905 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-13617: Reject embedded null characters in wchar* strings. (GH-2302) (bpo-2462)
0834905

New changeset ccdc09e by Serhiy Storchaka in branch 'master':
Fix compiler warnings on Windows introduced in bpo-13617. (bpo-2464)
ccdc09e

New changeset 54ba940 by Serhiy Storchaka in branch '3.5':
[3.5] bpo-13617: Reject embedded null characters in wchar* strings. (GH-2302) (bpo-2463)
54ba940

Backporting this to 2.7 requires too much work taking to account that PyArg_Parse and other argument parsing functions don't check for null characters in 2.7. The most serious security issue is fixed in bpo-30730, other cases unlikely can be used for attacks.

Thank you very much Serhiy of taking care of this bug!