I have described the error message I got when I recently installed Python 3.5.0 (via Anaconda3) and then subsequently tried to import mathplotlib.pyplot and seaborn packages at this StackOverflow post (http://stackoverflow.com/questions/34004063/error-on-import-matplotlib-pyplot-on-anaconda3-for-windows-10-home-64-bit-pc)

Another person responded to the above post with a simple 1-line patch to the file "fontList.py3k.cache" (please see the above post for details).

This does appear to be a purely Python 3 issue on some Windows platforms. It would be nice if Python 3 developers could check the patch out for any possible side-effects, and incorporate a permanent fix in future versions.  Thanks.

The stackoverflow comment is this:

"The issue is that winreg.EnumValue is not cutting string values at their length properly for some reason, and strings will include null characters which os.path.abspath is not able to process."

The "one line patch" in the stackoverflow comment is against matplotlib, not python, so it doesn't help toward fixing this.

What would be most useful toward fixing this is a reproducer, but since that will depend on data in the registry whoever creates the reproducer may as well do it as a unit test.

I'm adding 2.7 to versions since it seems unlikely the code is different there, but I haven't checked.  Anshul, if you have tested this on python2.7 and not seen the problem there, please update the issue with that info.

Before I got the "one line patch" on stackoverflow, I tried creating a new environment with Python 2.7.10 and did *not* get the error message I got with Python 3.5.0.  Here's an outline of what I did:

1) Created a new environment to use Python 2.7.10: conda create --name python2.7.10 python=2.7.10
2) Installed matplotlib package within it: conda install --name python2.7.10 matplotlib
3) Installed seaborn package within it: conda install --name python2.7.10 seaborn

When I started Python within this environment (verified it was version 2.7.10 with "print(sys.version)"), and did "import matplotlib.pyplot" and "import seaborn", the FileNotFoundError message did not appear.  So the problem appears to be associated with Python3, not Python2.

Hope this helps.

Based on matplotlib's win32InstalledFonts function [1], I created a small test to check the data strings returned by winreg.EnumValue for the presence of null characters. I tested on Windows 7 and 10 but couldn't reproduce the problem. Please run nullcheck.py to check the results on your system.

[1]: https://github.com/matplotlib/matplotlib/blob/v1.5.0/lib/matplotlib/font_manager.py#L200

Please tell me where I can find nullcheck.py and how I should run it (I assume something like "python nullcheck.py" in the Windows Command Prompt?)  I'm relatively new to Python, so explicit instructions would be appreciated.  Thanks!

Anshul, it's attached to this issue, before messages start under Files heading. Here is the direct link https://bugs.python.org/file41208/nullcheck.py

You should be able to run nullcheck.py in the command prompt by changing to the directory where it's saved and entering nullcheck.py. For example, if you saved it in your Downloads folder:

    >cd /d C:\Users\Anshul\Downloads
    >nullcheck.py

If that fails because the .py file association isn't set up properly, instead try

    >py nullcheck.py

Ok thanks.  Here's what I got:

C:\Users\Anshul\Downloads\Python>conda info --envs
# conda environments:
#
python2.7.10             C:\Anaconda3\envs\python2.7.10
root                  *  C:\Anaconda3


C:\Users\Anshul\Downloads\Python>python -V
Python 3.5.0 :: Anaconda 2.4.0 (64-bit)

C:\Users\Anshul\Downloads\Python>python nullcheck.py
Checking: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Fonts
('tahoma.ttf\x000\x000\x00\x00\x00', 1) Tahoma (TrueType)
('verdref.ttf\x000\x000\x00PA\x00\x00\x00\x00', 1) Verdana Ref (TrueType)
('grgaref.ttf\x000\x000\x00PA\x00\x00\x00\x00', 1) Georgia Ref (TrueType)
('refspec.ttf\x000\x000\x00PA\x10\x00%APP', 1) RefSpecialty (TrueType)
Cannot open: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Fonts
done

C:\Users\Anshul\Downloads\Python>activate python2.7.10
Activating environment "C:\Anaconda3\envs\python2.7.10"...

[python2.7] C:\Users\Anshul\Downloads\Python>python -V
Python 2.7.10 :: Continuum Analytics, Inc.

[python2.7] C:\Users\Anshul\Downloads\Python>python nullcheck.py
Traceback (most recent call last):
  File "nullcheck.py", line 3, in <module>
    import winreg
ImportError: No module named winreg

[python2.7] C:\Users\Anshul\Downloads\Python>

I only wrote it for Python 3, but it would be interesting to see what you get with Python 2. Please try nullcheck2.py.

Here it is:

[python2.7] C:\Users\Anshul\Downloads\Python>python -V
Python 2.7.10 :: Continuum Analytics, Inc.

[python2.7] C:\Users\Anshul\Downloads\Python>python nullcheck2.py
Checking: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Fonts
(u'tahoma.ttf\x000\x000\x00\x00\x00', 1), Tahoma (TrueType)
(u'verdref.ttf\x000\x000\x00PA\x00\x00\x00\x00', 1), Verdana Ref (TrueType)
(u'grgaref.ttf\x000\x000\x00PA\x00\x00\x00\x00', 1), Georgia Ref (TrueType)
(u'refspec.ttf\x000\x000\x00PA\x10\x00%APP', 1), RefSpecialty (TrueType)
Cannot open: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Fonts
done

[python2.7] C:\Users\Anshul\Downloads\Python>activate
Deactivating environment "C:\Anaconda3\envs\python2.7.10"...
Activating environment "C:\Anaconda3\Scripts\..\"...

C:\Users\Anshul\Downloads\Python>python -V
Python 3.5.0 :: Anaconda 2.4.0 (64-bit)

C:\Users\Anshul\Downloads\Python>python nullcheck2.py
Checking: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Fonts
('tahoma.ttf\x000\x000\x00\x00\x00', 1), Tahoma (TrueType)
('verdref.ttf\x000\x000\x00PA\x00\x00\x00\x00', 1), Verdana Ref (TrueType)
('grgaref.ttf\x000\x000\x00PA\x00\x00\x00\x00', 1), Georgia Ref (TrueType)
('refspec.ttf\x000\x000\x00PA\x10\x00%APP', 1), RefSpecialty (TrueType)
Cannot open: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Fonts
done

C:\Users\Anshul\Downloads\Python>

Here's a patch for Python 3 that modifies the Reg2Py function in PC/winreg.c for the case of REG_SZ/REG_EXPAND_SZ. The existing code took a conservative approach by only removing a null character at the end of a buffer. I modified it to use wcsnlen instead.

I added a test that depends on setting a string containing null values via winreg.SetValueEx. It could be rewritten using ctypes if it's desired to also prevent the setting of strings that contain null values. Note that the patch also fixes an error in the delete_tree method. The desired access for OpenKey was mistakenly being passed as the "reserved" argument.

Thanks Eryk, but I'm afraid I didn't follow most of what you said in your post.  As an end-user, I just wanted to be able to install the packages, import them, and then use within Python.  It is still not clear to me whether this is a problem with Python3, or a Matplotlib/Seaborn problem (and if it is the latter, are those folks informed?)

Do you need me to do any further tests with your patch - if so, kindly give detailed step-by-step instructions.  If not, please confirm that your patch will simply be incorporated in future releases of Python3, so that users wouldn't have to resort to the "one line patch" mentioned in my very first post.  Thanks for all your help!

Left one minor comment on Rietveld.

However, I'm not yet convinced that this is a bug in Python.  I can't find an authoritative source to say whether \0 is valid in a REG_SZ value, but the fact that you can set one and get it back makes me think it is (for certain values of 'valid').  I don't see why the original issue with matplotlib would only happen with Python 3 and not Python 2, though.

Also, running nullcheck.py on my 8.1 VM gave no results.

> I don't see why the original issue with matplotlib would only happen with Python 3 and not Python 2, though.

Is it possible that Python 2 is using a non-unicode Windows API to get the values, and the non-unicode API is converting the string (which is, of course, in UTF-16 in the registry itself) in a way that ignores characters after the first null? (And presumably likewise after the first double null in a REG_MULTI_SZ)

This would be a pretty strong argument that embedded nulls aren't meant to be acceptable in REG_SZ values (as anything other than ignored garbage).

> Is it possible that Python 2 is using a non-unicode Windows API to get the
> values

It does, but according to Anshul's nullcheck2.py output that doesn't matter.

All registry values are blobs and it's up to the caller to decide how to read it - the "types" are hints and are not binding.

I don't mind the default behavior being to trim at the first null, but it is a lossy operation on our side and so we really ought to provide an easy way to get the true value back. Otherwise, we couldn't easily write scripts in Python to detect registry values with embedded nulls :)

> All registry values are blobs and it's up to the caller to decide how to read it - the "types" are hints and are not binding.

Just out of curiosity, what do we do if a REG_DWORD is more (or less?) than four bytes? Can that happen?

So then this is actually matplotlib's bug: Python is returning the blob as it should, but matplotlib is assuming too much about what it contains.  However, I suspect that this particular issue isn't entirely matplotlib's fault; it looks to me like the entries nullcheck2.py turned up were inserted badly.  Most of them look almost (but not quite...) like REG_MULTI_SZ entries.

The winreg module is a low-level interface; I don't think we should change this behavior.

> The winreg module is a low-level interface

High enough to return a string instead of bytes, an int for REG_DWORD, and to parse out REG_MULTI_SZ into a list. Maybe for if people really want the blob there should be an interface that always returns bytes and doesn't strip *any* null, even the one (if present) at the end.

REG_SZ and REG_EXPAND_SZ are documented as null-terminated strings [1], which shouldn't have an embedded null character. As such, the default result in the high-level environment of Python shouldn't have embedded nulls. However, since the buffer is sized, I think setting a REG_SZ or REG_EXPAND_SZ value with embedded nulls should be allowed. (My test function relies on this. :)

winreg could add a QueryRawValue[Ex] function that always returns the data as bytes. It just has to pass the type as REG_BINARY to Reg2PY:

        case REG_BINARY:
        /* ALSO handle ALL unknown data types here.  Even if we can't
           support it natively, we should handle the bits. */
        default:
            if (retDataSize == 0) {
                Py_INCREF(Py_None);
                obData = Py_None;
            }
            else
                obData = PyBytes_FromStringAndSize(
                             (char *)retDataBuf, retDataSize);

The problem with REG_SZ also exists in Python 2, but I didn't backport the patch. However, Anshul's error occurs in os.path.abspath, which doesn't mind nulls in Python 2:

    >>> os.path.abspath(u'a string\x00 with a null')
    u'Z:\\Temp\\a string'

whereas Python 3's implementation calls the built-in function os.path._getfullpathname. This uses the path_converter function defined in Modules/posixmodule.c, which rejects paths that have an embedded null:

    >>> os.path.abspath('a string\x00 with a null')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Program Files\Python 3.5\lib\ntpath.py", line 535, in abspath
        path = _getfullpathname(path)
    ValueError: _getfullpathname: embedded null character


[1]: https://msdn.microsoft.com/en-us/library/ms724884

> add a QueryRawValue[Ex] function 

Or QueryValueEx and EnumValue could take a boolean argument named "raw" or "binary", which if True forces the data to be returned as REG_BINARY regardless of its assigned type.

Anshul: you don't need to worry about this technical discussion unless our windows experts decide this is in fact a bug in matplotlib and/or your machine's registry :)

There's also backward compatibility to consider here; someone somewhere is depending on getting back \0 in his REG_SZes.  If we were to make a change, it could only go in 3.6.  With that restriction, I don't see a particularly nice way to get the same behavior from 3.6+ and 3.5-.

Also, I don't like the idea of having a mismatch between what we set and what we get, even if what we're setting technically shouldn't be allowed.

I'm more inclined to just document that REG_SZ and REG_EXPAND_SZ values may contain nulls if the key was added improperly.

Thanks David, and yes I had guessed as much.  Will continue to monitor these emails for a while, in case action is needed from my side.

From an end user perspective, it was discouraging to encounter a bug like this at such an early stage of using Python3 (and associated packages) as a tool for Data Analysis, especially since it's been around for a few years.  Glad to see such a spirited debugging effort.  Hoping for a successful outcome.  :-)

Your "backward compatibility" argument makes me think of https://xkcd.com/1172/

99% of programs written in C or other languages will cut the value off at the first null. One consequence of which is that no-one (except an unfortunate Python program) will _notice_ that one "was added improperly", which makes Python the squeaky wheel for breaking when encountering a value nothing else had any problem with.

This behavior is "out-there" enough that I think the claim that someone is relying on it should be justified with a concrete example.

> I don't like the idea of having a mismatch between what we set and
> what we get, even if what we're setting technically shouldn't be 
> allowed.

Currently if you set a string with null, you won't see it using either regedit.exe or reg.exe:

    >>> import os, winreg
    >>> data = "a string\x00 with a null"
    >>> HKCU = winreg.HKEY_CURRENT_USER
    >>> winreg.SetValueEx(HKCU, "test", 0, winreg.REG_SZ, data)
    >>> winreg.QueryValueEx(HKCU, "test")
    ('a string\x00 with a null', 1)

    >>> os.system('reg query HKCU /v test')

    HKEY_CURRENT_USER
        test    REG_SZ    a string

    0

The registry saves the whole buffer, agnostic to the type, but clearly Microsoft has documented and treats REG_SZ as a null-terminated string. We should follow suit.

Ok, that example is an argument I'll accept.

I'll give the patch another couple of days for others to review, and commit if nobody beats me to it.  It's too late for 3.5.1, but it can probably make 3.4.4.

Missed the boat on 3.4.

I tried out the new test on 2.7, and it does not fail with no change to _winreg.c.  Eryk, do you want to provide a patch for 2.7?  I'm not interested enough to figure out what's going on there, but if you provide a patch I'll commit it.

The current test works for 3.x because we keep the full string length via PyUnicode_AsWideCharString(value, &len) when creating a REG_SZ value. OTOH, 2.x Py2Reg gets the length via strlen. I'd prefer to make this consistent with 3.x by using the full string length from PyString_AsStringAndSize. Expanding the range of supported values is less disruptive to existing code.

I've added a patch for 2.7 that updates Py2Reg to use PyString_GET_SIZE instead of strlen and updates Reg2Py to use strnlen instead of returning strings with embedded NULs.

Zach - you still got this or do you want me to commit it?

Go for it :)

The last message on this thread was in January, and this item is Open. According to Pep 478, 3.5.2 final was released Sunday, June 26, 2016.

How is this issue not a release blocker?

Why does there appear to be no urgency in fixing this bug?

This bug bites for 64-bit versions of Python 3. When it bit, it caused Leo to crash during startup. When it bit, it was reason to recommend Python 2 over Python 3.

I have just released an ugly workaround in Leo. So now Leo itself can start up, but there is no guarantee that user plugins and scripts will work.

Imo, no future version of Python 3 should go out the door until this bug is fixed, for sure, and for all time. If you want people to use Python 3, it can NOT have this kind of bug in it.

Looks like it fell off everyone's radar.    It is now also too late in the 3.6 release cycle: it isn't "release critical" in the sense of being a breaking regression from 3.5, so it is not likely to get in.

Keep in mind that this is a distributed, mostly volunteer community.  Clearly no one else thought it was a release blocker, but you have presented a case for making it so.  I haven't evaluated that case; I'm leaving that to the windows devs.  Having your input a few weeks ago would have made it more likely to get in to 3.6, regardless of whether or not anyone else thought it was a release blocker.

And please check back after the 3.6 release, and advocate for getting it in to the next release of both 3.5 and 3.6.

This looks like it was my fault - I said I'd commit the patches and then never got back to it.

I've assigned the issue to me so it doesn't fall off my radar again (though if someone else wants to do it first they're welcome to). But at this stage I suspect it's best not to change 3.6.0.

On Sat, Dec 3, 2016 at 1:37 PM, Steve Dower <report@bugs.python.org> wrote:

​Thanks, Steve and David, for your replies. Getting this issue fixed
eventually will do.​ Glad to hear it was a mistake, and not policy ;-)

EKR

New changeset 8cee4862fb34 by Steve Dower in branch '3.6':
Issue #25778: winreg does not truncase string correctly (Patch by Eryk Sun)
https://hg.python.org/cpython/rev/8cee4862fb34

New changeset 3014854e68e4 by Steve Dower in branch 'default':
Issue #25778: winreg does not truncase string correctly (Patch by Eryk Sun)
https://hg.python.org/cpython/rev/3014854e68e4

I've committed to 3.6.1 and 3.7, but I'm not convinced that the problem is worth the risk of changing the behaviour in 2.7.14 and 3.5.3.

Considering we kept forgetting to commit this for so long and largely got away with it, I assume there aren't hundreds of people being bitten by this on a daily basis. There are also other ways to work around the problems that have occurred (editing and resaving the values in Registry Editor should suffice, and the attached nullcheck scripts can help find the problematic values).

So while 2.7 and 3.5 will continue to not trim REG_SZ values at the first embedded null, I think that's best for preserving the ability of users to upgrade easily between microversions (particularly for 2.7).

Thank you, Steve, et. al. for resolving this issue.