New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mimetypes initialization fails on Windows because of non-Latin characters in registry #53537
Comments
On Windows, mimetypes initialization reads the list of MIME types from the Windows registry. It assumes that all characters are Latin-1 encoded, and fails when it's not the case, with the following exception: Traceback (most recent call last):
File "mttest.py", line 3, in <module>
mimetypes.init()
File "c:\Python27\lib\mimetypes.py", line 355, in init
db.read_windows_registry()
File "c:\Python27\lib\mimetypes.py", line 260, in read_windows_registry
for ctype in enum_types(mimedb):
File "c:\Python27\lib\mimetypes.py", line 250, in enum_types
ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128) This can be reproduced, for example, on a Russian Windows XP installation which has QuickTime installed (QuickTime creates the non-Latin entries in the registry). The following line causes the exception to happen: import mimetypes; mimetypes.init() |
I'm guessing this problem doesn't occur in 3.x? If so, the quick fix would be to have the registry code catch UnicodeError instead of UnicodeEncodeError. That may be the correct fix anyway. The "fun" part of this bug is going to be creating a unit test for it. |
The problem doesn't happen on Python 3.1.2 because it doesn't have the code in mimetypes that accesses the Windows registry. Haven't tried the 3.2 alphas yet. |
Patch (suggested fix and unittest) attached. |
And by the way I've verified that the problem doesn't happen in py3k trunk. |
And just for clarity: py3k trunk does contain the _winreg code path. |
python 3.1.2 mimetypes initialization also fails in redhat linux: >>> import http.server
Traceback (most recent call last):
File "/home/public/i386-redhat-linux-gnu/python/lib/python3.1/http/server.py", line 588, in <module>
class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
File "/home/public/i386-redhat-linux-gnu/python/lib/python3.1/http/server.py", line 764, in SimpleHTTPRequestHandler
mimetypes.init() # try to read system mime.types
File "/home/public/i386-redhat-linux-gnu/python/lib/python3.1/mimetypes.py", line 305, in init
db.readfp(open(file))
File "/home/public/i386-redhat-linux-gnu/python/lib/python3.1/mimetypes.py", line 209, in readfp
line = fp.readline()
File "/home/public/i386-redhat-linux-gnu/bin/../python/lib/python3.1/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 3921: ordinal not in range(128) |
Here is a better patch. |
UnicodeDecodeException is thrown because 'ctype' is already a string, |
The encoding is wrong. We should read the registry using Unicode, or at least use the correct encoding. The correct encoding is the ANSI code page: sys.getfilesystemencoding(). Can you please try with: default_encoding = sys.getfilesystemencoding() ?
In Python 3.3, MimeTypes.read() opens files in UTF-8. The issue bpo-13025 explains why UTF-8 is used instead the locale encoding, or another encoding. I see that read_mime_types() uses the locale encoding, it looks like a bug, it should also use UTF-8. |
Only just been reminded of this one; it's possible that it's been superseded by bpo-15207. At the least, that issue resulted in a code change in this area of mimetypes. I'll have a look later. |
This does not work. In fact it doesn't matter what default_encoding is. The variable ctype, which is returned by _winreg.EnumKey(), is a byte string(b'blahblah'), at least on my computer(win2k3sp2, python 2.7.6). Because the interpreter is asked to encode a byte string, it tries to convert the byte string to unicode string first, by calling decode implicitly with 'ascii' encoding, so the exception UnicodeDecodeError. the variable ctype, which is read from registry key name, can be decoded correctly with sys.getfilesystemencoding()(which returns 'mbcs'), but in fact what we need is a byte string, so there should be neither encoding nor decoding here. if there is a case that _winreg.EnumKey() returns unicode string, then a type check should be added before the encode. Or maybe the case is that the return type of _winreg.EnumKey() is different in 2.x and 3.x? |
There is possibility that the installation of setuptools fails with I failed the installation of setuptools with Python 2.7.6 on my machine, Windows 8.1 Pro Japanese Edition 64bit, but no problem with both Python 2.7.4 and Python 3.3.3. |
OK, that means the bpo-15207 fix didn't fix it, since that's in 2.7.6. |
I'll try to look at this soonish. Thanks for bringing it back to the |
Issue bpo-20017 has been marked as a duplicate of this issue. Copy of the message: Running Windows 8 (64-bit) and Python 2.7.6 (64-bit). > python -m SimpleHTTPServer
Traceback (most recent call last):
File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\Python27\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "C:\Python27\lib\SimpleHTTPServer.py", line 27, in <module>
class SimpleHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
File "C:\Python27\lib\SimpleHTTPServer.py", line 208, in SimpleHTTPRequestHand
ler
mimetypes.init() # try to read system mime.types
File "C:\Python27\lib\mimetypes.py", line 358, in init
db.read_windows_registry()
File "C:\Python27\lib\mimetypes.py", line 258, in read_windows_registry
for subkeyname in enum_types(hkcr):
File "C:\Python27\lib\mimetypes.py", line 249, in enum_types
ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 2: ordinal
not in range(128) |
This issue affects mercurial too. |
An alternative solution, which worked for me, is: The contents of the file: The default encoding should be determined for every localized Windows version. |
The bug as reported against setuptools: https://bitbucket.org/pypa/setuptools/issue/127/unicodedecodeerror-when-install-in-windows |
I just hit this bug on 2.7.6, running on polish WinXP (I need to build some packages there, I hope I'll avoid a nasty py2exe bug). Any reasons this is not fixed yet? Do you need any assistance? |
Michał: Can you please report the exact registry key and value that is causing the problem? It's difficult to test a patch if one is not able to reproduce the problem. Of the patches suggested: does any of them fix the problem for you? If so, which one? I personally fine Vladimir's patch more plausible (EnumKeys gives bytes objects in 2.x, so it is pointless to apply .encode to them). The introduction of the count() call is unrelated, though, and should be omitted from a bug fix. |
Martin: I had the same problem after upgrading to 2.7.6. System here: German XP 32 Bit I used the solution from Alexandr with sitecustomize.py (with cp1252) and it works fine for me. |
Martin: the problematic key is "[HKEY_CLASSES_ROOT\BDATuner.Składniki]". I am pasting its name, because I suppose, that as bugs.python.org is utf-8, special characters will be pasted properly. Included you will find a .REG file, which is Windows Registry Editor file, which is plaintext. It is encoded with CP-1250 charset (I believe). In any case of confusion, I inlcude also the same file encoded with utf-8. If you add those information to your Windows registry, you should be able to reproduce this bug just by simply using "pip install" anything. "pip install wokkel", for example. |
Another REG file, encoded with CP1250, I believe. |
As for the fix, sitecustomize.py works for me, too, but I somehow believe, that adding sitecustomize.py for new Python installations would propably do more harm than good. I'll check those 2 patches and I'll let you know. |
9291.patch works for me too, but I am unsure about its idea. Silently ignoring non-ASCII registry entries - does it sound like a good idea? Maybe. Is it pythonic? I doubt so. I don't exactly understand what 9291a.patch is doing. For me it does look like a re-iteration of the first patch. I have not tested it. |
The attached patch bpo-9291.7.patch (which is essentially an amalgam of 9291.patch & 9291a.patch with some tweaks of my own) does appear to solve the issue. My Windows setup is UK, so if any of the people still watching this issue could test against a non-English Windows, that would be useful. Even this fix does leave some room for encoding mismatches between the stored values (mbcs encoded) and any string passed to guess_type. But it's not clear how that should be handled, and at least it doesn't crash out on .init. |
Another version of the patch: this one, in addition to removing the unnecessary encodes, also does the check for extensions before attempting to open the registry key, and narrows down the try-catch block to just the attempt to read the "Content Type" value. This does mean that if any process is unable to read HKCR or its subkeys the mimetypes.init will fail. Frankly, I can't see how that could happen, but if anyone feels strongly enough I can add extra guards so it fails silently. |
Alternative temporary solution |
New changeset 18cfc2a42772 by Tim Golden in branch '2.7': |
New changeset 0c8a7299c7e3 by Tim Golden in branch '2.7': |
Please see http://bugs.python.org/issue21652 for a regression introduced by this change. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: