New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add cp65001 to encodings/aliases.py #50308
Comments
Add 'cp65001' (Microsoft term for UTF-8) as an alias to 'utf_8' |
Could you provide some official reference defining the alias ? Thanks. |
Nevermind, I found this reference: http://msdn.microsoft.com/en-us/library/system.text.encoding(VS.80).aspx Looks like we could add a few more aliases for other encodings as well. |
I wouldn't trust this table. Microsoft is on record of implementing the |
Martin v. Löwis wrote:
Fair enough. Could someone with some IronPython/.NET foo check whether the The above page has some sample code to get started and IronPython Thanks,Marc-Andre Lemburg ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 |
Here's a script for IronPython 2.6 that checks a few encoders. Since IronPython doesn't appear to come with the full set of Python It's probably better to dump the encoded data to a file and compare Anyway, here's the output: Code Page 65000 vs. encoding 'utf-7' 0 errors Code Page 65001 vs. encoding 'utf-8' 0 errors Code Page 1200 vs. encoding 'utf-16-le' 0 errors Code Page 1201 vs. encoding 'utf-16-be' 0 errors Code Page 28591 vs. encoding 'iso-8859-1' 0 errors |
(I tried running your script under IronPython 2.6 with Mono but I got a |
I wrote a small C application that converts all possible Usage: cl.exe gen65001.c Except for the newline character and a sequence from Note, however, that cp65001 is a pseudo code page: http://www.postgresql.org/docs/faqs.FAQ_windows.html#2.6 For instance, setlocale will not work: http://blogs.msdn.com/michkap/archive/2006/03/13/550191.aspx |
re Martin's question, I can offer the indirect wisdom of Michael Kaplan http://blogs.msdn.com/michkap/archive/2008/03/18/8306597.aspx where he mentions that the easiest way to output unicode text in the int main(void) {
_setmode(_fileno(stdout), _O_U16TEXT);
wprintf(L"\x043a\x043e\x0448\x043a\x0430 \x65e5\x672c\x56fd\n");
return 0;
} _setmode being the special call needed. I haven't tested with any _O_U8TEXT (if such a thing exists), I don't do It also seems that Python —when stdin/stdout/stderr is under control of |
I also wonder whether stdin/stdout/stderr should be streams on Windows |
I created two scripts for exporting the IronPython findings and checking them in CPython. These are the results: Checking code Page 28591 against encoding 'iso-8859-1' using file 'iso-8859-1.map' 0 errors Checking code Page 28592 against encoding 'iso-8859-2' using file 'iso-8859-2.map' 0 errors Checking code Page 28593 against encoding 'iso-8859-3' using file 'iso-8859-3.map' 0 errors Checking code Page 28594 against encoding 'iso-8859-4' using file 'iso-8859-4.map' 0 errors Checking code Page 28595 against encoding 'iso-8859-5' using file 'iso-8859-5.map' 0 errors Checking code Page 1201 against encoding 'utf-16-be' using file 'utf-16-be.map' 2048 errors Checking code Page 1200 against encoding 'utf-16-le' using file 'utf-16-le.map' 2048 errors Checking code Page 65000 against encoding 'utf-7' using file 'utf-7.map' 21 errors Checking code Page 65001 against encoding 'utf-8' using file 'utf-8.map' 2048 errors Result: We can add aliases for the various ISO mappings, but not for the UTF ones. .NET encodes the surrogates differently than Python's codecs and |
What we could do is add new codecs based on the .NET tables for cp65000 et al. However, before doing this, I'd like to know where these code page settings can occur and what exact names are used for them. If they only appear in .NET and IronPython, I don't think it's worth adding extra codecs for the MS UTF variants. |
Would it be possible to implement a "cp65001" codec in Python using MultiByteToWideChar() / WideCharToMultiByte() with codepage=CP_UTF8? |
This problem causes {{{os.getcwdu()}}} to fail when the console code page is set to 65001 (always, I think): Microsoft Windows [Version 6.0.6002] t:\>chcp t:\>python -c "import os; print os.getcwdu()"
Traceback (most recent call last):
File "<string>", line 1, in <module>
LookupError: unknown encoding: cp65001 t:\>chcp 1252 t:\>python -c "import os; print os.getcwdu()" Incidentally, I don't agree that this codepage needs to be distinguished from UTF-8. The deviations in the Microsoft codec are just their bugs. There is only one correct way to encode/decode UTF-8, and cp65001 is supposed to be UTF-8 according to Microsoft (e.g. http://msdn.microsoft.com/en-us/library/86hf4sb8%28en-US,VS.80%29.aspx ). |
I said: "There is only one correct way to encode/decode UTF-8". This is true modulo differences in the treatment of initial byte order marks. |
I meant to say that the os.getcwdu() test in msg119440 was done with Windows native Python 2.6.2. |
Oops, false alarm. python -c "import os; print repr(os.getcwdu())" works as expected, so the exception is part of bpo-1602. (My command about there being no need to distinguish this codepage from UTF-8 stands.) |
Different tests proved that cp65001 can *not* be set as an alias to utf-8, and that's why I'm closing this issue. Anyway, I don't think that cp65001 is configured by default on any Windows setup. It is only set by the user, using the chcp command, to try to display unicode characters in the Windows console: but it is not possible to display any unicode character in this console (see issue bpo-1602). And chcp command should not be used in the Windows console because it does not only change the ANSI code page: it changes also the console code page, which is wrong (the console still expect text encoded to the previous code page). It is possible to implement a codec for cp65001 using utf-8 existing codec in surrogatepass mode, or by using MultiByteToWideChar() / WideCharToMultiByte() with codepage=CP_UTF8. But I don't think that we need cp65001 at all. If you need cp65001 for a good reason and you would like to implement a cp65001 Python codec, open a new issue. If you consider that we should use _O_U8TEXT or _O_U16TEXT, open another new issue. _O_U8TEXT or _O_U16TEXT might improve unicode support if Python output is redirected to a pipe, but I don't think that it would help to display unicode character in the Windows console. I also fear that it breaks existing code and any function not aware of this special mode. |
I added a cp65001 codec to Python 3.3: see issue bpo-13216. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: