classification
Title: PyUnicode_EncodeFSDefault win32 inconsistancy.
Type: behavior Stage: resolved
Components: Unicode, Windows Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, christian.heimes, ezio.melotti, haypo, ideasman42, pitrou, python-dev, shanzhengcheng
Priority: normal Keywords: patch

Created on 2012-09-04 05:24 by ideasman42, last changed 2016-03-28 11:10 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
fix_unicode.diff ideasman42, 2012-09-04 05:24 patch on python 3.3 review
fix_unicode_v2.diff ideasman42, 2014-10-14 13:13 review
gns3-errorlog shanzhengcheng, 2016-03-28 02:30
Messages (11)
msg169816 - (view) Author: Campbell Barton (ideasman42) Date: 2012-09-04 05:24
There is an inconsistency in PyUnicode_EncodeFSDefault(), on Linux its argument is checked to be unicode, and NULL is returned when its not. On windows however, this throws an assertion.

The problem with this is, in some CAPI code you may pass an argument and check for NULL as an error case, and allow the python API's exception to be exposed to the script author.

The problem here is a linux developer can use PyUnicode_EncodeFSDefault() this way, but not a windows developer.

A simplified use case below of a case where PyUnicode_EncodeFSDefault would fail.

Attached a fix so windows and posix systems behave the same.

---
const char *PyC_UnicodeAsByte(PyObject *py_str, PyObject **coerce)
{
	if (PyBytes_Check(py_str)) {
		return PyBytes_AS_STRING(py_str);
	}
	else if ((*coerce = PyUnicode_EncodeFSDefault(py_str))) {
		return PyBytes_AS_STRING(*coerce);
	}
---

Also: heres a list to the bug report in blenders tracker, where the bug was found.

https://projects.blender.org/tracker/index.php?func=detail&aid=31856&group_id=9&atid=498
msg169834 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-04 15:17
Sounds reasonable to me.
msg192793 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-10 10:04
The code has changed considerable since you have created the patch. Is this still an issue for Python 3.3?
msg192795 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-10 11:28
> Is this still an issue for Python 3.3?

Yes.
msg214543 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-03-23 03:24
msg192795 says this is still an issue so I'm not sure why it's been set to languishing as it's only 18 months old.
msg214559 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-03-23 10:05
It's not really a bug, Python behaves badly when you pass invalid parameters.

Campbell was asked to update his patch but he didn't 2 years later. So I just close the issue. Reopen a new issue with an updated patch if you want a nice exception message instead of a patch.

Or just fix your application to use Python C API correctly.
msg229288 - (view) Author: Campbell Barton (ideasman42) Date: 2014-10-14 13:13
Updated the patch for '93049:d9a3d23cf8f0'

Note, the link for the original bug report has changed: See https://developer.blender.org/T31856
msg234751 - (view) Author: Roundup Robot (python-dev) Date: 2015-01-26 15:45
New changeset e124aab5d9a0 by Victor Stinner in branch 'default':
Issue #15859: PyUnicode_EncodeFSDefault(), PyUnicode_EncodeMBCS() and
https://hg.python.org/cpython/rev/e124aab5d9a0
msg234752 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-01-26 15:48
I applied fix_unicode_v2.diff to Python 3.5.

For older Python versions, it's easy to workaround this issue: ensure explicitly that your object is a Unicode object using PyUnicode_Check().

You may write your function to call PyUnicode_EncodeFSDefault() with a PyUnicode_Check() check.

I prefer to not touch the stable 3.4 branch.
msg262548 - (view) Author: q20611152@163.com (shanzhengcheng) Date: 2016-03-28 02:30
this issue occureed when i run gns3-1.4.5 in python3.5.1 
 help me !
my system is linux 3.6.11  release version is   red flag inwise 8.0
msg262567 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-03-28 11:09
shanzhengcheng: Please don't update closed issues.

gns3-1.4.5 is not part of the Python standard library. You'll get better support by using the gns3 support channels.

If you still think that this is a bug in Python, please file a new issue by using the form at http://bugs.python.org/issue?@template=item (it would be great if you provide a reproducer without using gns3)

Thanks!
History
Date User Action Args
2016-03-28 11:10:38berker.peksagsetnosy: - paul.moore, tim.golden, zach.ware, steve.dower
2016-03-28 11:09:18berker.peksagsettitle: Fail update installation: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte -> PyUnicode_EncodeFSDefault win32 inconsistancy.
nosy: + steve.dower, christian.heimes, pitrou, python-dev, paul.moore, tim.golden, berker.peksag, zach.ware, ideasman42

messages: + msg262567

components: + Windows
stage: patch review -> resolved
2016-03-28 02:30:25shanzhengchengsetfiles: + gns3-errorlog
title: PyUnicode_EncodeFSDefault win32 inconsistancy. -> Fail update installation: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
nosy: + shanzhengcheng, - pitrou, christian.heimes, ideasman42, BreamoreBoy, python-dev

messages: + msg262548

components: - Windows
2015-01-26 15:48:24hayposetstatus: open -> closed
resolution: remind -> fixed
messages: + msg234752

versions: - Python 3.4
2015-01-26 15:45:05python-devsetnosy: + python-dev
messages: + msg234751
2014-10-14 13:13:49ideasman42setresolution: out of date -> remind
2014-10-14 13:13:32ideasman42setstatus: closed -> open
files: + fix_unicode_v2.diff
messages: + msg229288
2014-03-23 10:05:18hayposetstatus: languishing -> closed
resolution: out of date
messages: + msg214559
2014-03-23 03:24:54BreamoreBoysetnosy: + BreamoreBoy

messages: + msg214543
versions: + Python 3.5, - Python 3.2, Python 3.3
2013-07-10 11:28:34hayposetmessages: + msg192795
2013-07-10 10:04:57christian.heimessetstatus: open -> languishing
versions: + Python 3.4
nosy: + christian.heimes

messages: + msg192793
2012-09-04 15:17:10pitrousetnosy: + pitrou

messages: + msg169834
stage: patch review
2012-09-04 11:10:26ezio.melottisetnosy: + haypo
type: behavior
2012-09-04 05:25:03ideasman42setnosy: + ezio.melotti

components: + Unicode, Windows
versions: + Python 3.2, Python 3.3
2012-09-04 05:24:43ideasman42create