classification
Title: 2to3 can produce illegal bytes literals
Type: behavior Stage: resolved
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, r.david.murray, serhiy.storchaka, vinay.sajip
Priority: normal Keywords: patch

Created on 2013-09-08 00:06 by serhiy.storchaka, last changed 2015-04-09 20:14 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
2to3_nonascii_bytes-2.7.patch serhiy.storchaka, 2013-10-03 10:46 review
Messages (9)
msg197205 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-08 00:06
In Python 2 string literals are allowed to contain non-ascii characters. In Python 3 bytes literals can contains only ascii characters, non-ascii characters should be escaped with \xXX. However 2to3 doesn't change b'...' literals and produces illegal code.

I propose new fixer which escapes non-ascii bytes in bytes literals. Here is a patch (without tests yet).
msg198888 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-10-03 10:21
Added a test.
msg198889 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-10-03 10:46
Backported to 2.7.
msg199861 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-10-14 10:48
Benjamin, what would you say about this?
msg199893 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-10-14 14:18
I think it's too bad parsing of b-prefixed strings in Python 2 doesn't reject non-ASCII characters.
msg231657 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-11-25 12:51
Now Python 2 with the -3 option warns about b-prefixed strings with non-ASCII characters (issue19656).
msg240361 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-09 17:44
Does the addition of the warning in python3 make the fixer obsolete?  In other words, should we close this issue?
msg240362 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-09 18:06
The addition of the warning in python3 is not related to this issue, because produced code even is not compiled in Python 3.

But the addition of the warning in Python 2 made this issue less important.
msg240376 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-09 20:14
Sorry, I meant the addition of the -3 warning.
History
Date User Action Args
2015-04-09 20:14:13r.david.murraysetmessages: + msg240376
2015-04-09 18:06:08serhiy.storchakasetstatus: open -> closed
resolution: rejected
messages: + msg240362

stage: patch review -> resolved
2015-04-09 17:44:25r.david.murraysetnosy: + r.david.murray
messages: + msg240361
2014-11-25 12:51:46serhiy.storchakasetmessages: + msg231657
versions: + Python 3.5, - Python 3.3
2013-10-14 14:18:00benjamin.petersonsetmessages: + msg199893
2013-10-14 10:48:40serhiy.storchakasetmessages: + msg199861
2013-10-03 10:46:07serhiy.storchakasetfiles: + 2to3_nonascii_bytes-2.7.patch

messages: + msg198889
2013-10-03 10:45:32serhiy.storchakasetfiles: - 2to3_nonascii_bytes.patch
2013-10-03 10:21:20serhiy.storchakasetfiles: + 2to3_nonascii_bytes.patch

messages: + msg198888
2013-10-03 10:20:02serhiy.storchakasetfiles: - 2to3_nonascii_bytes.patch
2013-09-13 17:49:59terry.reedysetnosy: + benjamin.peterson
2013-09-08 00:06:39serhiy.storchakacreate