This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: IDNA2008 encoding is missing
Type: enhancement Stage: patch review
Components: Library (Lib), SSL Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Lukasa, SamWhited, Socob, andrei.avk, asvetlov, berker.peksag, case, christian.heimes, epicfaace, era, gregory.p.smith, loewis, lukasz.langa, marten, miss-islington, njs, r.david.murray, socketpair, underrun, wumpus
Priority: high Keywords: patch

Created on 2013-02-27 01:32 by marten, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
idna_translate.py marten, 2013-02-27 01:32
Pull Requests
URL Status Linked Edit
PR 25208 merged gregory.p.smith, 2021-04-06 07:36
PR 25210 merged miss-islington, 2021-04-06 07:56
PR 25211 merged miss-islington, 2021-04-06 07:56
Messages (35)
msg183104 - (view) Author: Marten Lehmann (marten) Date: 2013-02-27 01:32
Since Python 2.3 the idna encoding is available for Internationalized Domain Names. But the current encoding doesn't work according to the latest version of the spec.

There is a new IDNA2008 specification (RFCs 5890-5894). Although I'm not very deep into all the changes, I know that at least the nameprep has changed. For example, the German sharp S ('ß') isn't replaced by 'ss' any longer.

The attached file shows the difference between the expected translation and the actual translation.
msg183106 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-27 02:26
How are they handling interoperability?
msg183144 - (view) Author: Marten Lehmann (marten) Date: 2013-02-27 12:25
At least from the GNU people, two separate projects exists for this matter:

libidn, the original IDNA translation (http://www.gnu.org/software/libidn/)
libidn2, the IDNA2008 translation (http://www.gnu.org/software/libidn/libidn2/manual/libidn2.html)

Btw.: Does Python provide a way to decode the ASCII-representation back to UTF-8?

>>> name.encode('idna')
'xn--mller-kva.com'

>>> name.encode('idna').decode('utf-8')
u'xn--mller-kva.com'

Otherwise I'd look for Python bindings of libidn2 or idnkit-2.
msg183147 - (view) Author: Marten Lehmann (marten) Date: 2013-02-27 12:29
For the embedded Python examples, please prepend the following lines:

from __future__ import unicode_literals
name='müller.com'

So regarding interoperability: Usually you only use one implementation in your code and hopefully the latest release, but in case someone needs to old one, maybe there should be a separate encodings.idna2008 class.
msg183149 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-27 12:37
Does this mean the differences are only in the canonicalization of unicode values?  IDNA is a wire protocol, which means that an application can't know if it is being asked to decode an idna1 or idna2 string unless there's something in the protocol that tells it.  But if the differences are only on the encoding side, and an idna1 decoder will "do the right thing" with the idna2 string, then that would be interoperable.  I'll have to read the standard, but I don't have time right now :)

idna is a codec:

>>> b'xn--mller-kva.com'.decode('idna')
'müller.com'

(that's python3, it'll be a unicode string in python2, obviously).
msg183159 - (view) Author: Marten Lehmann (marten) Date: 2013-02-27 16:39
IDNA2008 should be backwards compatible. I can try to explain it in a practical example:

DENIC was the first registry that actually used IDNA2008 - at a time, where not even libidn2 officially included the changes required for it. This was mainly due to the point, that the German Latin Small Letter Sharp S ('ß') was treated differently to other German Umlauts ('ä', 'ö', 'ü') in the original IDNA spec: It was not punycoded, because the nameprep already replaced it by 'ss'. Replacing 'ß' with 'ss' is in general correct in German (e.g. if your keyboard doesn't allow to enter 'ß'), but then 'ä' would have to be replaced by 'ae', 'ö' by 'oe' and 'ü' by 'ue' as well. 

Punycoding 'ä', 'ö', 'ü', but not 'ß' was inconsistent and it wouldn't allow to register a domain name like straße.de, because it was translated to strasse.de. Therefor DENIC supported IDNA2008 very early to allow the registration of domain names containing 'ß'.

The only thing I'm aware of in this situation is, that previously straße.de was translated to strasse.de, while with IDNA2008 it's being translated to xn--strae-oqa.de. So people that have hardcoded a URL containing 'ß' and who are expecting it to be translated to 'ss' would fail, because with IDNA2008 it would be translated to a different ASCII-hostname. But those people could just change 'ß' to 'ss' in their code and everything would work again.

On the contrary, people that have registered a domain name containing 'ß' in the meantime couldn't access it right now by specifying the IDN version, because it would be translated to the wrong domain name with the current Python IDNA encoding. So the current IDNA-Encoding should be upgraded to IDNA2008.
msg183160 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-27 16:52
That doesn't sound like interoperability to me, that sounds like backward incompatibility :(.  I hope you are right that it only affects people with hardcoded domain names, but that is still an issue.

In any case, since this is a new feature it can only go into Python3.4, however we decide to do it.
msg183199 - (view) Author: Marten Lehmann (marten) Date: 2013-02-28 04:08
I found an interesting link about this issue:

http://www.unicode.org/faq/idn.html

I also checked a domain name of a client that ends with 'straße.de': IE, Firefox and Chrome still use IDNA2003, Opera already does IDNA2008.

In IDNA2008 a lot of characters aren't allowed any longer (like symbols or strike-through letters). But I think this doesn't have any practical relevance, because even while IDNA2003 formally allowed these characters, domain name registries disallowed to register internationalized domain names containing any of these characters.

Most registries restricted the allowed characters very strong, e.g. in the .de zone you cannot use Japanese characters, only those in use within the German language. Some other registries expect you to submit a language property during the domain registration and then only special characters within that language are allowed in the domain name. Also, most registries don't allow to register a domain name that mixes different languages.

So IDNA2008 is the future and hopefully shouldn't break a lot. I don't know of any real life use of the IDNA encoding other than DNS / URLs. I don't know how many existing modules in PyPI working with URLs already make use of the current encodings.idna class but I guess it would cause more work if they all would have to change their code to use name.encode('idna2008') or work with an outdated encoding in the end if unchanged than just silentely switching to IDNA2008 for encodings.idna and add encodings.idna2003 for those who really need the old one for some reason. Reminds me a bit on the range() / xrange() thing. Now the special new xrange() is the default and called just range() again. I guess in some years we'll look back on the IDNA2003/2008 transition the same way.
msg183202 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-28 04:20
Ah, excellent, that document looks like exactly what I was looking for.

Now, when someone is going to get around to working on this, I don't know.

(Note that the xrange/range change was made at the Python2/Python3 boundary, where we broke backward compatibility.  I doubt that we are ever going to do that kind of transition again, but we do have ways to phase in changes in the default behavior over time.)
msg205009 - (view) Author: (era) Date: 2013-12-02 13:07
At least the following existing domain names are rejected by the current implementation, apparently because they are not IDNA2003-compatible.

XN----NNC9BXA1KSA.COM
XN--14-CUD4D3A.COM
XN--YGB4AR5HPA.COM
XN---14-00E9E9A.COM
XN--MGB2DAM4BK.COM
XN--6-ZHCPPA1B7A.COM
XN--3-YMCCH8IVAY.COM
XN--3-YMCLXLE2A3F.COM
XN--4-ZHCJXA0E.COM
XN--014-QQEUW.COM
XN--118-Y2EK60DC2ZB.COM

As a workaround, in the code where I needed to process these, I used a fallback to string[4:].decode('punycode'); this was in a code path where I had already lowercased the string and established that string[0:4] == 'xn--'.

As a partial remedy, supporting a relaxed interpretation of the spec somehow would be useful; see also (tangentially) issue #12263.
msg205034 - (view) Author: Marten Lehmann (marten) Date: 2013-12-02 17:14
There's nice library called idna on PyPI doing idna2008: https://pypi.python.org/pypi/idna/0.1

I'd however prefer this standard encoding to be part of standard python.
msg217092 - (view) Author: Derek Wilson (underrun) Date: 2014-04-23 22:00
It is worth noting that the do exist some domains that have been registered in the past that work with IDNA2003 but not IDNA2008.

There definitely needs to be IDNA2008 support, for my use case I need to attempt IDNA2008 and then fall back to IDNA2003.

When support for IDNA2008 is added, please retain support for IDNA2003.

I would say that ideally there would be a codec that could handle both - attempt to use IDNA2008 and on error fallback to idna2003. I realize this isn't "official" but it would certainly be useful.
msg217218 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-04-26 21:20
I would propose this approach:

1. Python should implement both IDNA2008 and UTS#46, and keep IDNA2003
2. "idna" should become an alias for "idna2003".
3. The socket module and all other place that use the "idna" encoding should use "uts46" instead.
4. Pre-existing implementations of IDNA 2008 should be used as inspirations at best; Python will need a new implementation from scratch, one that puts all relevant tables into the unicodedata module if they aren't there already. This is in particular where the idna 0.1 library fails. The implementation should refer to the relevant parts of the specification, to be easily reviewable for correctness.

Contributions are welcome.
msg278493 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-10-11 14:52
I'm considering lack of IDNA 2008 a security issue for applications that perform DNS lookups and X.509 cert validation. Applications may end up connecting to the wrong machine and even validate the cert correctly.

Wrong:

>>> import socket
>>> u'straße.de'.encode('idna')
'strasse.de'
>>> socket.gethostbyname(u'straße.de'.encode('idna'))
'72.52.4.119'

Correct:
>>> import idna
>>> idna.encode(u'straße.de')
'xn--strae-oqa.de'
>>> socket.gethostbyname(idna.encode(u'straße.de'))
'81.169.145.78'
msg279904 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-11-02 08:16
I reported the issue for curl, CVE-2016-8625 https://curl.haxx.se/docs/adv_20161102K.html
msg310923 - (view) Author: Greg Lindahl (wumpus) Date: 2018-01-28 05:29
I am avoiding Python's built-in libraries as much as possible in my aiohttp-based crawler because of this issue, but I cannot open a connection to https://xn--ho-hia.de because there is an 'IDNA does not round-trip' raise in the python 3.6 library ssl.py code.

Happy to provide a code sample. I guess the 500-line async crawler in Guido's book was never used on German websites.
msg310924 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-01-28 05:44
Greg: That's bpo-28414. There's currently no motion towards builtin IDNA 2008 support (this bug), but I *think* in 3.7 the ssl module will be able to handle pre-encoded A-labels like that. I'm a little confused about the exact status right now but there's been lots of dicussion about that specific issue and I think Christian is planning to get one of the relevant PRs merged ASAP.
msg310943 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-01-28 12:15
A fix will land in 3.7 and maybe get backported to 3.6. Stay tuned!
msg310988 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-01-28 20:22
bpo-31399 has fixed hostname matching for IDNA 2003 compatible domain names. IDNA 2008 domain names with German ß are still broken, for example:

UnicodeError: ('IDNA does not round-trip', b'xn--knigsgchen-b4a3dun', b'xn--knigsgsschen-lcb0w')
msg311007 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-01-28 21:47
What we need for this issue is someone volunteering to writing the code.  Given how long it has already been, I don't think anyone already on the core team is going to pick it up.
msg311009 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-01-28 21:58
I lack the expertise and time to implement IDNA 2008 with UTS46 codec. I considered GNU libidn2, but the library requires two more helper libraries and LGPLv3 might be an issue for us.
msg311013 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-01-28 22:10
The "obvious" solution would be to move the "idna" module into the stdlib, but someone would still have to work that out, and it's clearly not happening for 3.7.
msg349642 - (view) Author: Ashwin Ramaswami (epicfaace) * Date: 2019-08-14 04:55
Why would chrome still be using IDNA 2003 to link http://straße.de to http://strasse.de?
msg349643 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-08-14 04:59
You have to ask the Chrome team.
msg349840 - (view) Author: Ashwin Ramaswami (epicfaace) * Date: 2019-08-16 01:27
So is the consensus that the best way to do this is to move the "idna" library to stdlib, or implement it from scratch?
msg349855 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-08-16 11:55
There is no consensus yet, IMHO.
There is a lack of resources for the issue.
msg349886 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-08-16 21:10
There is no consensus yet. Besides https://pypi.org/project/idna/ we could also consider to wrap libidn2 and ship it. Both PyPI idna and libidn2 have potential licensing issues. I don't like the idea to reinvent the wheel and implement our own idna2008 codec. It's not a trivial task.

Once Python has a working idna2008 encoder, we need to address integration into socket, ssl, http, and asyncio module.
msg370635 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-06-02 21:51
BPO #40845 is another case of IDNA 2003 / 2008 bug.
msg389932 - (view) Author: Derek Wilson (underrun) Date: 2021-03-31 20:51
why the downgrade from security to enhancement and critical to high?

this is a significant issue that can impact everything from phishing to TLS certificate domain validation and SNI.
msg390246 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-04-05 17:13
The issue has been waiting for contributions for 8 years now. So far nobody has shown an interested to address the problem and contribute an IDNA 2008 codec to Python's standard library.
msg390288 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-04-06 07:43
My PR merely adds a note to the docs linking to idna on pypi.  Don't get excited, it doesn't implement anything.  :P

re "Once Python has a working idna2008 encoder, we need to address integration into socket, ssl, http, and asyncio module."

... doing that _could_ be the same can of worms the browsers all had to go through?  We'd need to decide which behavior we wanted; pure? or matching what browsers do?  I suspect that is equivalent to the pypi idna https://github.com/kjd/idna 's uts46=True + transitional=True mode [*] but anyone doing this work would need to figure that out for sure if we wanted to default to behaving like browsers with the transitional compatibility mode.

That there is a need for a couple options on top of idna2008 as an encoding suggests it may not be a great fit for the Python codecs encodings system as those use a single string name.  We'd need to permute the useful possible combos of flag behavior in the names.  idna2003, idna2008, idna2008uts46, idna2008uts46transitional, and other combos of those if alternate combinations are deemed relevant.

I worry that a browser-transitional-behavior-matching situation may change over time as TLDs decide when to change their policies.  Is that an irrational fear?  Browsers are well equipped to deal with this as they've got frequent updates.  A PyPI package could as well.

[*] Browser history:

fwiw people wondering _why_ browsers like Chrome and Firefox don't "just blindly use idna2008 for everything" should go read the backwards compatibility transitional rationale and security concerns in https://bugs.chromium.org/p/chromium/issues/detail?id=61328
and https://bugzilla.mozilla.org/show_bug.cgi?id=479520

(caution: be ready to filter out the random internet whiners from those threads)
msg390291 - (view) Author: miss-islington (miss-islington) Date: 2021-04-06 07:56
New changeset 1d023e374cf96d143b065242131ddc9b889f9a1e by Gregory P. Smith in branch 'master':
bpo-17305: Link to the third-party idna package. (GH-25208)
https://github.com/python/cpython/commit/1d023e374cf96d143b065242131ddc9b889f9a1e
msg390296 - (view) Author: miss-islington (miss-islington) Date: 2021-04-06 08:19
New changeset c7ccb0ff61e443633d0c54cb18b5633a8e95b30c by Miss Islington (bot) in branch '3.9':
bpo-17305: Link to the third-party idna package. (GH-25208)
https://github.com/python/cpython/commit/c7ccb0ff61e443633d0c54cb18b5633a8e95b30c
msg391980 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-04-26 19:45
New changeset 2760a6711b0f510afbd09b19949bee786e098af9 by Miss Islington (bot) in branch '3.8':
bpo-17305: Link to the third-party idna package. (GH-25208) (#25211)
https://github.com/python/cpython/commit/2760a6711b0f510afbd09b19949bee786e098af9
msg396587 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-06-27 16:58
Maybe deprecate idna so that users are strongly prompted to consider the pypi idna?
History
Date User Action Args
2022-04-11 14:57:42adminsetgithub: 61507
2021-11-04 19:39:59casesetnosy: + case
2021-06-27 16:58:02andrei.avksetnosy: + andrei.avk
messages: + msg396587
2021-04-26 19:45:52lukasz.langasetnosy: + lukasz.langa
messages: + msg391980
2021-04-06 08:19:25miss-islingtonsetmessages: + msg390296
2021-04-06 07:56:27miss-islingtonsetpull_requests: + pull_request23950
2021-04-06 07:56:17miss-islingtonsetpull_requests: + pull_request23949
2021-04-06 07:56:10miss-islingtonsetnosy: + miss-islington
messages: + msg390291
2021-04-06 07:43:59gregory.p.smithsetmessages: + msg390288
2021-04-06 07:36:12gregory.p.smithsetkeywords: + patch
nosy: + gregory.p.smith

pull_requests: + pull_request23948
stage: needs patch -> patch review
2021-04-05 17:13:16christian.heimessetmessages: + msg390246
2021-03-31 20:51:11underrunsetmessages: + msg389932
2021-03-31 20:00:43christian.heimessetpriority: critical -> high
type: security -> enhancement
versions: + Python 3.10, - Python 3.8, Python 3.9
2020-12-02 12:18:41christian.heimeslinkissue42533 superseder
2020-06-02 21:51:48christian.heimessetmessages: + msg370635
2019-08-16 21:10:01christian.heimessetmessages: + msg349886
2019-08-16 11:55:40asvetlovsetmessages: + msg349855
2019-08-16 01:27:05epicfaacesetmessages: + msg349840
2019-08-14 04:59:58christian.heimessetmessages: + msg349643
versions: + Python 3.8
2019-08-14 04:55:34epicfaacesetmessages: + msg349642
2019-08-14 04:52:22epicfaacesetnosy: + epicfaace

versions: + Python 3.9, - Python 3.8
2018-11-03 12:23:03matrixisesetnosy: - matrixise
2018-10-24 14:40:50jmfrank63setnosy: + matrixise
2018-05-29 04:43:42asvetlovsettitle: IDNA2008 encoding missing -> IDNA2008 encoding is missing
2018-05-29 04:12:10socketpairsetnosy: + socketpair
2018-01-28 22:10:27njssetmessages: + msg311013
versions: + Python 3.8, - Python 3.7
2018-01-28 21:58:52christian.heimessetmessages: + msg311009
2018-01-28 21:47:22r.david.murraysetmessages: + msg311007
2018-01-28 20:22:53christian.heimessetmessages: + msg310988
2018-01-28 12:15:07christian.heimessetmessages: + msg310943
2018-01-28 05:44:53njssetmessages: + msg310924
2018-01-28 05:29:44wumpussetmessages: + msg310923
2017-12-28 18:17:11wumpussetnosy: + wumpus
2017-12-28 11:19:20asvetlovsetnosy: + asvetlov
2017-12-28 10:19:07berker.peksaglinkissue32437 superseder
2017-06-08 08:29:16njssetnosy: + njs
2017-01-09 18:33:47Socobsetnosy: + Socob
2016-11-02 08:16:45christian.heimessetmessages: + msg279904
2016-10-13 14:00:56SamWhitedsetnosy: + SamWhited
2016-10-12 16:15:01Lukasasetnosy: + Lukasa
2016-10-11 14:52:46christian.heimessetpriority: high -> critical
type: enhancement -> security
messages: + msg278493
2016-09-26 14:16:19christian.heimessetassignee: christian.heimes ->
2016-09-26 13:53:11christian.heimessetpriority: normal -> high
assignee: christian.heimes
components: + SSL
versions: + Python 3.7, - Python 3.5
2015-05-15 14:51:06christian.heimessetnosy: + christian.heimes
2015-03-25 18:16:02berker.peksagsetnosy: + berker.peksag

versions: + Python 3.5, - Python 3.4
2014-04-26 21:20:43loewissetnosy: + loewis
messages: + msg217218
2014-04-23 22:00:47underrunsetnosy: + underrun
messages: + msg217092
2013-12-02 17:14:07martensetmessages: + msg205034
2013-12-02 13:07:32erasetnosy: + era
messages: + msg205009
2013-02-28 04:20:16r.david.murraysetmessages: + msg183202
2013-02-28 04:08:46martensetmessages: + msg183199
2013-02-27 16:52:49r.david.murraysetstage: needs patch
messages: + msg183160
versions: - Python 2.6, Python 3.1, Python 2.7, Python 3.2, Python 3.3, Python 3.5
2013-02-27 16:39:54martensetmessages: + msg183159
2013-02-27 12:37:23r.david.murraysetmessages: + msg183149
2013-02-27 12:29:21martensetmessages: + msg183147
2013-02-27 12:25:21martensetmessages: + msg183144
2013-02-27 02:26:01r.david.murraysetnosy: + r.david.murray
messages: + msg183106
2013-02-27 01:32:46martencreate