b'abc\xff'.decode('punycode', 'ignore') raises the same error than b'abc\xff'.decode('punycode', 'replace') or b'abc\xff'.decode('punycode'): it uses the strict error handler, and simply ignores the error handler.

punycodec, as idna, should raise an exception if the error handler is different than strict.

base64_codec.py uses an assertion to check the error handler: it should use an if+raise (assertions are ignored in optimized mode, python -O).

punycode_errors.patch:

• raise an error if errors="replace": only accept strict and ignore
• use errors to encode/decode to/from ASCII
• add unit tests

I don't think that the change should be documented, because punycode has no section in Doc/library/codecs.rst, and I hope that nobody uses something different than strict :-)

@martin: Can you review my patch?

What's the point of disallowing the replace error handler? That's a slightly incompatible change, isn't it?

Oh, I forgot to give a little bit more details.

b'abc\xff-'.decode('punycode', 'ignore') and b'abc\xff-'.decode('punycode', 'replace') raise a UnicodeDecodeError: the error handler is just useless (ignored) here.

With my patch, b'abc\xff-'.decode('punycode', 'ignore') gives 'abc'. (If I change the code to accept replace) b'abc\xff-'.decode('punycode', 'replace') gives also 'abc', but 'replace' doesn't work correctly in the part after "-" contain illegal byte sequences.

For example, b'a\xff-\xffb\xffga\xff'.decode("punycode", "replace") gives 'a�', whereas I would expect 'a�é' or 'aé�'. b'a-bga\xff'.decode("punycode", "replace") gives 'aé' as b'a-bga'.decode("punycode", "replace"), whereas I would expect 'aé�' or something like that.

What's the point of disallowing the replace error handler?

It's just that I'm unable to patch punycode decoder to support the replace handler. Do you want to "implement" it?

That's a slightly incompatible change, isn't it?

I don't think so because I consider that the punycode decoder never supported error handlers (other than strict) in Python 3.

What do you think?

Oh, I forgot to give a little bit more details.

b'abc\xff-'.decode('punycode', 'ignore') and
b'abc\xff-'.decode('punycode', 'replace') raise a UnicodeDecodeError:
the error handler is just useless (ignored) here.

That's not my point:

b"foo".decode("punycode","replace")

currently succeeds, but raises an UnicodeError under the patch.

I prefer to not touch punycode right now, it works, there is no need to modify it.