getpass.getpass() fails with non-ASCII characters in prompt #65368

Arfrever · 2014-04-07T11:23:56Z

BPO	21169
Nosy	@loewis, @bitdancer, @serhiy-storchaka, @kushaldas
Files	getpass_test_python2 getpass_test_python3 issue21169.patch: Stops the breakage of nonascii chars in non utf-8 environment issue21169_v2.patch: Version2 of the patch with a test update. issue21169_v3.patch: Updated patch with discovering of currect locale and corresponding test case. issue21169_v4.patch: New patch with actual test case :) issue21169_v5.patch: Newer version of the patch with stream.encoding issue21169_v6.patch: New patchset. issue21169_v7.patch: Another patch with docs update and one line code comment.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-04-14.14:32:58.696>
created_at = <Date 2014-04-07.11:23:55.654>
labels = ['easy', 'type-bug', 'library']
title = 'getpass.getpass() fails with non-ASCII characters in prompt'
updated_at = <Date 2014-04-14.14:32:58.695>
user = 'https://bugs.python.org/Arfrever'

bugs.python.org fields:

activity = <Date 2014-04-14.14:32:58.695>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2014-04-14.14:32:58.696>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2014-04-07.11:23:55.654>
creator = 'Arfrever'
dependencies = []
files = ['34747', '34748', '34766', '34767', '34775', '34776', '34807', '34810', '34815']
hgrepos = []
issue_num = 21169
keywords = ['patch', 'easy']
message_count = 19.0
messages = ['215697', '215778', '215779', '215828', '215829', '216002', '216006', '216009', '216013', '216041', '216045', '216050', '216051', '216052', '216054', '216077', '216078', '216081', '216082']
nosy_count = 6.0
nosy_names = ['loewis', 'Arfrever', 'r.david.murray', 'python-dev', 'serhiy.storchaka', 'kushal.das']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21169'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

Arfrever · 2014-04-07T11:23:55Z

getpass.getpass() fails with non-ASCII characters in prompt.

The attached example scripts (for Python 2 and 3) contain non-ASCII unicode prompt (Polish "hasło" == English "password") written in UTF-8.
Python-2 version fails always. Python-3 version fails in non-UTF-8 locale.

$ ./getpass_test_python2
Traceback (most recent call last):
  File "./getpass_test_python2", line 5, in <module>
    getpass.getpass(u"Hasło: ")
  File "/usr/lib64/python2.7/getpass.py", line 71, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/usr/lib64/python2.7/getpass.py", line 128, in _raw_input
    prompt = str(prompt)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0142' in position 3: ordinal not in range(128)
$ LC_ALL="en_US.UTF-8" ./getpass_test_python3
Hasło: 
$ LC_ALL="C" ./getpass_test_python3
Traceback (most recent call last):
  File "./getpass_test_python3", line 5, in <module>
    getpass.getpass("Has\u0142o: ")
  File "/usr/lib64/python3.4/getpass.py", line 78, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/usr/lib64/python3.4/getpass.py", line 138, in _raw_input
    stream.write(prompt)
UnicodeEncodeError: 'ascii' codec can't encode character '\u0142' in position 3: ordinal not in range(128)
$

kushaldas · 2014-04-08T22:20:41Z

Here is a patch which stops the breakage in getpass for python3.

kushaldas · 2014-04-08T22:46:53Z

Version 2 of the patch with a test update.

kushaldas · 2014-04-09T17:24:16Z

Updated patch with discovering of currect locale and corresponding test case.

kushaldas · 2014-04-09T17:26:44Z

New patch with actual test case :)

serhiy-storchaka · 2014-04-13T08:51:20Z

I don't think this is a bug. Any text output operation can fail when outputs unencodable string. You should use a stream with proper encoding and/or error handler.

loewis · 2014-04-13T09:14:29Z

I agree that it is not a bug if the device where the prompt is shown simply does not support the characters; on Unix, this includes cases where the locale does not support the characters.

Arfrever: when you say that it fails in Python 3 in a non-UTF-8 locale, which specific locale was that that it failed in?

serhiy-storchaka · 2014-04-13T10:10:10Z

$ LC_ALL=en_US.iso88591 ./python -c "print('\u20ac')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 0: ordinal not in range(256)

$ LC_ALL=en_US.iso88591 ./python -c "input('\u20ac')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 0: ordinal not in range(256)

$ LC_ALL=en_US.iso88591 ./python -c "import getpass; getpass.getpass('\u20ac')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 78, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 138, in _raw_input
    stream.write(prompt)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 0: ordinal not in range(256)

Arfrever · 2014-04-13T10:42:25Z

Martin v. Löwis: In this case, device support non-ASCII characters, but Python's getpass module forgets to properly encode string. Message 215697 contains example with C locale.

kushaldas · 2014-04-14T00:58:36Z

Here is a new patch which uses stream.encoding instead getting the encoding from the locale as suggested by David. It also contains the new test.

kushaldas · 2014-04-14T01:33:15Z

New patchset with updated test, now sending ascii stream into the call as argument.

loewis · 2014-04-14T01:58:41Z

Arfrever: If you set the locale to C, the device does *not* (anymore) support the character. The terminal application you are using may, but the "system" does not. It only supports the characters available in the locale, which your character is not. There simply is no way in which Python *could* encode the character.

python-dev · 2014-04-14T02:09:46Z

New changeset f430fdd1628e by R David Murray in branch '3.4':
bpo-21169: fix getpass to use replace error handler on UnicodeEncodeError.
http://hg.python.org/cpython/rev/f430fdd1628e

New changeset 461f5863f2aa by R David Murray in branch 'default':
Mierge bpo-21169: fix getpass to use replace error handler on UnicodeEncodeError.
http://hg.python.org/cpython/rev/461f5863f2aa

bitdancer · 2014-04-14T02:14:14Z

Since we don't want the prompting for the password to fail, what we do in the patch is use the replace error handler so that you get as much as could be encoded of the prompt. (Note: this approach was reviewed by both Toshio and Marc Andre.)

Thanks for the patch, Kushal.

loewis · 2014-04-14T02:19:23Z

Ok. I wish the patch had a comment saying that, or better even a documentation change pointing out that feature.

bitdancer · 2014-04-14T12:21:04Z

Ok, I'll reopen the issue to do that.

kushaldas · 2014-04-14T13:49:17Z

Another patch with docs update and one line code comment.

python-dev · 2014-04-14T14:31:08Z

New changeset bdde36cd9048 by R David Murray in branch '3.4':
bpo-21169: add comment and doc update for getpass change.
http://hg.python.org/cpython/rev/bdde36cd9048

New changeset fe532dccf8f6 by R David Murray in branch 'default':
Merge: bpo-21169: add comment and doc update for getpass change.
http://hg.python.org/cpython/rev/fe532dccf8f6

bitdancer · 2014-04-14T14:32:59Z

I decided to tweak the language slightly, Kushal. If this isn't what you were looking for, Martin, let me know.

Arfrever mannequin added stdlib Python modules in the Lib dir easy labels Apr 7, 2014

serhiy-storchaka added the invalid label Apr 13, 2014

Arfrever mannequin removed the invalid label Apr 13, 2014

bitdancer closed this as completed Apr 14, 2014

bitdancer added the type-bug An unexpected behavior, bug, or error label Apr 14, 2014

bitdancer reopened this Apr 14, 2014

bitdancer closed this as completed Apr 14, 2014

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

getpass.getpass() fails with non-ASCII characters in prompt #65368

getpass.getpass() fails with non-ASCII characters in prompt #65368

Arfrever mannequin commented Apr 7, 2014

Arfrever mannequin commented Apr 7, 2014

kushaldas commented Apr 8, 2014

kushaldas commented Apr 8, 2014

kushaldas commented Apr 9, 2014

kushaldas commented Apr 9, 2014

serhiy-storchaka commented Apr 13, 2014

loewis mannequin commented Apr 13, 2014

serhiy-storchaka commented Apr 13, 2014

Arfrever mannequin commented Apr 13, 2014

kushaldas commented Apr 14, 2014

kushaldas commented Apr 14, 2014

loewis mannequin commented Apr 14, 2014

python-dev mannequin commented Apr 14, 2014

bitdancer commented Apr 14, 2014

loewis mannequin commented Apr 14, 2014

bitdancer commented Apr 14, 2014

kushaldas commented Apr 14, 2014

python-dev mannequin commented Apr 14, 2014

bitdancer commented Apr 14, 2014

getpass.getpass() fails with non-ASCII characters in prompt #65368

getpass.getpass() fails with non-ASCII characters in prompt #65368

Comments

Arfrever mannequin commented Apr 7, 2014

Arfrever mannequin commented Apr 7, 2014

kushaldas commented Apr 8, 2014

kushaldas commented Apr 8, 2014

kushaldas commented Apr 9, 2014

kushaldas commented Apr 9, 2014

serhiy-storchaka commented Apr 13, 2014

loewis mannequin commented Apr 13, 2014

serhiy-storchaka commented Apr 13, 2014

Arfrever mannequin commented Apr 13, 2014

kushaldas commented Apr 14, 2014

kushaldas commented Apr 14, 2014

loewis mannequin commented Apr 14, 2014

python-dev mannequin commented Apr 14, 2014

bitdancer commented Apr 14, 2014

loewis mannequin commented Apr 14, 2014

bitdancer commented Apr 14, 2014

kushaldas commented Apr 14, 2014

python-dev mannequin commented Apr 14, 2014

bitdancer commented Apr 14, 2014