classification
Title: getpass.unix_getpass() always fallback to sys.stdin
Type: behavior Stage: resolved
Components: IO, Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alex, benjamin.peterson, gregory.p.smith, hynek, martin.panter, nikratio, pitrou, python-dev, r.david.murray, serhiy.storchaka, stutzbach, vajrasky
Priority: normal Keywords: patch

Created on 2013-06-01 23:50 by nikratio, last changed 2014-04-19 03:25 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
issue18116.diff alex, 2013-06-01 23:56 review
getpass_fix_resourcewarning.patch vajrasky, 2013-06-02 07:55 Patch to fix resourcewarning warning when using getpass review
getpass_fix_resourcewarning_pass_the_test.patch vajrasky, 2013-06-02 08:09 Patch to fix resourcewarning warning when using getpass and pass the test review
getpass_tty_with_encoding_and_test.patch vajrasky, 2013-06-04 15:09 Patch to fix resourcewarning warning when using getpass with encoding from os module review
buffered_open_fd_leak.py serhiy.storchaka, 2013-06-06 10:25
getpass_fix_works_with_stringio_and_stdout_stream.patch vajrasky, 2013-06-06 13:48 Patch to fix resourcewarning warning when using getpass and works with stringio and stdout stream review
getpass_tty.patch serhiy.storchaka, 2013-06-07 09:26 review
Messages (22)
msg190458 - (view) Author: Nikolaus Rath (nikratio) * Date: 2013-06-01 23:50
[0] nikratio@vostro:~/tmp$ cat bugme.py 
#!python
import getpass
import warnings

warnings.simplefilter('default')
getpass.getpass("What's up?")

[0] nikratio@vostro:~/tmp$ python3 --version
Python 3.3.2

[0] nikratio@vostro:~/tmp$ python3 bugme.py 
/usr/lib/python3.3/os.py:1043: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'>
  return io.open(fd, *args, **kwargs)
What's up?
msg190459 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2013-06-01 23:56
Attached patch should fix this issue.
msg190460 - (view) Author: Nikolaus Rath (nikratio) * Date: 2013-06-02 00:01
No, it doesn't.
msg190461 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2013-06-02 00:04
Are you sure you applied it correctly? With and without:

Alexanders-MacBook-Pro:cpython alex_gaynor$ ./python.exe x.py
What's up?
Alexanders-MacBook-Pro:cpython alex_gaynor$ hg revert --all --no-backup
reverting Lib/getpass.py
Alexanders-MacBook-Pro:cpython alex_gaynor$ ./python.exe x.py
What's up?
x.py:6: ResourceWarning: unclosed file <_io.TextIOWrapper name=3 mode='w+' encoding='UTF-8'>
  getpass.getpass("What's up?")
msg190462 - (view) Author: Nikolaus Rath (nikratio) * Date: 2013-06-02 00:24
Yes, I'm pretty sure:

[0] nikratio@vostro:~/tmp$ cp  /usr/lib/python3.3/getpass.py  .
[0] nikratio@vostro:~/tmp$ patch -p2 < issue18116.diff 
patching file getpass.py
Hunk #1 succeeded at 57 (offset -1 lines).

[0] nikratio@vostro:~/tmp$ python3 bugme.py 
/usr/lib/python3.3/os.py:1043: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'>
  return io.open(fd, *args, **kwargs)
What's up?

# Test if we're using the patched getpass.py...
[0] nikratio@vostro:~/tmp$ vim getpass.py
[0] nikratio@vostro:~/tmp$ python3 bugme.py 
Hello
/usr/lib/python3.3/os.py:1043: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'>
  return io.open(fd, *args, **kwargs)
What's up?
msg190468 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-06-02 03:26
This bug happens in Python 3.4 as well.

[sky@localhost cpython]$ ./python --version
Python 3.4.0a0
[sky@localhost cpython]$ ./python /tmp/bugme.py
/home/sky/Code/python/programming_language/cpython/Lib/os.py:1025: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'>
  return io.open(fd, *args, **kwargs)
What's up?

I tried to apply the patch manually (by copying, cutting and pasting) from Alex but Nikolaus is right. The patch does not work. The bug still happens
msg190471 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-06-02 07:24
I isolate the bug. It happens in these lines:

        # Always try reading and writing directly on the tty first.
        fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY)
        tty = os.fdopen(fd, 'w+', 1)

So to produce the bug more specifically, you can try this python file:

# bugme2.py
import os

fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY)
os.fdopen(fd, 'w+', 1)
# end of bugme2.py

In Linux Fedora 18, I would get this error:

/home/sky/Code/python/programming_language/cpython/Lib/os.py:1025: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'>
  return io.open(fd, *args, **kwargs)
Traceback (most recent call last):
  File "/tmp/bugme2.py", line 4, in <module>
    os.fdopen(fd, 'w+', 1)
  File "/home/sky/Code/python/programming_language/cpython/Lib/os.py", line 1025, in fdopen
    return io.open(fd, *args, **kwargs)
io.UnsupportedOperation: File or stream is not seekable.
msg190472 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-06-02 07:55
I have investigated this problem and come up with the patch to fix the problem. This patch does the job. Caution: only for Python 3.4. But translating this patch to Python 3.3 should be straightforward.

I hope this patch could be the foundation for better programmers to create better patch.

Some of the issues with this patch are: I am not sure how to handle encoding and where the best place to close tty is.

Reference:
https://github.com/stefanholek/term/issues/1
http://stackoverflow.com/questions/5471158/typeerror-str-does-not-support-the-buffer-interface
msg190473 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-06-02 08:09
Sorry,

My previous patch breaks the test. This one should pass the test and fix the bug.

Still, there are ugly code in the patch that I hope better programmers could fix.
msg190520 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-06-03 07:02
This code is pretty broken. I don't think ttys are ever seekable, so the os.fdopen has probably been always failing since 3.0. It thus always leaks an fd to '/dev/tty' if the first os.open succeeds. The whole function should probably be rewriten to work with byte streams encoding the prompt with os.device_encoding(tty_fd) falling back on locale.getpreferredencoding().
msg190603 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-06-04 15:09
Based on the input from Benjamin Peterson, I grab encoding from the os module in the getpass module.

I put some test as well.

Until the whole function is rewritten, I hope this patch will suffice and do the job properly.
msg190706 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-06-06 08:50
>>> getpass.getpass('Password: ', sys.stdout)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 72, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 146, in _raw_input
    stream.write(bytes(prompt, tty_encoding))
TypeError: must be str, not bytes

>>> getpass.getpass('Password: ', io.StringIO())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 72, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 146, in _raw_input
    stream.write(bytes(prompt, tty_encoding))
TypeError: string argument expected, got 'bytes'
msg190710 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-06-06 10:25
The problem is in io.open, not in getpass. Here is a test script.

$ ./python buffered_open_fd_leak.py 
buffered_open_fd_leak.py:7: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+'>
  tty = io.open(fd, 'w+', 1)
msg190711 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-06-06 10:40
So the correct fix should be:
1. Make sure we can open /dev/tty in non-binary mode,
2. We can write string to /dev/tty,
3. Close the leak if we can not open /dev/tty.

Is it?
msg190714 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-06-06 13:48
Fixing IO leak resource would fix this bug but leave another bug opened which I try to fix as well.

These statements with Python3 under Linux will always fail because we need to open /dev/tty file in binary mode (for whatever reason).

fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY)
tty = os.fdopen(fd, 'w+', 1)

So I guess the correct fix would be to open /dev/tty in binary mode (and set buffering off) and go on from there.

Of course, one can argue whether this bug should be separated from the original bug (resource warning). I am not sure either.

Anyway, here is the patch that will work with stream StringIO and stdout.

Thank you for Serhiy for pointing out my mistakes.
msg190716 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-06-06 14:32
>>> getpass.getpass('Password: ', open('/dev/stdout', 'w'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 72, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 143, in _raw_input
    stream.write(bytes(prompt, tty_encoding))
TypeError: must be str, not bytes

It seems that you are moving in the wrong direction. No need to test explicitly for stdout/stderr/etc, the code should work with arbitrary text stream.
msg190740 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-06-07 09:26
Here is a patch which fixes getpass bug: unix_getpass() always fallback to sys.stdin. As side effect it also fixes resource warning in getpass().

I'm not sure I have correctly changed tests. David, could you please review the patch?
msg190746 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-07 13:29
The test changes look correct to me, but it sure would be nice to come up with less fragile tests.  For a function like this, though, it probably isn't possible.
msg192836 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-10 21:10
New changeset 70f55dc9d43f by R David Murray in branch 'default':
#18116: getpass no longer always falls back to stdin.
http://hg.python.org/cpython/rev/70f55dc9d43f
msg192837 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-07-10 21:13
I played around with this for a bit, but I couldn't come up with any test improvements, or any way to test the bug that is being fixed.  So I just committed it as is.  Thanks, Serhiy.  And thanks Vajrasky for giving it a try and figuring out some of the stuff that *doesn't* work as a fix :)

I decided to only commit this to 3.4.  I don't think the risk of an unexpected behavior change in a maintenance release is worth the relatively small benefit that this fix provides.
msg206999 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-27 16:48
New changeset 100f632d4306 by R David Murray in branch '3.3':
#18116: backport fix to 3.3 since real-world failure mode demonstrated.
http://hg.python.org/cpython/rev/100f632d4306

New changeset 29a5a5b39dd6 by R David Murray in branch 'default':
Mostly-null merge of #18116 backport (updated NEWS entry).
http://hg.python.org/cpython/rev/29a5a5b39dd6
msg216839 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-04-19 03:25
I opened Issue 21310 about a ResourceWarning from open() which I suspect is the same as what was originally described here.
History
Date User Action Args
2014-04-19 03:25:13martin.pantersetnosy: + martin.panter
messages: + msg216839
2013-12-27 16:48:22python-devsetmessages: + msg206999
2013-07-10 21:13:57r.david.murraysetstatus: open -> closed
versions: - Python 3.3
messages: + msg192837

resolution: fixed
stage: patch review -> resolved
2013-07-10 21:10:44python-devsetnosy: + python-dev
messages: + msg192836
2013-06-07 13:29:08r.david.murraysetmessages: + msg190746
2013-06-07 09:26:33serhiy.storchakasetfiles: + getpass_tty.patch

type: resource usage -> behavior
title: getpass.getpass() triggers ResourceWarning -> getpass.unix_getpass() always fallback to sys.stdin
nosy: + gregory.p.smith, r.david.murray

messages: + msg190740
stage: patch review
2013-06-06 14:32:39serhiy.storchakasetmessages: + msg190716
2013-06-06 13:48:39vajraskysetfiles: + getpass_fix_works_with_stringio_and_stdout_stream.patch

messages: + msg190714
2013-06-06 10:40:41vajraskysetmessages: + msg190711
2013-06-06 10:26:58serhiy.storchakasetnosy: + pitrou, stutzbach, hynek
components: + IO
2013-06-06 10:26:00serhiy.storchakasetfiles: + buffered_open_fd_leak.py

messages: + msg190710
versions: + Python 3.4
2013-06-06 08:50:59serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg190706
2013-06-04 15:09:13vajraskysetfiles: + getpass_tty_with_encoding_and_test.patch

messages: + msg190603
2013-06-03 07:02:55benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg190520
2013-06-02 08:09:05vajraskysetfiles: + getpass_fix_resourcewarning_pass_the_test.patch

messages: + msg190473
2013-06-02 07:55:30vajraskysetfiles: + getpass_fix_resourcewarning.patch

messages: + msg190472
2013-06-02 07:24:10vajraskysetmessages: + msg190471
2013-06-02 03:26:46vajraskysetnosy: + vajrasky
messages: + msg190468
2013-06-02 00:24:41nikratiosetmessages: + msg190462
2013-06-02 00:04:36alexsetmessages: + msg190461
2013-06-02 00:01:47nikratiosetmessages: + msg190460
2013-06-01 23:56:56alexsetfiles: + issue18116.diff

nosy: + alex
messages: + msg190459

keywords: + patch
2013-06-01 23:50:06nikratiocreate