classification
Title: Document how to make pty.spawn not copy data
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: RadicalZephyr, cheryl.sabella, docs@python, martin.panter, twouters, vstinner
Priority: normal Keywords: patch

Created on 2014-11-13 18:01 by RadicalZephyr, last changed 2019-05-21 09:38 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
pty.patch RadicalZephyr, 2014-11-13 18:01 review
pty.patch RadicalZephyr, 2015-01-07 03:34 Updated patch review
pty.patch RadicalZephyr, 2015-05-07 20:09 Documentation update
Pull Requests
URL Status Linked Edit
PR 11980 merged python-dev, 2019-02-21 17:49
PR 13455 merged RadicalZephyr, 2019-05-20 23:41
Messages (15)
msg231126 - (view) Author: Geoff Shannon (RadicalZephyr) * Date: 2014-11-13 18:01
While using the pty.py module I had cause to want to be able to silently eat characters from the input stream.  I discovered that because of the check for end of file being "if not data" that there was no way to do this.

Since the default function used by spawn and _copy is a thin wrapper on os.read, I figured that it would be useful to interpret None as a sentinel value meaning "this is not EOF, but there's nothing I actually want to copy over."
msg233561 - (view) Author: Geoff Shannon (RadicalZephyr) * Date: 2015-01-07 03:34
I tweaked the patch a bit to not include the parentheses since that seems to be the style here.
msg242730 - (view) Author: Geoff Shannon (RadicalZephyr) * Date: 2015-05-07 20:09
Okay, I just found another way to achieve the same effect of letting the _read function ignore data but not inadvertantly close the stream.  It relies on the fact that terminals will ignore null bytes fed to them.

Now there are no code changes required, just an addition to the documentation.
msg244956 - (view) Author: Geoff Shannon (RadicalZephyr) * Date: 2015-06-07 13:32
Hey, pinging this issue.  Hoping someone has time to take a look at it.
msg252444 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-10-07 01:51
If you return null bytes, they will be written to the parent process’s stdout file descriptor, rather than being ignored. I do not think it is possible to ignore output with the spawn() API, unless perhaps you previously set up stdout write to /dev/null or similar.

Do you have a reference for your fact about terminals ignoring null input bytes? I’m skeptical because a basic serial terminal can send arbitrary data including nulls, but I am not so familiar with pseudo-terminals.

However the bit about interpreting an empty string could be useful. I would drop the “or any falsey value”; is that a typo? I suspect that the functions have to return byte strings (not text). It would be nice to say what the consequences of the EOF condition is, e.g. I can see for the child’s output it would stop copying, but not actually close the parent’s stdout to signal EOF. What about in the other direction? Can you signal multiple EOFs with data in between?
msg252446 - (view) Author: Geoff Shannon (RadicalZephyr) * Date: 2015-10-07 02:34
Hmm, I spoke improperly.  I think you are entirely correct in your statements.  What I meant by "terminals ignore null bytes" is that returning a string consisting of only a null byte doesn't cause anything observable to happen, including anything to be written to the screen.  But I have no idea why this is.

My basis for this is limited to empirical observation of how my terminal behaves (I use the iTerm2 terminal emulator on Mac OS X), and I don't recall whether I checked it with other terminals like the regular terminal on Mac and terminal emulators on Linux.

Re: "or any falsey value".  In general, the _read_ functions should return byte strings yes, but since the check for EOF in python is whether a stream returned an empty string, and the way that pty.spawn() currently checks for that is `if not data` then any falsey value will cause pty.spawn to think that the underlying fd was closed.

I agree that the documentation on this could use more fleshing out in general, especially regarding what happens when certain values are returned.  I'm not very familiar with this code any more, but I might have time to play with it and try to add some more information in the coming weeks.
msg252453 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-10-07 06:46
I would avoid suggesting one can return non-byte-strings; treat that as an implementation detail. Maybe something like:

The functions *master_read* and *stdin_read* are passed a file descriptor which they should read from, and should return a byte string. The defaults try to read 1024 bytes each time they are called. Returning an empty string is be interpreted as an EOF condition, and the *_read* function will no longer be called.
msg336171 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2019-02-21 01:47
@RadicalZephyr, are you interested in making a GitHub pull request with the recommended the documentation change?
msg336246 - (view) Author: Geoff Shannon (RadicalZephyr) * Date: 2019-02-21 17:50
It is submitted @cheryl.sabella. Thanks for reviving this, I had totally lost track of it.
msg336257 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2019-02-21 20:33
Thanks @RadicalZephyr!  

@martin.panter, please review the PR when you get a chance.  Thank you!
msg336268 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2019-02-21 23:03
I'm not sure it is wise for the Python documentation to suggest inserting null bytes in general. This seems more like an application-specific hack. There is nothing in Python that handles these null bytes specially, and I expect they will be seen if the child reads the terminal in raw mode, or if the parent's output is redirected to a file, sent over the network or a real serial link.
msg342871 - (view) Author: Geoff Shannon (RadicalZephyr) * Date: 2019-05-19 21:20
@martin.panter I removed the mention of inserting null bytes and restricted the documentation updates to more fully documenting the current behaviour.
msg342910 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-20 15:06
New changeset 522ccef8690970fc4f78f51a3adb995f2547871a by Victor Stinner (Geoff Shannon) in branch 'master':
bpo-22865: Expand on documentation for the pty.spawn function (GH-11980)
https://github.com/python/cpython/commit/522ccef8690970fc4f78f51a3adb995f2547871a
msg342993 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-21 09:37
New changeset cdb2dbfe92b95dcd19ccab1a1e9b8c39263c54b0 by Victor Stinner (Geoff Shannon) in branch '3.7':
[3.7] bpo-22865: Expand on documentation for the pty.spawn function (GH-11980) (GH-13455)
https://github.com/python/cpython/commit/cdb2dbfe92b95dcd19ccab1a1e9b8c39263c54b0
msg342994 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-21 09:38
Thanks Geoff Shannon for the doc enhancement!
History
Date User Action Args
2019-05-21 09:38:34vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg342994

stage: patch review -> resolved
2019-05-21 09:37:15vstinnersetmessages: + msg342993
2019-05-20 23:41:22RadicalZephyrsetpull_requests: + pull_request13364
2019-05-20 15:06:32vstinnersetnosy: + vstinner
messages: + msg342910
2019-05-19 21:20:23RadicalZephyrsetmessages: + msg342871
2019-02-21 23:03:53martin.pantersetmessages: + msg336268
2019-02-21 20:33:02cheryl.sabellasetmessages: + msg336257
2019-02-21 17:50:50RadicalZephyrsetmessages: + msg336246
2019-02-21 17:49:26python-devsetpull_requests: + pull_request12004
2019-02-21 01:47:47cheryl.sabellasetnosy: + cheryl.sabella

messages: + msg336171
versions: + Python 3.7, Python 3.8, - Python 2.7, Python 3.4, Python 3.5, Python 3.6
2015-10-07 06:46:27martin.pantersetmessages: + msg252453
2015-10-07 02:34:43RadicalZephyrsetmessages: + msg252446
2015-10-07 01:51:04martin.pantersetversions: - Python 3.2, Python 3.3
nosy: + martin.panter

messages: + msg252444

stage: patch review
2015-06-07 15:13:32r.david.murraysetnosy: + twouters
2015-06-07 13:32:16RadicalZephyrsetmessages: + msg244956
2015-05-07 20:09:23RadicalZephyrsetfiles: + pty.patch

messages: + msg242730
components: - Library (Lib)
title: Allow pty.spawn to ignore data to copy -> Document how to make pty.spawn not copy data
2015-01-07 03:34:42RadicalZephyrsetfiles: + pty.patch

messages: + msg233561
2014-11-13 18:01:26RadicalZephyrcreate