classification
Title: socket makefile read-write discards received data
Type: behavior Stage: patch review
Components: IO Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ammar2, kc, martin.panter, pravn, serhiy.storchaka
Priority: normal Keywords: 3.6regression, patch

Created on 2019-02-07 12:30 by pravn, last changed 2019-02-15 18:00 by kc.

Pull Requests
URL Status Linked Edit
PR 11878 closed kc, 2019-02-15 16:30
Messages (5)
msg335017 - (view) Author: Palle Ravn (pravn) Date: 2019-02-07 12:30
Using socket.makefile in read-write mode had a bug introduced between version 3.6.6 and 3.6.7. The same bug is present in version 3.7.x.

The below code example will behave very differently between 3.6.6 and 3.6.7. It's based on the echo-server example from the docs.

import socket

HOST = '127.0.0.1'
PORT = 0

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((HOST, PORT))

    print(f'Waiting for connection on port {s.getsockname()[1]}')
    s.listen(1)

    conn, addr = s.accept()
    print(f'Connected by {addr}')

    with conn:
        f = conn.makefile(mode='rw')

        while True:
            m = f.readline()
            print(f'msg: {m!r}')

            if not m:
                exit(0)

            f.write(m)
            f.flush()


Python 3.6.7:
Sending the string "Hello\nYou\n" will only print "Hello\n" and also only return "Hello\n" to the client.
Removing the lines with f.write(m) and f.flush() and both "Hello\n" and "You\n" will be returned to the client.
It's like the call to f.write() somehow empties the read buffer.

Python 3.6.6:
Sending "Hello\nYou\n" will return "Hello\n" and "You\n" to the client without any modifications to the above code.
msg335117 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2019-02-08 23:12
Looking over the changelog, my guess (untested) is this is caused by commit d6a283b3 for Issue 25862. That change looks like it drops the internal TextIOWrapper decoding buffer for each successful write.

I don't have the right version of Python to test with, but I expect this to also be broken without using a socket:

>>> f = TextIOWrapper(BufferedRWPair(BytesIO(b"Hello\nYou\n"), BytesIO()))
>>> f.readline()
'Hello\n'
>>> f.write(_)
6
>>> f.readline()  # Does this now return EOF?
'You\n'
msg335196 - (view) Author: Palle Ravn (pravn) Date: 2019-02-11 07:39
>>> f = TextIOWrapper(BufferedRWPair(BytesIO(b"Hello\nYou\n"), BytesIO()))
>>> f.readline()
'Hello\n'
>>> f.write(_)
6
>>> f.readline()  # Returns empty string
''
msg335197 - (view) Author: Ammar Askar (ammar2) * (Python triager) Date: 2019-02-11 07:52
Recreatable on master as well, also Martin your suspicion seems correct, reverting https://github.com/python/cpython/commit/23db935bcf258657682e66464bf8512def8af830 fixes it.
msg335631 - (view) Author: kc (kc) * Date: 2019-02-15 18:00
Added PR 11878, this will pass both this bug report and PR 3918 regression, the commit Ammar noted, it is an addition to this change.
History
Date User Action Args
2019-02-15 18:00:48kcsetnosy: + kc
messages: + msg335631
2019-02-15 16:30:00kcsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request11912
2019-02-11 07:52:57ammar2setversions: + Python 3.8
nosy: + serhiy.storchaka, ammar2

messages: + msg335197

stage: needs patch
2019-02-11 07:39:18pravnsetmessages: + msg335196
2019-02-08 23:12:59martin.pantersetkeywords: + 3.6regression
nosy: + martin.panter
messages: + msg335117

2019-02-07 12:30:32pravncreate