This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author jmadden
Recipients Jonathan Kamens, Paolo Veglia, jmadden, marcjofre, martin.panter, pje
Date 2016-04-03.20:12:33
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1459714353.51.0.543504147245.issue24291@psf.upfronthosting.co.za>
In-reply-to
Content
gevent has another simple reproducer for this. I do believe it's not gevent's fault, the fault is in the standard library; SimpleHandler._write needs to loop until `sent += self.stdeout.write(data)` is `len(data)`.

I have written up more on this at https://github.com/gevent/gevent/issues/778#issuecomment-205046001 For convenience I'll reproduce the bulk of that comment here:

The user supplied a django application that produced a very large response that was getting truncated when using gevent under Python 3.4. (I believe gevent's non-blocking sockets are simply running into a different buffering behaviour, making it more likely to be hit under those conditions simply because they are faster).

This looks like a bug in the standard library's `wsgiref` implementation.

I tracked this down to a call to `socket.send(data)`. This method only sends whatever portion of the data it is possible to send at the time, and it returns the count of the data that was sent. The caller of `socket.send()` is responsible for looping to make sure the full `len` of the data is sent. This [is clearly documented](https://docs.python.org/3/library/socket.html#socket.socket.send). 

In this case, there is a call to `send` trying to send the full response, but only a portion of it is able to be immediately written. Here's a transcript of the first request (I modified gevent's `socket.send` method to print how much data is actually sent each time):

```
Django version 1.9.5, using settings 'gdc.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
SENDING 17
SENT 17 OF 17
SENDING 37
SENT 37 OF 37
SENDING 38
SENT 38 OF 38
SENDING 71
SENT 71 OF 71
SENDING 1757905
SENT 555444 OF 1757905
[03/Apr/2016 19:48:31] "GET / HTTP/1.1" 200 1757905
```

Note that there's no retry on the short send.

Here's the stack trace for that short send; we can clearly see that there is no retry loop in place:

```
  //3.4/lib/python3.4/wsgiref/handlers.py(138)run()
    136             self.setup_environ()
    137             self.result = application(self.environ, self.start_response)
--> 138             self.finish_response()
    139         except:
    140             try:

  //3.4/lib/python3.4/wsgiref/handlers.py(180)finish_response()
    178             if not self.result_is_file() or not self.sendfile():
    179                 for data in self.result:
--> 180                     self.write(data)
    181                 self.finish_content()
    182         finally:

  //3.4/lib/python3.4/wsgiref/handlers.py(279)write()
    277
    278         # XXX check Content-Length and truncate if too many bytes written?
--> 279         self._write(data)
    280         self._flush()
    281

  //3.4/lib/python3.4/wsgiref/handlers.py(453)_write()
    451
    452     def _write(self,data):
--> 453         self.stdout.write(data)
    454
    455     def _flush(self):

  //3.4/lib/python3.4/socket.py(398)write()
    396         self._checkWritable()
    397         try:
--> 398             return self._sock.send(b)
    399         except error as e:
    400             # XXX what about EINTR?

> //gevent/_socket3.py(384)send()
    382                 from IPython.core.debugger import Tracer; Tracer()() ## DEBUG ##
    383
--> 384             return count
```

`self.stdout` is an instance of `socket.SocketIO` (which is returned from `socket.makefile`). This is not documented on the web, but the [docstring also clearly documents](https://github.com/python/cpython/blob/3.4/Lib/socket.py#L389) that callers of `write` should loop to make sure all data gets sent.
History
Date User Action Args
2016-04-03 20:12:33jmaddensetrecipients: + jmadden, pje, martin.panter, Jonathan Kamens, marcjofre, Paolo Veglia
2016-04-03 20:12:33jmaddensetmessageid: <1459714353.51.0.543504147245.issue24291@psf.upfronthosting.co.za>
2016-04-03 20:12:33jmaddenlinkissue24291 messages
2016-04-03 20:12:33jmaddencreate