➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Remote end closed connection without response
Type: Stage: resolved
Components: Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Michal Arbet, an7e, gvanrossum, martin.panter
Priority: normal Keywords:

Created on 2020-07-20 11:43 by Michal Arbet, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
mycap.pcap Michal Arbet, 2020-07-20 11:43
Messages (8)
msg374000 - (view) Author: Michal Arbet (Michal Arbet) Date: 2020-07-20 11:43 
Hi,

I'm not sure if this is really python bug, but I hope that you can check the issue.

Issue is that from time to time i'm getting exception from python when sending request to server which has http keepalive option turned on.

Requests send a request but in few miliseconds apache2 server is closing persistent connection by sending FIN packet which generate traceback.

I can reproduce it by following simple script.

#!/usr/bin/python3

import requests
from time import sleep

import logging
logging.basicConfig(level=logging.DEBUG)

s = requests.Session()
s.verify = False  # self-signed cert

counter = 0
txt = "test"
while True:
    counter = counter + 1
    s.post('http://localhost', data={counter:txt})
    sleep(5)

Everything is working fine, but from time to time I get following traceback. When FIN is received right after request was sent.

michalarbet@pixla:~/work$ ./request_test.py 
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:80
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 0
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/michalarbet/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./request_test.py", line 26, in <module>
    s.post('http://localhost', data={counter:txt})
  File "/home/michalarbet/.local/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/michalarbet/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/michalarbet/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/michalarbet/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Apache using keepalive.
/etc/apache2/apache2.conf:KeepAliveTimeout 5
/etc/apache2/apache2.conf:MaxKeepAliveRequests 100
/etc/apache2/apache2.conf:KeepAlive On


Sending also pcap file from different reproduce test, but behaviour is same.
msg374004 - (view) Author: Ilia Androshchuk (an7e) * Date: 2020-07-20 12:54 
Hi Michal,
Which version of operating system and apache are you running?
I'll try to reproduce
msg374008 - (view) Author: Michal Arbet (Michal Arbet) Date: 2020-07-20 13:48 
I'm using  Ubuntu 20.04 with dist apache2 and default settings :

michalarbet@pixla:~/work$ dpkg -l | grep apache
ii  apache2                                    2.4.41-4ubuntu3
          amd64        Apache HTTP Server
ii  apache2-bin                                2.4.41-4ubuntu3
          amd64        Apache HTTP Server (modules and other binary files)
ii  apache2-data                               2.4.41-4ubuntu3
          all          Apache HTTP Server (common files)
ii  apache2-utils                              2.4.41-4ubuntu3
          amd64        Apache HTTP Server (utility programs for web servers)
ii  libapache2-mod-php                         2:7.4+75
         all          server-side, HTML-embedded scripting language (Apache
2 module) (default)
ii  libapache2-mod-php7.4                      7.4.3-4ubuntu2.2
         amd64        server-side, HTML-embedded scripting language (Apache
2 module)

michalarbet@pixla:~/work$ uname -a
Linux pixla 5.4.0-40-generic #44-Ubuntu SMP Tue Jun 23 00:01:04 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux

When reproducing issue, be patient , you have to wait several iterations
before it will fail.
Some bug reports and discussion related to :

https://bugs.python.org/issue41345
https://github.com/psf/requests/issues/4664
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=899406

po 20. 7. 2020 v 14:54 odesílatel Ilia Androshchuk <report@bugs.python.org>
napsal:

>
> Ilia Androshchuk <an7e@outlook.com> added the comment:
>
> Hi Michal,
> Which version of operating system and apache are you running?
> I'll try to reproduce
>
> ----------
> nosy: +an7e
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue41345>
> _______________________________________
>
msg374034 - (view) Author: Ilia Androshchuk (an7e) * Date: 2020-07-20 23:17 
Hi, Michal
It took a lot of time, but I achieved exactly the same behavior.

My Traceback :

DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:80 "POST / HTTP/1.1" 200 3138
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test1.py", line 14, in <module>
    s.post('http://localhost', data={counter:txt})
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I'm using same dist and settings:

lerfi@UbuntuVM:~/work/psf/41345$ uname -a
Linux UbuntuVM 5.4.0-40-generic #44-Ubuntu SMP Tue Jun 23 00:01:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
erfi@UbuntuVM:~/work/psf/41345$ dpkg -l | grep apache
ii  apache2                                    2.4.41-4ubuntu3                       amd64        Apache HTTP Server
ii  apache2-bin                                2.4.41-4ubuntu3                       amd64        Apache HTTP Server (modules and other binary files)
ii  apache2-data                               2.4.41-4ubuntu3                       all          Apache HTTP Server (common files)
ii  apache2-utils                              2.4.41-4ubuntu3                       amd64        Apache HTTP Server (utility programs for web servers)
pip3 requests==2.22.0

it looks like there's a problem with this urllib3 package
msg374066 - (view) Author: Ilia Androshchuk (an7e) * Date: 2020-07-21 13:18 
Hi Michal,
I found some information in the package documentation
Please check this page
https://requests.readthedocs.io/en/master/user/advanced/#keep-alive
msg374200 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-07-24 18:28 
This surely is some application corner case. Closing.
msg374365 - (view) Author: Michal Arbet (Michal Arbet) Date: 2020-07-27 08:04 
Hello,

Thank you very much for help, I created pull request for urllib3 ->
https://github.com/urllib3/urllib3/pull/1911
Can u just confirm that this could be fixed in urllib3 as it is in pull
request ? Have I fixed it by good way ?

Thanks.
Michal Arbet
( kevko )

pá 24. 7. 2020 v 20:29 odesílatel Guido van Rossum <report@bugs.python.org>
napsal:

>
> Guido van Rossum <guido@python.org> added the comment:
>
> This surely is some application corner case. Closing.
>
> ----------
> nosy: +gvanrossum
> resolution:  -> not a bug
> stage:  -> resolved
> status: open -> closed
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue41345>
> _______________________________________
>
msg374466 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2020-07-28 05:47 
Previous report about Requests to the Python bug tracker: Issue 33620.

I suspect this is an unavoidable race condition with trying a POST (or other non-idempotent) request on an idle HTTP connection. I think it has to be up to the higher-level application or user to decide if it is safe to retry a POST request. Otherwise you risk e.g. accidentally ordering two pizzas because the server received two POST requests but something interfered with the response of the first response. On the other hand, I noticed some browsers seem to automatically retry a POST once if it is interrupted, which makes me uneasy.

A concrete example of the problem is a firmware upload that triggers a reboot. If the reboot is too quick and prevents the POST response being sent, I found that a web browser will repeat the firmware upload once more after my firmware boots up again.

If it is not safe to retry the POST request, other options would be to avoid the server thinking the connection is stale:

* always do the POST request on a fresh HTTP connection
* "ping" the connection with a dummy request (e.g. OPTIONS, HEAD, or GET) immediately before the POST request

Another option that comes to mind is to try using the 100 Continue mechanism, but this is not a general solution and depends on the application.
History
Date User Action Args
2022-04-11 14:59:33adminsetgithub: 85517
2020-07-28 05:47:14martin.pantersetnosy: + martin.panter
messages: + msg374466
2020-07-27 08:04:37Michal Arbetsetmessages: + msg374365
2020-07-24 18:28:51gvanrossumsetstatus: open -> closed

nosy: + gvanrossum
messages: + msg374200

resolution: not a bug
stage: resolved
2020-07-21 13:18:28an7esetmessages: + msg374066
2020-07-20 23:17:21an7esetmessages: + msg374034
2020-07-20 13:48:46Michal Arbetsetmessages: + msg374008
2020-07-20 12:54:48an7esetnosy: + an7e
messages: + msg374004
2020-07-20 11:43:46Michal Arbetcreate