classification
Title: xml rpc server fails with connection reset by peer error no 104
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Jim.Jewett, iritkatriel, manu, martin.panter, r.david.murray
Priority: normal Keywords:

Created on 2016-12-14 09:53 by manu, last changed 2021-04-26 10:57 by iritkatriel. This issue is now closed.

Messages (14)
msg283180 - (view) Author: Manish Singh (manu) Date: 2016-12-14 09:53
I have used xml rpc library with transport http. My client and server are running on same host.

In normal load scenario(20% cpu usage, 80% memory usage, 18 GB memory free), some request of xml rpc client fails with connection reset by peer error. I have used xmlrpclib.ServerProxy() to call remote method on xml rpc server running on an empherial port.

This issue has happen many times.

log snippet,

  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1237, in request
    errcode, errmsg, headers = h.getreply()
  File "/usr/lib64/python2.6/httplib.py", line 1064, in getreply
    response = self._conn.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
    line = self.fp.readline()
  File "/usr/lib64/python2.6/socket.py", line 433, in readline
    data = recv(1)
error: [Errno 104] Connection reset by peer
msg283247 - (view) Author: Manish Singh (manu) Date: 2016-12-15 07:04
Hi loewis,

Can you please look into this issue
msg283307 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-12-15 14:23
Connection reset by peer means the far end has terminated the connection.  You'll probably need to do some protocol analysis (wireshark or a similar tool) to find out why the connection is being dropped, or perhaps you can look at the far end logging.  As it is, there isn't enough information here to speculate as to a cause, or if there is any bug in the python library code.

(FYI Martin von Loewis hasn't had much time for CPython development lately, so I doubt he'll respond.)
msg283366 - (view) Author: Manish Singh (manu) Date: 2016-12-16 06:22
Hi David Murray,
Thanks for replying.

I have mentioned that my client and server are running on same host. So communication is going through localhost address.

I have checked for possible reasons of connection reset by peer error, below are the possible causes,

1) The machine is connected to the network, and the network is not responsive. 
2) The other side of the connection is not running normally. 
3) There are not enough system resources available. Free up system resources if they are running low. 

Possibility for 1 and 2 are not applicable as it is loopback communication.

For Possibility 3, I have already checked system resource and there are enough resources. 

You can check below stack overflow link.
http://stackoverflow.com/questions/1434451/what-does-connection-reset-by-peer-mean?rq=1
msg283388 - (view) Author: Manish Singh (manu) Date: 2016-12-16 08:00
Hi David Murray,

Please refer this link,

http://stackoverflow.com/questions/383738/104-connection-reset-by-peer-socket-error-or-when-does-closing-a-socket-resu
msg283670 - (view) Author: Manish Singh (manu) Date: 2016-12-20 06:40
Hi Davud Murray,

Can you reply to above comments.
msg283692 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-12-20 14:40
I'm sorry, I don't have time to dig in to this right now.  Hopefully someone else can pick it up.  You might try posting on the python-list mailing list, there are several core devs who hang out there and might be interested in the problem.
msg283741 - (view) Author: Manish Singh (manu) Date: 2016-12-21 11:08
Hi David,

How can i port it to main python-dev list.

Need to mail on it or some other group is there so that we can send the message to it.
msg283753 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-12-21 12:55
The python-list mailing list subscription info can be found on the mail.python.org web page.  It is also gatewayed to a usenet news group.
msg283866 - (view) Author: Jim Jewett (Jim.Jewett) * (Python triager) Date: 2016-12-23 02:44
When I see a message like that, it normally means there is a firewall getting in my way.
msg283869 - (view) Author: Manish Singh (manu) Date: 2016-12-23 03:29
Hi Jim,
Thank you for replying.

As you are suggesting that there is a firewall setting.

Then in that case every request should fail, right?

But in my case only few request fail(4 out 20 requests).

Have you seen my mail sent to python-dev group through gmail?
msg283870 - (view) Author: Manish Singh (manu) Date: 2016-12-23 03:45
Below is the information sent to python-dev group....

=======================================================================

[ Issue ]
I have used xml rpc library with transport as http. My client and server are running on same host.

Some xml rpc requests fail with connection reset by peer error number. I have used xmlrpclib.ServerProxy() to call remote method on xml rpc server running on an ephemeral port.

This issue has happen many times.

log snippet,

  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1237, in request
    errcode, errmsg, headers = h.getreply()
  File "/usr/lib64/python2.6/httplib.py", line 1064, in getreply
    response = self._conn.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
    line = self.fp.readline()
  File "/usr/lib64/python2.6/socket.py", line 433, in readline
    data = recv(1)
error: [Errno 104] Connection reset by peer

[ Test Environment ]
RHEL 6 with linux kernel 2.6.32-504.16.2.el6.
Python 2.6.6
glibc-2.12-1.149.7


[ Possible Reasons for it ]

1) The machine is connected to the network, and the network is not responsive. 
2) The other side of the connection is not running normally. 
3) There are not enough system resources available. Free up system resources if they are running low. 

Possibility for 1 and 2 are not applicable as it is loop back communication(Client and Server running on same machine).
For Possibility 3, I have already checked system resource and there are enough resources(80% RAM used, 20% cpu usage, around 10 GB RAM free).

I checked for other reasons and i found that this issue may be related with GIL,
Please refer this link,
http://stackoverflow.com/questions/383738/104-connection-reset-by-peer-socket-error-or-when-does-closing-a-socket-resu 

1> Can you please let me know, is it really a issue realted with GIL?
2> If yes, How to resolve this issue?
3> If no, what other reason may exists for such failure. [Note: Those rpc requests fail which return python's dictionary data to client]


========================================================================
msg283908 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-12-23 23:32
Python 2.6 is quite old and doesn’t even receive security patches any more as far as I know. I would start by trying 2.7, or failing that, try backporting the changes from Issue 6267. My guess is this is related to persistent HTTP connections being dropped.

I don’t think the bug tracker is the right place to discuss 2.6 these days. Even if you can find someone interested in helping with 2.6, you may need to provide more information (e.g. client and server code) to produce the problem.
msg391899 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-04-26 10:57
2.7 is no longer maintained. Please create a new issue with full reproduction information if you are still having this problem in 3.8+.
History
Date User Action Args
2021-04-26 10:57:09iritkatrielsetstatus: open -> closed

nosy: + iritkatriel
messages: + msg391899

stage: resolved
2016-12-23 23:32:32martin.pantersetresolution: out of date

messages: + msg283908
nosy: + martin.panter
2016-12-23 03:45:28manusetmessages: + msg283870
2016-12-23 03:29:56manusetmessages: + msg283869
2016-12-23 02:44:24Jim.Jewettsetnosy: + Jim.Jewett
messages: + msg283866
2016-12-21 12:55:07r.david.murraysetmessages: + msg283753
2016-12-21 11:08:09manusetmessages: + msg283741
2016-12-20 14:40:14r.david.murraysetnosy: - loewis
messages: + msg283692
2016-12-20 06:40:33manusetmessages: + msg283670
2016-12-16 08:00:29manusetmessages: + msg283388
2016-12-16 06:22:13manusetmessages: + msg283366
2016-12-15 14:23:41r.david.murraysetnosy: + r.david.murray
messages: + msg283307
2016-12-15 07:05:52manusetnosy: + loewis
2016-12-15 07:04:07manusetmessages: + msg283247
2016-12-14 09:53:43manucreate