ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket) #49041

chrismahan · 2008-12-31T13:00:01Z

BPO	4791
Nosy	@vstinner, @giampaolo, @devdanzin
Files	from_filezilla.png: same folder from filezilla

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-01-06.01:15:09.830>
created_at = <Date 2008-12-31.13:00:00.861>
labels = ['extension-modules', 'type-crash']
title = "ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket)"
updated_at = <Date 2009-01-06.01:15:09.829>
user = 'https://bugs.python.org/chrismahan'

bugs.python.org fields:

activity = <Date 2009-01-06.01:15:09.829>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2009-01-06.01:15:09.830>
closer = 'vstinner'
components = ['Extension Modules']
creation = <Date 2008-12-31.13:00:00.861>
creator = 'chris.mahan'
dependencies = []
files = ['12508']
hgrepos = []
issue_num = 4791
keywords = ['patch']
message_count = 29.0
messages = ['78603', '78604', '78618', '78630', '78633', '78636', '78720', '78726', '78727', '78760', '78761', '78763', '78799', '78802', '78804', '78879', '78914', '78918', '78919', '78920', '79184', '79202', '79205', '79207', '79210', '79216', '79218', '79219', '79222']
nosy_count = 4.0
nosy_names = ['vstinner', 'giampaolo.rodola', 'ajaksu2', 'chris.mahan']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue4791'
versions = ['Python 3.0']

chrismahan · 2008-12-31T12:59:59Z

The python program crashes (stops responding) both from the command line
and in IDLE (ver 3.0), after listing all the files in the approprate
directory, both with ftp.dir() and with ftp.retrlines('LIST') (see prog
listing below).

I have to close the window (both window shell and IDLE). No other key
combo responds.

Python env (from IDLE):
Python 3.0 (r30:67507, Dec 3 2008, 20:14:27) [MSC v.1500 32 bit
(Intel)] on win32

Program:

import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername',
passwd='thepassword')
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()

The username and password are correct.
I am able to use the ftp resource with filezilla (3.1.6) with no problem.

This runs on a Windows Server 2003 Standard Edition, Service Pack 2.

This is also a completely clean install of Python 3.0. I installed it,
then wrote this program.

Python 2.5 is installed on this server and runs fine.

Any questions: chris.mahan@gmail.com

chrismahan · 2008-12-31T13:12:15Z

Update: Ran the same code with python 2.6.1 on the same computer, and
that worked fine.

vstinner · 2008-12-31T16:06:06Z

Can you paste the expected result of ftp.retrlines('LIST')? Does a
directory contains a non-ASCII character?

chrismahan · 2008-12-31T16:43:10Z

the output: just before the non-responsiveness:

-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi
-rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg

chrismahan · 2008-12-31T16:48:08Z

Added file screenshot of filezilla view of the folder in question.

chrismahan · 2008-12-31T17:09:51Z

The list does not seem to contain non-ascii characters.

vstinner · 2009-01-01T19:09:10Z

I'm unable to reproduce the issue. I tried to create files on my FTP
server with non-ASCII characters or spaces in the filenames, but
everything is fine. Can you reproduce the problem outside IDLE? Or is
the issue specific to IDLE? I mean: write a script and execute "python
script.py" in a terminal.

devdanzin · 2009-01-01T20:31:27Z

Hi Chris

Since dir calls retrlines and retrlines has a 'while 1:' loop, the bug
probably comes from there. Either it hangs in the fp.readline call or
the break condition is never met.

Can you put some print diagnostics inside Lib/ftplib.py->FTP.retrlines
(around line 405) to help pinpointing the issue?

HTH,
Daniel

vstinner · 2009-01-01T20:41:43Z

Can you put some print diagnostics inside Lib/ftplib.py (...)

<your ftp object>.set_debuglevel(2) already prints a lot of
informations.

chrismahan · 2009-01-02T06:15:41Z

I modified the program by adding line: ftp.set_debuglevel(2)
START LISTING----------------

import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='myuserid',
passwd='mypassword')
ftp.set_debuglevel(2)
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()
END LISTING

Running from idle 1.2.4 with python 2.5.4 gives this output:

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\r\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\r\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,234,6).\r\n'
*resp* '227 Entering Passive Mode (72,21,82,190,234,6).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\r\n'
*resp* '150 Opening ASCII mode data connection for file list'
-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi
-rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'

Then running with idle 3.0 with python 3.0 gives the following result.

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,148,178).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,148,178).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi
-rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg

Note that the last two lines are missing.

From a command prompt:
C:\>c:\python30\python.exe c:\python_scripts\python3\candee_processor.py
*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,219,8).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,219,8).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi
-rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg

same command prompt but with python 2.5:

C:\>c:\python25\python.exe c:\python_scripts\python3\candee_processor.py
*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\r\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\r\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,147,65).\r\n'
*resp* '227 Entering Passive Mode (72,21,82,190,147,65).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\r\n'
*resp* '150 Opening ASCII mode data connection for file list'
-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi
-rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'

C:\>

chrismahan · 2009-01-02T06:35:13Z

I went into the source for Libs/ftplib.py and I noted that line 423 is:

if self.debugging > 2: print('*retr*', repr(line))

so I changed the debuglevel to 3, as such: ftp.set_debuglevel(3)
and I got these last 4 lines:

-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'

First line, the last line of the listing.
second line, the debug, showing the last line read
third and fourth lines: the program actually completed and didn't hang!

still digging

chrismahan · 2009-01-02T06:50:17Z

I added the following two lines and I was able to run the code.

Before:
line = fp.readline()
if self.debugging > 2: print('*retr*', repr(line))
if not line:
break
if line[-2:] == CRLF:
line = line[:-2]
elif line[-1:] == '\n':
line = line[:-1]
callback(line)

After:
line = fp.readline()
if self.debugging > 2: print('*retr*', repr(line))
if not line:
break
if line == '':
break
if line[-2:] == CRLF:
line = line[:-2]
elif line[-1:] == '\n':
line = line[:-1]
callback(line)

As a patch:

426,427d425
< if line == '':
< break

What else do I need to do?

vstinner · 2009-01-02T14:16:31Z

Can you paste the full output with debuglevel=3?

"if not line:" should be equivalent to "if len(line) == 0:" and to "if
line == '':".

ftp.dir() is equivalent to ftp.retrlines('LIST').

By default, the socket created to get the result of LIST has no
timeout (is blocking). So fp.readline() only returns an empty string
when the socket is closed (by the server).

fp is a io.BufferedReader(SocketIO(socket, "r"),
io.DEFAULT_BUFFER_SIZE).

chrismahan · 2009-01-02T14:45:45Z

Full output with debuglevel 3

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,217,227).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,217,227).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
*retr* '-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski
Beat - Why.mp3\n'
-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
*retr* '-rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi\n'
-rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi
*retr* '-rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28
test9_lg.wmv\n'
-rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv
*retr* '-rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28
test9_lg.wmv.jpg\n'
-rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg
*retr* '-rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04
test9_med.flv\n'
-rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv
*retr* '-rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04
test9_med.flv.jpg\n'
-rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg
*retr* '-rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53
test9_sm.flv\n'
-rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv
*retr* '-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53
test9_sm.flv.jpg\n'
-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''

chrismahan · 2009-01-02T14:48:25Z

haypo writes:

By default, the socket created to get the result of LIST has no
timeout (is blocking). So fp.readline() only returns an empty string
when the socket is closed (by the server).

Even if the server closed the socket and fp.readline() returned an empty
string, the prog should not hang, no?

giampaolo · 2009-01-02T21:28:37Z

I'm unable to reproduce the issue.
Could you please repeat the test by using the "timeout" parameter as
such?

< ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername',
passwd='thepassword')

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername',
passwd='thepassword', timeout=2)

chrismahan · 2009-01-03T00:56:07Z

Per Giampaolo's suggestion, I added the timeout. Program listing and
output below:

-----begin listing----

import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername',
passwd='thepassword', timeout=2)
ftp.set_debuglevel(3)
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()
----end listing

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,172,22).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,172,22).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
*retr* '-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski
Beat - Why.mp3\n'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
*retr* '-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi\n'
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
*retr* '-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28
test9_lg.wmv\n'
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28
test9_lg.wmv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04
test9_med.flv\n'
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04
test9_med.flv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53
test9_sm.flv\n'
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53
test9_sm.flv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
Traceback (most recent call last):
  File "C:\python_scripts\python3\candee_processor.py", line 6, in <module>
    ftp.dir()
  File "c:\python30\lib\ftplib.py", line 511, in dir
    self.retrlines(cmd, func)
  File "c:\python30\lib\ftplib.py", line 433, in retrlines
    return self.voidresp()
  File "c:\python30\lib\ftplib.py", line 225, in voidresp
    resp = self.getresp()
  File "c:\python30\lib\ftplib.py", line 211, in getresp
    resp = self.getmultiline()
  File "c:\python30\lib\ftplib.py", line 197, in getmultiline
    line = self.getline()
  File "c:\python30\lib\ftplib.py", line 184, in getline
    line = self.file.readline()
  File "C:\Python30\lib\io.py", line 1813, in readline
    while self._read_chunk():
  File "C:\Python30\lib\io.py", line 1560, in _read_chunk
    input_chunk = self.buffer.read1(self._CHUNK_SIZE)
  File "C:\Python30\lib\io.py", line 994, in read1
    self._peek_unlocked(1)
  File "C:\Python30\lib\io.py", line 981, in _peek_unlocked
    current = self.raw.read(to_read)
  File "C:\Python30\lib\io.py", line 575, in read
    n = self.readinto(b)
  File "C:\Python30\lib\socket.py", line 214, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

vstinner · 2009-01-03T01:14:58Z

The traceback shows that the problem is not related to the socket
created to retrieve the content of the directory listing, but
the "main" socket (the one used for the whole ftp session) used by
dir() command the retrieve the command answer code:

   Traceback (most recent call last):
     ftp.dir()
      => self.retrlines(cmd, func)
      => return self.voidresp()
      => line = self.getline()
      => line = self.file.readline() (ftplib.py:184)
      ...
    socket.timeout: timed out

I don't understand why the readline() blocks. It's maybe related to
the newline: voidresp() uses getline() which reads a line using
self.file. The file is created by self.sock.makefile('r',
encoding=self.encoding) and so the newline is always \n whereas ftplib
of Python2 uses \r\n (CRLF) or \n (LF).

vstinner · 2009-01-03T01:16:14Z

Christopher: Can you paste the full output with debuglevel=3 of Python
2.x (python2.5)?

chrismahan · 2009-01-03T01:36:35Z

When running scritp with python 2.5.4, got this error. I'll remove the
"timeout=2" and run again. see below.

Traceback (most recent call last):
  File "C:\python_scripts\python3\candee_processor.py", line 3, in <module>
    ftp = ftplib.FTP('ftp.edgecastcdn.net',
user='client1@arcostream.com', passwd='1210184520', timeout=2)
TypeError: __init__() got an unexpected keyword argument 'timeout'

Same without the timeout argument:

>>> 
*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\r\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\r\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,228,195).\r\n'
*resp* '227 Entering Passive Mode (72,21,82,190,228,195).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\r\n'
*resp* '150 Opening ASCII mode data connection for file list'
*retr* '-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski
Beat - Why.mp3\r\n'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
*retr* '-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53
test9.avi\r\n'
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
*retr* '-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28
test9_lg.wmv\r\n'
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28
test9_lg.wmv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04
test9_med.flv\r\n'
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04
test9_med.flv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53
test9_sm.flv\r\n'
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53
test9_sm.flv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'
>>>

chrismahan · 2009-01-05T19:08:40Z

Update:

I requested a test ftp account from edgecast. The tech's response was
that he would file a request for it. Will update.

chrismahan · 2009-01-05T21:43:54Z

Edgecast.com support created a test ftp account for the purpose of
troubleshooting this issue. Please email me (chris.mahan@gmail.com) and
I'll forward you the login info.

vstinner · 2009-01-05T22:40:30Z

Ok, some news about this bug (I tested the ftp test account): the
problem is that the data socket is not closed when self.voidresp() is
called. I can't see close() syscall before the call to
self.voidresp(). SocketIO.close() does nothing: it does not really
close the socket, just the file. And socket.close() does also nothing
because there is still an io reference... the closed file :-/ The io
reference counter is only updated when the file is destroyed.
conn = <create a socket>
# conn._io_refs=0
fp = conn.makefile('r', encoding=self.encoding)
# conn._io_refs=1
...
fp.close() # only set fp.closed to True
conn.close() # only set conn._closed to True
# conn._io_refs=1 and the socket (file descriptor) is still open
del fp
# conn._io_refs=0 -> the socket is finally closed!

vstinner · 2009-01-05T23:04:07Z

Patch: SocketIO.close() decrements the io reference of its socket, so
socket.close() will really close the socket.

About the FTP server ftp2.edgecastcdn.net, it looks like the server
doesn't write "226 Transfert completed" until the data socket is
really closed. The server says that it's running EdgeFTPD 1.1. nmap
says that the ftp server is ProFTPD 1.2.10 running on Linux 2.6.

vstinner · 2009-01-05T23:13:47Z

Issue bpo-3826 is the same issue and has already patch with 2 unit tests.
The patch is different: it uses "del self._sock".

vstinner · 2009-01-05T23:43:45Z

New patch:

Change _socket.fileno() behaviour: raise socket.error("I/O
operation on closed socket") (instead of returning -1) if the socket
is closed
Change SocketIO.fileno(): raise a socket.error("I/O operation on
closed socket") if the file is closed, even if the socket is still
open

TODO: Check all operations on closed _socket, socket or SocketIO and
make sure that it raises also an error.

exakrun on IRC:
What's the fascination with -1?
Isn't this Python? How about an exception? Or at least a different
type... None, perhaps?
Returning -1 is what you would do in a language like C where you
*can't* do something nice.

vstinner · 2009-01-06T00:39:39Z

I created a separated patch to block operation on closed socket: Issue
bpo-4853.

So I simplified the patch to change on Modules/socketmodule.c and I
reused self._checkClosed() in SocketIO.

chrismahan · 2009-01-06T01:06:23Z

edgecast support said the ftp server is ProFTPD version 1.3.1.

vstinner · 2009-01-06T01:15:09Z

I'm closing this issue because it's a duplicate of bpo-3826, issue with a
longer history and more details about the different problems of
closing a socket.

chrismahan mannequin added extension-modules C modules in the Modules dir type-crash A hard crash of the interpreter, possibly with a core dump labels Dec 31, 2008

chrismahan mannequin changed the title ~~retrlines('LIST') and dir hang at end of listing in ftplib~~ retrlines('LIST') and dir hang at end of listing in ftplib (python3.0) Dec 31, 2008

vstinner changed the title ~~retrlines('LIST') and dir hang at end of listing in ftplib (python3.0)~~ ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket) Jan 5, 2009

vstinner closed this as completed Jan 6, 2009

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket) #49041

ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket) #49041

chrismahan mannequin commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

vstinner commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

vstinner commented Jan 1, 2009

devdanzin mannequin commented Jan 1, 2009

vstinner commented Jan 1, 2009

chrismahan mannequin commented Jan 2, 2009

chrismahan mannequin commented Jan 2, 2009

chrismahan mannequin commented Jan 2, 2009

vstinner commented Jan 2, 2009

chrismahan mannequin commented Jan 2, 2009

chrismahan mannequin commented Jan 2, 2009

giampaolo commented Jan 2, 2009

chrismahan mannequin commented Jan 3, 2009

vstinner commented Jan 3, 2009

vstinner commented Jan 3, 2009

chrismahan mannequin commented Jan 3, 2009

chrismahan mannequin commented Jan 5, 2009

chrismahan mannequin commented Jan 5, 2009

vstinner commented Jan 5, 2009

vstinner commented Jan 5, 2009

vstinner commented Jan 5, 2009

vstinner commented Jan 5, 2009

vstinner commented Jan 6, 2009

chrismahan mannequin commented Jan 6, 2009

vstinner commented Jan 6, 2009

ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket) #49041

ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket) #49041

Comments

chrismahan mannequin commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

vstinner commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

chrismahan mannequin commented Dec 31, 2008

vstinner commented Jan 1, 2009

devdanzin mannequin commented Jan 1, 2009

vstinner commented Jan 1, 2009

chrismahan mannequin commented Jan 2, 2009

chrismahan mannequin commented Jan 2, 2009

chrismahan mannequin commented Jan 2, 2009

vstinner commented Jan 2, 2009

chrismahan mannequin commented Jan 2, 2009

chrismahan mannequin commented Jan 2, 2009

giampaolo commented Jan 2, 2009

chrismahan mannequin commented Jan 3, 2009

vstinner commented Jan 3, 2009

vstinner commented Jan 3, 2009

chrismahan mannequin commented Jan 3, 2009

chrismahan mannequin commented Jan 5, 2009

chrismahan mannequin commented Jan 5, 2009

vstinner commented Jan 5, 2009

vstinner commented Jan 5, 2009

vstinner commented Jan 5, 2009

vstinner commented Jan 5, 2009

vstinner commented Jan 6, 2009

chrismahan mannequin commented Jan 6, 2009

vstinner commented Jan 6, 2009