This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket)
Type: crash Stage:
Components: Extension Modules Versions: Python 3.0
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: ajaksu2, chris.mahan, giampaolo.rodola, vstinner
Priority: normal Keywords: patch

Created on 2008-12-31 13:00 by chris.mahan, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
from_filezilla.png chris.mahan, 2008-12-31 16:43 same folder from filezilla
Messages (29)
msg78603 - (view) Author: Christopher Mahan (chris.mahan) Date: 2008-12-31 12:59
The python program crashes (stops responding) both from the command line
and in IDLE (ver 3.0), after listing all the files in the approprate
directory, both with ftp.dir() and with ftp.retrlines('LIST') (see prog
listing below).

I have to close the window (both window shell and IDLE). No other key
combo responds.


Python env (from IDLE):
Python 3.0 (r30:67507, Dec  3 2008, 20:14:27) [MSC v.1500 32 bit
(Intel)] on win32

Program:

import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername',
passwd='thepassword')
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()

The username and password are correct.
I am able to use the ftp resource with filezilla (3.1.6) with no problem.

This runs on a Windows Server 2003 Standard Edition, Service Pack 2.

This is also a completely clean install of Python 3.0. I installed it,
then wrote this program.

Python 2.5 is installed on this server and runs fine.

Any questions: chris.mahan@gmail.com
msg78604 - (view) Author: Christopher Mahan (chris.mahan) Date: 2008-12-31 13:12
Update: Ran the same code with python 2.6.1 on the same computer, and
that worked fine.
msg78618 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-31 16:06
Can you paste the expected result of ftp.retrlines('LIST')? Does a 
directory contains a non-ASCII character?
msg78630 - (view) Author: Christopher Mahan (chris.mahan) Date: 2008-12-31 16:43
the output: just before the non-responsiveness:

-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
msg78633 - (view) Author: Christopher Mahan (chris.mahan) Date: 2008-12-31 16:48
Added file screenshot of filezilla view of the folder in question.
msg78636 - (view) Author: Christopher Mahan (chris.mahan) Date: 2008-12-31 17:09
The list does not seem to contain non-ascii characters.
msg78720 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-01 19:09
I'm unable to reproduce the issue. I tried to create files on my FTP 
server with non-ASCII characters or spaces in the filenames, but 
everything is fine. Can you reproduce the problem outside IDLE? Or is 
the issue specific to IDLE? I mean: write a script and execute "python 
script.py" in a terminal.
msg78726 - (view) Author: Daniel Diniz (ajaksu2) * (Python triager) Date: 2009-01-01 20:31
Hi Chris

Since dir calls retrlines and retrlines has a 'while 1:' loop, the bug
probably comes from there. Either it hangs in the fp.readline call or
the break condition is never met.

Can you put some print diagnostics inside Lib/ftplib.py->FTP.retrlines
(around line 405) to help pinpointing the issue?

HTH,
Daniel
msg78727 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-01 20:41
> Can you put some print diagnostics inside Lib/ftplib.py (...)

<your ftp object>.set_debuglevel(2) already prints a lot of 
informations.
msg78760 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-02 06:15
I modified the program by adding line: ftp.set_debuglevel(2)
START LISTING----------------
import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='myuserid',
passwd='mypassword')
ftp.set_debuglevel(2)
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()
END LISTING ---------------

Running from idle 1.2.4 with python 2.5.4 gives this output:

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\r\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\r\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,234,6).\r\n'
*resp* '227 Entering Passive Mode (72,21,82,190,234,6).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\r\n'
*resp* '150 Opening ASCII mode data connection for file list'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'


Then running with idle 3.0 with python 3.0 gives the following result.

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,148,178).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,148,178).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg

Note that the last two lines are missing.


From a command prompt:
C:\>c:\python30\python.exe c:\python_scripts\python3\candee_processor.py
*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,219,8).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,219,8).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg

same command prompt but with python 2.5:

C:\>c:\python25\python.exe c:\python_scripts\python3\candee_processor.py
*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\r\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\r\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,147,65).\r\n'
*resp* '227 Entering Passive Mode (72,21,82,190,147,65).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\r\n'
*resp* '150 Opening ASCII mode data connection for file list'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'

C:\>
msg78761 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-02 06:35
I went into the source for Libs/ftplib.py  and I noted that line 423 is: 

if self.debugging > 2: print('*retr*', repr(line))

so I changed the debuglevel to 3, as such:  ftp.set_debuglevel(3)
and I got these last 4 lines:

-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'

First line, the last line of the listing.
second line, the debug, showing the last  line read
third and fourth lines: the program actually completed and didn't hang!

still digging
msg78763 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-02 06:50
I added the following two lines and I was able to run the code.


Before: 
            line = fp.readline()
            if self.debugging > 2: print('*retr*', repr(line))
            if not line:
                break
            if line[-2:] == CRLF:
                line = line[:-2]
            elif line[-1:] == '\n':
                line = line[:-1]
            callback(line)

After: 
            line = fp.readline()
            if self.debugging > 2: print('*retr*', repr(line))
            if not line:
                break
            if line == '':
                break
            if line[-2:] == CRLF:
                line = line[:-2]
            elif line[-1:] == '\n':
                line = line[:-1]
            callback(line)

As a patch:

426,427d425
<             if line == '':
<                 break

What else do I need to do?
msg78799 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-02 14:16
Can you paste the full output with debuglevel=3?

"if not line:" should be equivalent to "if len(line) == 0:" and to "if 
line == '':".

ftp.dir() is equivalent to ftp.retrlines('LIST').

By default, the socket created to get the result of LIST has no 
timeout (is blocking). So fp.readline() only returns an empty string 
when the socket is closed (by the server).

fp is a io.BufferedReader(SocketIO(socket, "r"), 
io.DEFAULT_BUFFER_SIZE).
msg78802 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-02 14:45
Full output with debuglevel 3

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,217,227).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,217,227).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
*retr* '-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski
Beat - Why.mp3\n'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
*retr* '-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi\n'
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
*retr* '-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28
test9_lg.wmv\n'
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28
test9_lg.wmv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04
test9_med.flv\n'
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04
test9_med.flv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53
test9_sm.flv\n'
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53
test9_sm.flv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
msg78804 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-02 14:48
haypo writes:
>By default, the socket created to get the result of LIST has no 
>timeout (is blocking). So fp.readline() only returns an empty string 
>when the socket is closed (by the server).

Even if the server closed the socket and fp.readline() returned an empty
string, the prog should not hang, no?
msg78879 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2009-01-02 21:28
I'm unable to reproduce the issue.
Could you please repeat the test by using the "timeout" parameter as 
such?

< ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername', 
passwd='thepassword')
> ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername', 
passwd='thepassword', timeout=2)
msg78914 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-03 00:56
Per Giampaolo's suggestion, I added the timeout. Program listing and
output below:

-----begin listing----
import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername',
passwd='thepassword', timeout=2)
ftp.set_debuglevel(3)
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()
----end listing----

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,172,22).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,172,22).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
*retr* '-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski
Beat - Why.mp3\n'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
*retr* '-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi\n'
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
*retr* '-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28
test9_lg.wmv\n'
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28
test9_lg.wmv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04
test9_med.flv\n'
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04
test9_med.flv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53
test9_sm.flv\n'
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53
test9_sm.flv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
Traceback (most recent call last):
  File "C:\python_scripts\python3\candee_processor.py", line 6, in <module>
    ftp.dir()
  File "c:\python30\lib\ftplib.py", line 511, in dir
    self.retrlines(cmd, func)
  File "c:\python30\lib\ftplib.py", line 433, in retrlines
    return self.voidresp()
  File "c:\python30\lib\ftplib.py", line 225, in voidresp
    resp = self.getresp()
  File "c:\python30\lib\ftplib.py", line 211, in getresp
    resp = self.getmultiline()
  File "c:\python30\lib\ftplib.py", line 197, in getmultiline
    line = self.getline()
  File "c:\python30\lib\ftplib.py", line 184, in getline
    line = self.file.readline()
  File "C:\Python30\lib\io.py", line 1813, in readline
    while self._read_chunk():
  File "C:\Python30\lib\io.py", line 1560, in _read_chunk
    input_chunk = self.buffer.read1(self._CHUNK_SIZE)
  File "C:\Python30\lib\io.py", line 994, in read1
    self._peek_unlocked(1)
  File "C:\Python30\lib\io.py", line 981, in _peek_unlocked
    current = self.raw.read(to_read)
  File "C:\Python30\lib\io.py", line 575, in read
    n = self.readinto(b)
  File "C:\Python30\lib\socket.py", line 214, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out
msg78918 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-03 01:14
The traceback shows that the problem is not related to the socket 
created to retrieve the content of the directory listing, but 
the "main" socket (the one used for the whole ftp session) used by 
dir() command the retrieve the command answer code:

   Traceback (most recent call last):
     ftp.dir()
      => self.retrlines(cmd, func)
      => return self.voidresp()
      => line = self.getline()
      => line = self.file.readline() (ftplib.py:184)
      ...
    socket.timeout: timed out

I don't understand why the readline() blocks. It's maybe related to 
the newline: voidresp() uses getline() which reads a line using 
self.file. The file is created by self.sock.makefile('r', 
encoding=self.encoding) and so the newline is always \n whereas ftplib 
of Python2 uses \r\n (CRLF) or \n (LF).
msg78919 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-03 01:16
Christopher: Can you paste the full output with debuglevel=3 of Python 
2.x (python2.5)?
msg78920 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-03 01:36
When running scritp with python 2.5.4, got this error. I'll remove the
"timeout=2" and run again. see below.

Traceback (most recent call last):
  File "C:\python_scripts\python3\candee_processor.py", line 3, in <module>
    ftp = ftplib.FTP('ftp.edgecastcdn.net',
user='client1@arcostream.com', passwd='1210184520', timeout=2)
TypeError: __init__() got an unexpected keyword argument 'timeout'

Same without the timeout argument:

>>> 
*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\r\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\r\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,228,195).\r\n'
*resp* '227 Entering Passive Mode (72,21,82,190,228,195).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\r\n'
*resp* '150 Opening ASCII mode data connection for file list'
*retr* '-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski
Beat - Why.mp3\r\n'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
*retr* '-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53
test9.avi\r\n'
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
*retr* '-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28
test9_lg.wmv\r\n'
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28
test9_lg.wmv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04
test9_med.flv\r\n'
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04
test9_med.flv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53
test9_sm.flv\r\n'
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53
test9_sm.flv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'
>>>
msg79184 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-05 19:08
Update:

I requested a test ftp account from edgecast. The tech's response was
that he would file a request for it.  Will update.
msg79202 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-05 21:43
Edgecast.com support created a test ftp account for the purpose of
troubleshooting this issue. Please email me (chris.mahan@gmail.com) and
I'll forward you the login info.
msg79205 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-05 22:40
Ok, some news about this bug (I tested the ftp test account): the 
problem is that the data socket is not closed when self.voidresp() is 
called. I can't see close() syscall before the call to 
self.voidresp(). SocketIO.close() does nothing: it does not really 
close the socket, just the file. And socket.close() does also nothing 
because there is still an io reference... the closed file :-/ The io 
reference counter is only updated when the file is destroyed.
   conn = <create a socket>
   # conn._io_refs=0
   fp = conn.makefile('r', encoding=self.encoding)
   # conn._io_refs=1
   ...
   fp.close()    # only set fp.closed to True
   conn.close()  # only set conn._closed to True
   # conn._io_refs=1 and the socket (file descriptor) is still open
   del fp
   # conn._io_refs=0 -> the socket is finally closed!
msg79207 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-05 23:04
Patch: SocketIO.close() decrements the io reference of its socket, so 
socket.close() will really close the socket.

About the FTP server ftp2.edgecastcdn.net, it looks like the server 
doesn't write "226 Transfert completed" until the data socket is 
really closed. The server says that it's running EdgeFTPD 1.1. nmap 
says that the ftp server is ProFTPD 1.2.10 running on Linux 2.6.
msg79210 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-05 23:13
Issue #3826 is the same issue and has already patch with 2 unit tests. 
The patch is different: it uses "del self._sock".
msg79216 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-05 23:43
New patch:
 - Change _socket.fileno() behaviour: raise socket.error("I/O 
operation on closed socket") (instead of returning -1) if the socket 
is closed
 - Change SocketIO.fileno(): raise a socket.error("I/O operation on 
closed socket") if the file is closed, even if the socket is still 
open

TODO: Check all operations on closed _socket, socket or SocketIO and 
make sure that it raises also an error.

exakrun on IRC:
   What's the fascination with -1?
   Isn't this Python?  How about an exception? Or at least a different 
type... None, perhaps?
   Returning -1 is what you would do in a language like C where you 
*can't* do something nice.
msg79218 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-06 00:39
I created a separated patch to block operation on closed socket: Issue 
#4853.

So I simplified the patch to change on Modules/socketmodule.c and I 
reused self._checkClosed() in SocketIO.
msg79219 - (view) Author: Christopher Mahan (chris.mahan) Date: 2009-01-06 01:06
edgecast support said the ftp server is ProFTPD version 1.3.1.
msg79222 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-06 01:15
I'm closing this issue because it's a duplicate of #3826, issue with a 
longer history and more details about the different problems of 
closing a socket.
History
Date User Action Args
2022-04-11 14:56:43adminsetgithub: 49041
2009-01-06 01:15:09vstinnersetstatus: open -> closed
resolution: duplicate
messages: + msg79222
2009-01-06 01:09:58vstinnersetfiles: - socket_real_close-4.patch
2009-01-06 01:06:23chris.mahansetmessages: + msg79219
2009-01-06 00:39:45vstinnersetfiles: - socket_real_close-3.patch
2009-01-06 00:39:39vstinnersetfiles: + socket_real_close-4.patch
messages: + msg79218
2009-01-05 23:45:16vstinnersettitle: retrlines('LIST') and dir hang at end of listing in ftplib (python3.0) -> ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket)
2009-01-05 23:43:50vstinnersetfiles: - socket_real_close-2.patch
2009-01-05 23:43:45vstinnersetfiles: + socket_real_close-3.patch
messages: + msg79216
2009-01-05 23:13:46vstinnersetmessages: + msg79210
2009-01-05 23:08:31vstinnersetfiles: + socket_real_close-2.patch
2009-01-05 23:08:14vstinnersetfiles: - socket_real_close.patch
2009-01-05 23:04:08vstinnersetfiles: + socket_real_close.patch
keywords: + patch
messages: + msg79207
2009-01-05 22:40:30vstinnersetmessages: + msg79205
2009-01-05 21:43:54chris.mahansetmessages: + msg79202
2009-01-05 19:08:40chris.mahansetmessages: + msg79184
2009-01-03 01:36:36chris.mahansetmessages: + msg78920
2009-01-03 01:16:13vstinnersetmessages: + msg78919
2009-01-03 01:14:58vstinnersetmessages: + msg78918
2009-01-03 00:56:08chris.mahansetmessages: + msg78914
2009-01-02 21:28:37giampaolo.rodolasetmessages: + msg78879
2009-01-02 21:18:58giampaolo.rodolasetnosy: + giampaolo.rodola
2009-01-02 14:48:25chris.mahansetmessages: + msg78804
2009-01-02 14:45:45chris.mahansetmessages: + msg78802
2009-01-02 14:16:31vstinnersetmessages: + msg78799
2009-01-02 06:50:17chris.mahansetmessages: + msg78763
2009-01-02 06:35:13chris.mahansetmessages: + msg78761
2009-01-02 06:15:43chris.mahansetmessages: + msg78760
2009-01-01 20:41:43vstinnersetmessages: + msg78727
2009-01-01 20:31:27ajaksu2setnosy: + ajaksu2
messages: + msg78726
2009-01-01 19:09:10vstinnersetmessages: + msg78720
2008-12-31 17:09:51chris.mahansetmessages: + msg78636
2008-12-31 16:48:08chris.mahansetmessages: + msg78633
2008-12-31 16:43:10chris.mahansetfiles: + from_filezilla.png
messages: + msg78630
2008-12-31 16:06:06vstinnersetnosy: + vstinner
messages: + msg78618
2008-12-31 13:12:14chris.mahansetmessages: + msg78604
2008-12-31 13:01:59chris.mahansettitle: retrlines('LIST') and dir hang at end of listing in ftplib -> retrlines('LIST') and dir hang at end of listing in ftplib (python3.0)
2008-12-31 13:00:00chris.mahancreate