classification
Title: urllib ignores FTP 226 response, breaking further FTP requests
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ivan.Pozdeev, giampaolo.rodola
Priority: normal Keywords: patch

Created on 2016-12-10 14:54 by Ivan.Pozdeev, last changed 2016-12-15 05:13 by Ivan.Pozdeev.

Files
File name Uploaded Description Edit
ftp_error_illustration.txt Ivan.Pozdeev, 2016-12-10 14:54 FTP traffic that leads to error
105577.patch Ivan.Pozdeev, 2016-12-10 19:58 patch for 2.7 review
105577.patch Ivan.Pozdeev, 2016-12-11 03:51 patch for 3.x review
Messages (4)
msg282853 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-12-10 14:54
>>> urllib.urlretrieve("ftp://ftp.zlatkovic.com/pub/libxml/md5sum.txt","t.bin")
('t.bin', <mimetools.Message instance at 0x00D8C5D0>)
>>> urllib.urlretrieve("ftp://ftp.zlatkovic.com/pub/libxml/md5sum.txt","t.bin")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Py\lib\urllib.py", line 98, in urlretrieve
    return opener.retrieve(url, filename, reporthook, data)
  File "C:\Py\lib\urllib.py", line 245, in retrieve
    fp = self.open(url, data)
  File "C:\Py\lib\urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "C:\Py\lib\urllib.py", line 558, in open_ftp
    (fp, retrlen) = self.ftpcache[key].retrfile(file, type)
  File "C:\Py\lib\urllib.py", line 906, in retrfile
    conn, retrlen = self.ftp.ntransfercmd(cmd)
  File "C:\Py\lib\ftplib.py", line 334, in ntransfercmd
    host, port = self.makepasv()
  File "C:\Py\lib\ftplib.py", line 312, in makepasv
    host, port = parse227(self.sendcmd('PASV'))
  File "C:\Py\lib\ftplib.py", line 830, in parse227
    raise error_reply, resp
IOError: [Errno ftp error] 200 Type set to I

The cause is the 226 Transfer Complete response being ignored, causing further response lines to be incorrectly matched to request lines.

The bug is a result of issue16270 for 3.x and issue26960 for 2.7.
msg282855 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-12-10 15:17
The attached patch fixes this case. It still leaves the following problem:

>>> u=urllib.urlopen("ftp://ftp.zlatkovic.com/pub/libxml/md5sum.txt").read()
>>> urllib.urlretrieve("ftp://ftp.zlatkovic.com/pub/libxml/md5sum.txt","t.bin")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Py\lib\urllib.py", line 98, in urlretrieve
    return opener.retrieve(url, filename, reporthook, data)
  File "C:\Py\lib\urllib.py", line 245, in retrieve
    fp = self.open(url, data)
  File "C:\Py\lib\urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "C:\Py\lib\urllib.py", line 558, in open_ftp
    (fp, retrlen) = self.ftpcache[key].retrfile(file, type)
  File "C:\Py\lib\urllib.py", line 906, in retrfile
    conn, retrlen = self.ftp.ntransfercmd(cmd)
  File "C:\Py\lib\ftplib.py", line 334, in ntransfercmd
    host, port = self.makepasv()
  File "C:\Py\lib\ftplib.py", line 312, in makepasv
    host, port = parse227(self.sendcmd('PASV'))
  File "C:\Py\lib\ftplib.py", line 830, in parse227
    raise error_reply, resp
IOError: [Errno ftp error] 200 Type set to I

I reckon this can't be fully fixed without fixing `ftplib' first, see issue25458.
msg283219 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2016-12-14 21:16
I can't reproduce the issue on Python 3.5 (but I can on 2.7).
msg283241 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-12-15 05:13
Was fixed in c741ba9e37ef (3.2) . urllib gets around the ftplib issue by closing the session after each transfer.
History
Date User Action Args
2016-12-15 05:13:53Ivan.Pozdeevsetmessages: + msg283241
2016-12-14 21:16:58giampaolo.rodolasetnosy: + giampaolo.rodola

messages: + msg283219
versions: - Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7
2016-12-11 03:51:29Ivan.Pozdeevsetfiles: + 105577.patch
2016-12-10 19:58:40Ivan.Pozdeevsetfiles: + 105577.patch
keywords: + patch
2016-12-10 15:17:39Ivan.Pozdeevsetmessages: + msg282855
2016-12-10 14:54:33Ivan.Pozdeevcreate