classification
Title: FTP_TLS errors when use certain subcommands
Type: behavior Stage: test needed
Components: SSL, Windows Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Grégoire Chauvet, Matthieu Pepin, christian.heimes, corona10, jonathan-lp, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-10-08 17:08 by jonathan-lp, last changed 2018-05-14 16:39 by Grégoire Chauvet.

Messages (8)
msg303914 - (view) Author: Jonathan (jonathan-lp) Date: 2017-10-08 17:08
Using Python 3.6.3.

The below issue only happens with FTP_TLS. It works fine via a plain FTP connection.

I am connecting to an FTP to using `ftplib` via FTP_TLS.

	ftps = ftplib.FTP_TLS('example.com', timeout=5)

	# TLS is more secure than SSL
	ftps.ssl_version = ssl.PROTOCOL_TLS

	# login before securing control channel
	ftps.login('username@example.com', 'password')

	# switch to secure data connection
	ftps.prot_p()

	# Explicitly set Passive mode
	ftps.set_pasv(True)

This all works. I can send `ftps.mkd('mydir')` (make directory) and `ftps.cwd('mydir')` (change working dir), and both work fine.

But if I send **any** of these (they're all basically synonyms as far as I can tell):

			ftps.dir()
			ftps.nlst()
			ftps.retrlines('LIST')
			ftps.retrlines('MLSD')

Then I get back an exception (below also includes all ftplib debug info as well; generally matches up with what FileZilla shows too):

	*cmd* 'AUTH TLS'
	*put* 'AUTH TLS\r\n'
	*get* '234 AUTH TLS OK.\n'
	*resp* '234 AUTH TLS OK.'
	*cmd* 'USER username@example.com'
	*put* 'USER username@example.com\r\n'
	*get* '331 User username@example.com OK. Password required\n'
	*resp* '331 User username@example.com OK. Password required'
	*cmd* 'PASS ******************************'
	*put* 'PASS ******************************\r\n'
	*get* '230 OK. Current restricted directory is /\n'
	*resp* '230 OK. Current restricted directory is /'
	*cmd* 'PBSZ 0'
	*put* 'PBSZ 0\r\n'
	*get* '200 PBSZ=0\n'
	*resp* '200 PBSZ=0'
	*cmd* 'PROT P'
	*put* 'PROT P\r\n'
	*get* '200 Data protection level set to "private"\n'
	*resp* '200 Data protection level set to "private"'
	*cmd* 'MKD mydir'
	*put* 'MKD mydir\r\n'
	*get* '257 "mydir" : The directory was successfully created\n'
	*resp* '257 "mydir" : The directory was successfully created'
	*cmd* 'CWD mydir'
	*put* 'CWD mydir\r\n'
	*get* '250 OK. Current directory is /mydir\n'
	*resp* '250 OK. Current directory is /mydir'
	*cmd* 'TYPE A'
	*put* 'TYPE A\r\n'
	*get* '200 TYPE is now ASCII\n'
	*resp* '200 TYPE is now ASCII'
	*cmd* 'PASV'
	*put* 'PASV\r\n'
	*get* '227 Entering Passive Mode (8,8,8,8,8,8)\n'
	*resp* '227 Entering Passive Mode (8,8,8,8,8,8)'
	*cmd* 'MLSD'
	*put* 'MLSD\r\n'
	*get* '150 Accepted data connection\n'
	*resp* '150 Accepted data connection'
	Traceback (most recent call last):
	  File "c:\my_script.py", line 384, in run_ftps
		ftps.retrlines('MLSD')
	  File "c:\libs\Python36\lib\ftplib.py", line 485, in retrlines
		conn.unwrap()
	  File "C:\libs\Python36\lib\ssl.py", line 1051, in unwrap
		s = self._sslobj.unwrap()
	  File "C:\libs\Python36\lib\ssl.py", line 698, in unwrap
		return self._sslobj.shutdown()
	OSError: [Errno 0] Error

The same FTP command (LIST) works fine via filezilla.

The closest thing I can find with googling is this: https://bugs.python.org/msg253161 - and I'm not sure if it's related or relevant.

Short version: What does "OSError: [Errno 0] Error" actually mean, and how do I list my directory contents?
msg303979 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-10-09 15:52
"OSError: [Errno 0] Error" typically means that OpenSSL failed due to a Windows error, but we assumed it failed due to a POSIX error and so read the wrong error number (errno instead of GetLastError()).

It's worth testing with Python 3.7.0a1, since we made some changes to OpenSSL integration (as well as updating it to a significantly newer version) that may impact this scenario.
msg304166 - (view) Author: Jonathan (jonathan-lp) Date: 2017-10-11 17:32
Just tested this with Python 3.7.0a1. I'm afraid it makes no difference. Exact same error:

*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Accepted data connection\n'
*resp* '150 Accepted data connection'
Traceback (most recent call last):
  File "c:\backup_script.py", line 385, in run_ftps
    ftps.dir()
  File "c:\Python37\lib\ftplib.py", line 575, in dir
    self.retrlines(cmd, func)
  File "c:\Python37\lib\ftplib.py", line 485, in retrlines
    conn.unwrap()
  File "c:\Python37\lib\ssl.py", line 1059, in unwrap
    s = self._sslobj.unwrap()
  File "c:\Python37\lib\ssl.py", line 706, in unwrap
    return self._sslobj.shutdown()
OSError: [Errno 0] Error
msg312889 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-02-26 08:31
Jonathan, are you able to provide a reproducer for this bug? I fear we cannot move forward without a way to test and reproduce the issue.
msg312988 - (view) Author: Jonathan (jonathan-lp) Date: 2018-02-27 09:34
Hi Christian - Can you not reproduce it from the code included in my original report? I think that's pretty much all that should be needed, as it's all I was using when I discovered the issue.

I figure the hard part will be finding a FTP_TLS server to test with.
msg315840 - (view) Author: Matthieu Pepin (Matthieu Pepin) Date: 2018-04-27 13:13
I can confirm. I'm having the exact same issue in Python 3.6.5.
msg315875 - (view) Author: Dong-hee Na (corona10) * Date: 2018-04-29 01:22
I can not reproduce this issue on Python 3.8.0a0 with OpenSSL 0.9.8zh 14 Jan 2016 and OpenSSL 1.0.2o  27 Mar 2018


import ssl
import ftplib

print(ssl.OPENSSL_VERSION)

ftps = ftplib.FTP_TLS('test.rebex.net', timeout=2)
ftps.ssl_version = ssl.PROTOCOL_TLS
ftps.login('demo','password')

ftps.prot_p()
ftps.set_pasv(True)
ftps.dir()
ftps.nlst()
ftps.retrlines('LIST')
msg316537 - (view) Author: Grégoire Chauvet (Grégoire Chauvet) Date: 2018-05-14 16:39
I want to confirm that I have this exact same issue as described.

To add some information, it occurs on:
Python 3.6.3 on Windows
Python 3.6.5 on Debian, with OpenSSL 1.0.1t  3 May 2016
Python 3.5.3 on Debian, with OpenSSL 1.1.0f  25 May 2017
Python 3.6.5 on Fedora, with OpenSSL 1.1.0h-fips  27 Mar 2018
Python 3.8.0a0 on Fedora, with OpenSSL 1.1.0h-fips  27 Mar 2018

The server is FileZilla server 0.9.60 beta (this is the last version available as of today).

Depending on the version of python or OpenSSL, the error is :
  [...]
  File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

Or:
  [...]
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:833)


Here is my code:
# I use a workaround to connect with implicit TLS as described here:
# https://stackoverflow.com/a/36049814/3859915
# But from what I can see, it is not related to this.

import ssl
import ftplib

class ImplicitFTP_TLS(ftplib.FTP_TLS):
    """FTP_TLS subclass that automatically wraps sockets in SSL to support implicit FTPS."""
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._sock = None

    @property
    def sock(self):
        """Return the socket."""
        return self._sock

    @sock.setter
    def sock(self, value):
        """When modifying the socket, ensure that it is ssl wrapped."""
        if value is not None and not isinstance(value, ssl.SSLSocket):
            value = self.context.wrap_socket(value)
        self._sock = value

ftps = ImplicitFTP_TLS()
ftps.set_debuglevel(2)
ftps.connect(host="hostname", port=990)
ftps.login(user="user", passwd="password")
ftps.prot_p()
ftps.nlst()

I can connect, create directories, retrieve modification date, but I cannot list files nor download them.
History
Date User Action Args
2018-05-14 16:39:53Grégoire Chauvetsetnosy: + Grégoire Chauvet
messages: + msg316537
2018-04-29 01:22:21corona10setnosy: + corona10
messages: + msg315875
2018-04-27 13:13:17Matthieu Pepinsetnosy: + Matthieu Pepin
messages: + msg315840
2018-02-27 09:34:12jonathan-lpsetstatus: pending -> open

messages: + msg312988
2018-02-26 08:31:47christian.heimessetstatus: open -> pending
assignee: christian.heimes ->
messages: + msg312889

versions: + Python 3.8
2017-10-13 17:50:31terry.reedysettitle: FTP_TLS errors when -> FTP_TLS errors when use certain subcommands
stage: test needed
type: behavior
versions: + Python 3.7
2017-10-11 17:32:01jonathan-lpsetmessages: + msg304166
2017-10-09 15:52:41steve.dowersetnosy: + christian.heimes
messages: + msg303979

assignee: christian.heimes
components: + SSL
2017-10-08 17:12:30ned.deilysetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2017-10-08 17:08:41jonathan-lpcreate