classification
Title: ftplib uses latin-1 as default encoding
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: SebastianGPedersen, benjamin.peterson, corona10, eric.smith, ezio.melotti, giampaolo.rodola, inada.naoki, lemburg, vstinner
Priority: normal Keywords: patch

Created on 2020-01-18 09:57 by SebastianGPedersen, last changed 2020-03-22 15:56 by SebastianGPedersen.

Pull Requests
URL Status Linked Edit
PR 18048 open SebastianGPedersen, 2020-01-18 10:27
Messages (13)
msg360549 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-01-23 10:44
What's the reason behind this change? I love UTF-8 and all, but is there some standard (de facto or de jure) that discusses this?

What does this change fix? What's the implication to existing clients?

I'm not opposed to the change per se, but I'd like to understand the reasoning.
msg360610 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2020-01-24 11:12
Can we have some deprecation period?

def __init__(self, ..., encoding=None):
    ...
    if encoding is not None:
        self.encoding = encoding
    else:
        warnings.warn("The default encoding of the FTP class will be changed from 'latin1' to 'utf-8' in 3.11. Pass the encoding explicitly for now.",
            DeprecationWarning)
msg360633 - (view) Author: Dong-hee Na (corona10) * (Python triager) Date: 2020-01-24 17:23
I agree with inada.naoki
msg360683 - (view) Author: Sebastian G Pedersen (SebastianGPedersen) * Date: 2020-01-25 14:03
Thank you for the feedback.

I will elaborate a little bit on the reasons and thoughts behind the pull request:
Since RFC 2640, the industry standard within FTP Clients is UTF-8 (see e.g. FileZilla here: https://wiki.filezilla-project.org/Character_Encoding, or WinSCP here: https://winscp.net/eng/docs/faq_utf8).

Given this, I believe the majority of the users that have not investigated the code wrongly assumes UTF-8 encoding for ftplib as well (as it is now).

I am new to contributing, and not sure, how much deprecation warnings are used (I simply followed the previous encoding change on ftplib), so I will change it based on the feedback. However, shouldn't it be a FutureWarning, so it will be reported by default at initialisation?
msg360696 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2020-01-25 16:38
It's been a long time since I implemented UTF-8 support in pyftpdlib, but long story short is that:
- most recent servers are supposed to use UTF-8 by default
- such servers must include "UTF-8" in the FEAT command response
- some servers may enable UTF-8 only if client explicitly sends "OPTS UTF-8 ON" first, but that is based on an draft RFC. Server implementors usually treat this command as a no-op and simply assume UTF-8 as the default.

With that said, I am -1 about implementing logic based on FEAT/OPTS: that should be done before login, and at that point some servers may erroneously reject any command other than USER, PASS and ACCT. 

Personally I think it makes more sense to just use UTF-8 without going through a deprecation period, document *encoding* attribute and mention that in order to deal with servers not supporting UTF8 you can pre-emptively check FEAT response and set ASCII encoding. But I am not a unicode expert, so I would like to hear some other opinion re. the implications of going from latin-1 to utf8 in terms of potential code breakage.
msg360738 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2020-01-27 05:39
I'm not FTP user so I don't have strong opinion.
If it is too late to change already, change it in 3.9 might be OK.

> However, shouldn't it be a FutureWarning, so it will be reported by default at initialisation?

If it is warning for end users, it should be FutureWarning.
If the warning is for developers, it should be DeprecationWarning.
Warning for API changes is DeprecationWarning in general.
msg360740 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-27 07:59
I'm in favor of changing the default encoding to UTF-8, but it requires good documentation, especially to provide a solution working on Python 3.8 and 3.9 to change the encoding (see below).

--

The encoding is used to encode commands with the FTP server and decode the server replies. I expect that most replies are basically letters, digits and spaces. I guess that the most problematic commands are:

* send user and password
* decode filenames of LIST command reply
* encode filename in STOR command

I expect that the original FTP protocol doesn't specify any encoding and so that FTP server implementations took some freedom. I would not be surprised to use ANSI code pages used on servers running on Windows.

Currently, encoding is a class attribute: it's not convenient to override it depending on the host. I would prefer to have a new parameter for the constructor.

Giampaolo:
> some servers may enable UTF-8 only if client explicitly sends "OPTS UTF-8 ON" first, but that is based on an draft RFC. Server implementors usually treat this command as a no-op and simply assume UTF-8 as the default.
> With that said, I am -1 about implementing logic based on FEAT/OPTS: that should be done before login, and at that point some servers may erroneously reject any command other than USER, PASS and ACCT. 

Oh. In this case, always send "OPTS UTF-8 ON" just after the socket is connected sounds like a bad idea.


Sebastian:
> Since RFC 2640, the industry standard within FTP Clients is UTF-8 (see e.g. FileZilla here: https://wiki.filezilla-project.org/Character_Encoding, or WinSCP here: https://winscp.net/eng/docs/faq_utf8).

"Internationalization of the File Transfer Protocol" was published in 1999. It recommends the UTF-8. Following a RFC recommendation is good argument to change the default encoding to UTF-8.
https://tools.ietf.org/html/rfc2640


Giampaolo:
> Personally I think it makes more sense to just use UTF-8 without going through a deprecation period

I concur. Deprecation is usually used for features which are going to be removed (module, function or function parameter). Here it's just about a default parameter value. I expect to have encoding="utf-8" default in the constructor.

The annoying part is that Python 3.8 only has a class attribute. The simplest option seems to be creating a FTP object, modify its encoding attribute and *then* logs in. Another options is to subclass the FTP class. IMO the worst is to modify ftplib.FTP.encoding attribute (monkey patch the module).

I expect that most users use username, password and filenames encodable to ASCII and so will not notify the change to UTF-8. We can document a solution working on all Python versions to use different encoding name.
msg360741 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-27 08:01
Before changing the default, I would prefer to see a PR adding an encoding parameter to the FTP constructor.
msg364237 - (view) Author: Sebastian G Pedersen (SebastianGPedersen) * Date: 2020-03-15 14:54
Thanks again for the engagement.
I am also in favor of adding the encoding to the constructor. However, following Giampaolo's comment, that utf-8 is standard and has been for a long time (and the RFC dating back to year 1999), I am also in favor of changing the default encoding in the same PR. As mentioned in the previous comments, most users use ASCII and will not be affected by this anyway - which is probably also why this issue hasn't been raised before.
Please let me know your thoughts on this.
msg364702 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-20 18:38
I'm now fine with changing the default. But I would still prefer to have an encoding parameter in the constructor. Making these two changes at once now makes sense to me.
msg364704 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-03-20 18:58
I agree with Victor.
msg364705 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2020-03-20 19:01
+1 from me as well. @SebastianGPedersen could you update the PR (constructor + doc changes)?
msg364806 - (view) Author: Sebastian G Pedersen (SebastianGPedersen) * Date: 2020-03-22 15:56
Yes, I will update the PR before the end of next week.
History
Date User Action Args
2020-03-22 15:56:30SebastianGPedersensetmessages: + msg364806
2020-03-20 19:01:03giampaolo.rodolasetmessages: + msg364705
2020-03-20 18:58:45eric.smithsetmessages: + msg364704
2020-03-20 18:38:39vstinnersetmessages: + msg364702
2020-03-15 14:54:33SebastianGPedersensetmessages: + msg364237
2020-01-27 08:01:02vstinnersetmessages: + msg360741
2020-01-27 07:59:03vstinnersetnosy: + vstinner
messages: + msg360740
2020-01-27 05:39:03inada.naokisetmessages: + msg360738
2020-01-25 16:38:28giampaolo.rodolasetnosy: + lemburg, benjamin.peterson, ezio.melotti
messages: + msg360696
2020-01-25 14:03:45SebastianGPedersensetmessages: + msg360683
2020-01-24 17:23:17corona10setnosy: + corona10
messages: + msg360633
2020-01-24 11:12:30inada.naokisetnosy: + inada.naoki
messages: + msg360610
2020-01-23 10:44:20eric.smithsetnosy: + eric.smith
messages: + msg360549
2020-01-18 13:07:51SilentGhostsetversions: - Python 3.5, Python 3.6, Python 3.7, Python 3.8
2020-01-18 10:27:29SebastianGPedersensetkeywords: + patch
stage: patch review
pull_requests: + pull_request17443
2020-01-18 10:06:03xtreaksetnosy: + giampaolo.rodola
2020-01-18 09:57:38SebastianGPedersencreate