On Windows 8.1 x64 with Python 3.5.1 I was able to reproduce the issue by attempting to load a file at "C:\Users\غازي\AppData\Local\Temp\_غازي_70e5wbxo\cacert.pem".

    locale.getdefaultlocale()
    > ('en_US', 'cp1252')

    locale.getpreferredencoding()
    > 'cp1252'

    sys.getfilesystemencoding()
    > 'mbcs'
    
    sys.getdefaultencoding()
    > 'utf-8'

    c = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)

    c.load_verify_locations(cafile=r"C:\Users\غازي\AppData\Local\Temp\_غازي_70e5wbxo\cacert.pem")
    > TypeError: cafile should be a valid filesystem path

    c.load_verify_locations(cafile=r"C:\Users\غازي\AppData\Local\Temp\_غازي_70e5wbxo\cacert.pem".encode(sys.getfilesystemencoding()))
    > UnicodeEncodeError: 'mbcs' codec can't encode characters in positions 0--1: invalid character

    c.load_verify_locations(cafile=r"C:\Users\غازي\AppData\Local\Temp\_غازي_70e5wbxo\cacert.pem".encode('utf-8'))
    > ok

I believe this is a bug, because path suitable for os.path (or pathlib), should be equally suitable for load_verify_locations.

The Python ssl module is a wrapper to the OpenSSL library. For this issue, we are talking about the function SSL_CTX_load_verify_locations():
https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_load_verify_locations.html

OpenSSL expects a byte string for CAfile and CApath. On Windows, it means a string encoded to the ANSI code page.

OpenSSL doesn't seem to support paths not encodable to the ANSI code page on Windows. I suggest you to report the issue to the OpenSSL bug tracker:
https://www.openssl.org/community/#bugs

A workaround is to avoid characters not encodable to the ANSI code page, maybe by using symbolic links?

"OpenSSL doesn't seem to support paths not encodable to the ANSI code page on Windows. I suggest you to report the issue to the OpenSSL bug tracker: (...)"

For example, it wass already requested 6 years ago on the openssl-users@openssl.org mailing list:
http://comments.gmane.org/gmane.comp.encryption.openssl.user/38104

--

Oh, another link is more useful:
https://stackoverflow.com/questions/2401059/openssl-with-unicode-paths

"you will have to manually load the certificate files yourself using standard OS file I/O functions that support Unicode paths, and then parse the raw data and load it into OpenSSL, such as via PEM_read_bio_X509 with sk_X509_NAME_push, PEM_read_bio_PrivateKey/d2i_PrivateKey_bio with SSL_CTX_use_PrivateKey, d2i_X509_bio/PEM_read_bio_X509 with SSL_CTX_use_certificate, etc."

Viktor, I also came across this thread but it is rather old (we're using OpenSSL 1.0.2h). And it would only explain if _neither_ of my methods had worked.
But as you can see, the last one (passing UTF-8 encoded bytes) works.

I checked the source code of OpenSSL, specifically the `bss_file.c:file_fopen` function (https://github.com/openssl/openssl/blob/OpenSSL_1_0_2h/crypto/bio/bss_file.c#L118-L167).

As you can see it support UTF-8 encoded strings under Windows. Python must follow up.

Is this still an issue?

> Is this still an issue?

I'm not 100% sure that SSL_CTX_load_verify_locations() accepts a path encoded to UTF-8 on Windows.


To be 100% sure, it's "simple": try a filename not encoded to the ANSI code page on Windows.

The best would be to have an unit test for that. You use use test.support.TESTFN_UNENCODABLE for example.

TESTFN_UNENCODABLE will be invalid utf8 now, as the name is chosen by attempting to encode a list of names and using the first one to fail.

No code pages have emoji in them AFAIK, so a test with one of those would do.

ISTR looking at this function though and finding that OpenSSL will decide utf8 if that's what is passed, in which case we're fine now. Or maybe I'm thinking of Tcl...

Steve Dower added the comment:
> TESTFN_UNENCODABLE will be invalid utf8 now, as the name is chosen by attempting to encode a list of names and using the first one to fail.

Oh, maybe we should reject filenames not encodable to utf8? Filenames
containg surrogate characters.

I haven't tracked it down in 1.1, but in 1.0.2 OpenSSL handles ASCII, UTF-8 and mbcs/ANSI paths explicitly: https://github.com/openssl/openssl/blob/OpenSSL_1_0_2-stable/crypto/bio/bss_file.c#L138

So for 3.6 and later, if we're encoding the paths with fsencode(), it'll be fine, but we could also use utf-8 unconditionally.

Doing a search of the codebase though, there's only the one place that does this and everywhere else just uses fopen() without attempting to decode. I don't think we're exposing many of those publicly though.

3.6 uses PyUnicode_FSConverter() to convert the cafile and capath arguments, then passes PyBytes_AS_STRING() to OpenSSL. What needs to change to support non-ASCII chars on Windows?

Christian,

If you have windows under your hand and can try an alike path, you should see the problem right away if it's still there.

I think the original problem was unnecessary PyUnicode_FSConverter: it failed to encode string into mbcs, while OpenSSL did not have any problems understanding a UTF-8 on Windows.

Ilya,

The exception "TypeError: cafile should be a valid filesystem path" is raised by Python, not by OpenSSL. On Python 3.5, PyUnicode_FSConverter() uses MBCS, which is CP-1552 on your system. CP-1552 cannot convert Arabic character set and fails to encode your user name to bytes. This issue has been addressed by PEP 529 in Python 3.6. Starting with Python 3.6, the file system encoding is UTF-8, see https://www.python.org/dev/peps/pep-0529/

3.5 is in security fix-only mode. You either have to keep encoding the path as UTF-8 explicitly or update to Python 3.6. I'm closing this issue was WONTFIX 3.5 and FIXED >= 3.6.

> I'm closing this issue was WONTFIX 3.5

If I understood correctly, it's possible to work around the issue by encoding the filename manually to utf-8.

ssl_function(filename.encode('utf-8'))

> On Python 3.5, PyUnicode_FSConverter() uses MBCS, which is CP-1552 on your system.

Will the behavior of Python 3.6 be different? Could you point me to relevant notes or code?

> If I understood correctly, it's possible to work around the issue by encoding the filename manually to utf-8.

That's correct. I'm not sure that this is true for all versions though.

>> On Python 3.5, PyUnicode_FSConverter() uses MBCS, which is CP-1552 on your system.
>
> Will the behavior of Python 3.6 be different? Could you point me to relevant notes or code?

See the PEP 529.