Please by default enforce ssl certificate checking in modules that can have user's which *depend* on the security of the ssl connection.

See https://bugs.launchpad.net/ubuntu/+source/offlineimap/+bug/675120
http://bugs.python.org/issue10274
http://bugs.python.org/issue1589

and http://seclists.org/oss-sec/2010/q4/33

So I will name the following modules(as starting point):
1. httplib http://docs.python.org/library/httplib.html
2. urllib http://docs.python.org/library/urllib.html
3. urllib2 http://docs.python.org/library/urllib2.html
4. imaplib http://docs.python.org/library/imaplib.html

http.client and urllib.request are already done (*), and imaplib is covered in issue10274.

(*) http://docs.python.org/dev/library/http.client.html
http://docs.python.org/dev/library/urllib.request.html

Do you have other modules in mind or should I close this issue?

Sorry, I don't remember seeing the change-set /commit showing that is now on by default. (for those modules).

Are you referring to
http://code.python.org/hg/branches/py3k/rev/86f97255bfc8

where there is now 
"
    2.29 +   .. warning::
    2.30 +      If neither *cafile* nor *capath* is specified, an HTTPS request
    2.31 +      will not do any verification of the server's certificate.
"
This doesnt' by default check the certificate does it ? 
IMHO it should use a sane system capath by default and do the checking.

> Are you referring to
> http://code.python.org/hg/branches/py3k/rev/86f97255bfc8
> 
> where there is now 
> "
>     2.29 +   .. warning::
>     2.30 +      If neither *cafile* nor *capath* is specified, an HTTPS request
>     2.31 +      will not do any verification of the server's certificate.
> "
> This doesnt' by default check the certificate does it ? 
> IMHO it should use a sane system capath by default and do the checking.

I'm afraid Python has no business validating and distributing CA
certificates. You have to use what is provided by your system libraries,
if possible, and otherwise bundle your own.

Well, what are the usual paths for windows and linux?
Just try those(by default) and if this fails (no ca's paths work) then
raise an exception and have a parameter to disable this behaviour.

> Well, what are the usual paths for windows and linux?
> Just try those(by default) and if this fails (no ca's paths work) then
> raise an exception and have a parameter to disable this behaviour.

That's a possible resolution, but it needs a patch.

Actually, it seems OpenSSL is often built with the paths to system certificates, which can be enabled by an undocumented function. See issue10443.

Interesting but you may want to ask the openssl developers about this first.

db: there *is* no "sane system ca path", or Python would have done that long ago. On Windows, it may be possible to drop OpenSSL, and use the system certificate store. However, that would be a major rewrite, and it may not be possible to provide the entire API anymore.

The best that could be done is to provide a configuration option (e.g. global variable) that should be treated as a default value, and then leave it to people distributing Python to fill out this variable in a sensible way.

> The best that could be done is to provide a configuration option (e.g.
> global variable) that should be treated as a default value, and then
> leave it to people distributing Python to fill out this variable in a
> sensible way.

Actually, OpenSSL already does a similar thing (see issue10443).

>> The best that could be done is to provide a configuration option (e.g.
>> global variable) that should be treated as a default value, and then
>> leave it to people distributing Python to fill out this variable in a
>> sensible way.
> 
> Actually, OpenSSL already does a similar thing (see issue10443).

This may not be satisfying to users. For example, our Windows
distribution doesn't ship with any certicates (AFAIK); I have no
clue where exactly OpenSSL would be looking for them, either.
People worried about this problem probably would want a way to
fill the list of trusted CA certificates.

Regards,
Martin

On 19 November 2010 03:18, Martin v. Löwis <report@bugs.python.org> wrote:
>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
>>> The best that could be done is to provide a configuration option (e.g.
>>> global variable) that should be treated as a default value, and then
>>> leave it to people distributing Python to fill out this variable in a
>>> sensible way.
>>
>> Actually, OpenSSL already does a similar thing (see issue10443).
>
> This may not be satisfying to users. For example, our Windows
> distribution doesn't ship with any certicates (AFAIK); I have no
> clue where exactly OpenSSL would be looking for them, either.
> People worried about this problem probably would want a way to
> fill the list of trusted CA certificates.
>

Martin does it matter?
To be honest I don't know about that many client side python windows
applications for which this is a problem for. Maybe I am mistaken. If
this is the case, then how do these projects work at the moment? (or
do they just not care about this...) . However, they could bundle
their own certificates, so I don't see this as an issue.

However, you seem confused here:
" I have no
> clue where exactly OpenSSL would be looking for them, either.
> People worried about this problem probably would want a way to
> fill the list of trusted CA certificates."

Erh, those people can already do this, but the problem is by default
none are selected.
IMHO something is probably better than nothing in this case(by default).

> > This may not be satisfying to users. For example, our Windows
> > distribution doesn't ship with any certicates (AFAIK); I have no
> > clue where exactly OpenSSL would be looking for them, either.
> > People worried about this problem probably would want a way to
> > fill the list of trusted CA certificates.

Right, this is just a helper in case OpenSSL is configured correctly by
the OS vendor (the OpenSSL packaged by Linux distros usually is).

> Erh, those people can already do this, but the problem is by default
> none are selected.
> IMHO something is probably better than nothing in this case(by default).

We can't change anything *by default* since it would break
compatibility. We can just provide helpers and arguments to make it easy
to switch to a more "secure" behaviour (for some meaning of secure).

On 19 November 2010 03:48, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
>> > This may not be satisfying to users. For example, our Windows
>> > distribution doesn't ship with any certicates (AFAIK); I have no
>> > clue where exactly OpenSSL would be looking for them, either.
>> > People worried about this problem probably would want a way to
>> > fill the list of trusted CA certificates.
>
> Right, this is just a helper in case OpenSSL is configured correctly by
> the OS vendor (the OpenSSL packaged by Linux distros usually is).
>
>> Erh, those people can already do this, but the problem is by default
>> none are selected.
>> IMHO something is probably better than nothing in this case(by default).
>
> We can't change anything *by default* since it would break
> compatibility. We can just provide helpers and arguments to make it easy
> to switch to a more "secure" behaviour (for some meaning of secure).

what about an environmental setting that can be used to enforce
checking (or the like) ?

>> This may not be satisfying to users. For example, our Windows
>> distribution doesn't ship with any certicates (AFAIK); I have no
>> clue where exactly OpenSSL would be looking for them, either.
>> People worried about this problem probably would want a way to
>> fill the list of trusted CA certificates.
>>
> 
> Martin does it matter?
> To be honest I don't know about that many client side python windows
> applications for which this is a problem for. Maybe I am mistaken.

I can't understand why you are saying that. The very same issues
that people perceive as problems on Unix ("users can be victim
to man in the middle attack") also exist on Windows. If you run
a Python script that does https on Windows, you can *also* be
MITM-victim (as likely as you can on Unix, that is).

Or are you suggesting that Python Windows applications don't use SSL?

> If
> this is the case, then how do these projects work at the moment? (or
> do they just not care about this...) .

"The projects" may be scripts that somebody developed that never get
released. But yes, most people ignore/accept the problem (often as
gruntingly as the Unix users).

> However, they could bundle
> their own certificates, so I don't see this as an issue.

Who is "they"? Most people get their Python binaries from python.org,
and they don't build "applications" from it, but run "scripts".

> However, you seem confused here:
> " I have no
>> clue where exactly OpenSSL would be looking for them, either.
>> People worried about this problem probably would want a way to
>> fill the list of trusted CA certificates."
> 
> Erh, those people can already do this, but the problem is by default
> none are selected.

You misunderstood. I was not proposing that scripts provide a CA
list, but that users might deploy a CA list into their Python
installation, which is then picked up in the same way as you are asking
for on Ubuntu.

On 19 November 2010 04:40, Martin v. Löwis <report@bugs.python.org> wrote:
>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
>>> This may not be satisfying to users. For example, our Windows
>>> distribution doesn't ship with any certicates (AFAIK); I have no
>>> clue where exactly OpenSSL would be looking for them, either.
>>> People worried about this problem probably would want a way to
>>> fill the list of trusted CA certificates.
>>>
>>
>> Martin does it matter?
>> To be honest I don't know about that many client side python windows
>> applications for which this is a problem for. Maybe I am mistaken.
>
> I can't understand why you are saying that. The very same issues
> that people perceive as problems on Unix ("users can be victim
> to man in the middle attack") also exist on Windows. If you run
> a Python script that does https on Windows, you can *also* be
> MITM-victim (as likely as you can on Unix, that is).
>
> Or are you suggesting that Python Windows applications don't use SSL?
>
>> If
>> this is the case, then how do these projects work at the moment? (or
>> do they just not care about this...) .
>
> "The projects" may be scripts that somebody developed that never get
> released. But yes, most people ignore/accept the problem (often as
> gruntingly as the Unix users).
>
>> However, they could bundle
>> their own certificates, so I don't see this as an issue.
>
> Who is "they"? Most people get their Python binaries from python.org,
> and they don't build "applications" from it, but run "scripts".
>
>> However, you seem confused here:
>> " I have no
>>> clue where exactly OpenSSL would be looking for them, either.
>>> People worried about this problem probably would want a way to
>>> fill the list of trusted CA certificates."
>>
>> Erh, those people can already do this, but the problem is by default
>> none are selected.
>
> You misunderstood. I was not proposing that scripts provide a CA
> list, but that users might deploy a CA list into their Python
> installation, which is then picked up in the same way as you are asking
> for on Ubuntu.

No I did not misunderstand at all.
I am pushing for safer defaults or a way to enable safe defaults.
Having to tamper with my python path and point at a modified version
of the ssl module doesn't sound like fun.

OH windows users those guys. Well if they don't have any certificates
at the moment and they don't know this, perhaps some one should tell
them?
I don't know I am not a windows python user.

Martin,

> You misunderstood. I was not proposing that scripts provide a CA
> list, but that users might deploy a CA list into their Python
> installation, which is then picked up in the same way as you are asking
> for on Ubuntu.

Could you elaborate on what kind of scheme you are proposing?

It should be noted that the default OpenSSL paths can be modified at runtime using environment variables SSL_CERT_FILE and SSL_CERT_DIR. Not sure we should document this, though.

On 21 November 2010 09:27, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Martin,
>
>> You misunderstood. I was not proposing that scripts provide a CA
>> list, but that users might deploy a CA list into their Python
>> installation, which is then picked up in the same way as you are asking
>> for on Ubuntu.
>
> Could you elaborate on what kind of scheme you are proposing?
>
> It should be noted that the default OpenSSL paths can be modified at runtime using environment variables SSL_CERT_FILE and SSL_CERT_DIR. Not sure we should document this, though.

Sorry is this question aimed at me?
No I was saying that if we can't move to a sane default then an
environmental setting or other configuration maybe nice to have to
enforce certificate checking etc.
It had nothing to do with those variables, but perhaps we would use them?

Antoine,

I was proposing that the Windows installation should have a folder where OpenSSL looks for certificates, and that users put trusted certificates into that folder. These should be the ones that set_default_verify_paths will use. Unfortunately, I don't know where that directory might be in the windows builds we provide. So I think we should find out.

Back to the main issue here :

So for python3 is it possible to make attempting to use capath(some
common ones OR the openssl location capath if this is ok for use) the
default(with failure to find a valid capath result in an exception
being raised) ?

Obviously if cafile or capath is provided by the caller then --> do
not follow this behaviour.

> So for python3 is it possible to make attempting to use capath(some
> common ones OR the openssl location capath if this is ok for use) the
> default(with failure to find a valid capath result in an exception
> being raised) ?

The default? That would be an incompatible change, and cause many
complaints. So I'm very skeptical that this can be done.

Having applications/scripts explicitly opt-in to a default CA
certificate list would be an option (then making those applications
break in installations where the default CA list is empty).

On 21 November 2010 20:50, Martin v. Löwis <report@bugs.python.org> wrote:
>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
>> So for python3 is it possible to make attempting to use capath(some
>> common ones OR the openssl location capath if this is ok for use) the
>> default(with failure to find a valid capath result in an exception
>> being raised) ?
>
> The default? That would be an incompatible change, and cause many
> complaints. So I'm very skeptical that this can be done.
>
> Having applications/scripts explicitly opt-in to a default CA
> certificate list would be an option (then making those applications
> break in installations where the default CA list is empty).

"Errors should never pass silently."
IMHO it is an error not to check by default.
No it wouldn't break anything that shouldn't break.
Users can then pass in None for the capath (as an example).

> > Having applications/scripts explicitly opt-in to a default CA
> > certificate list would be an option (then making those applications
> > break in installations where the default CA list is empty).
> 
> "Errors should never pass silently."
> IMHO it is an error not to check by default.
> No it wouldn't break anything that shouldn't break.
> Users can then pass in None for the capath (as an example).

Well, can you stop insisting? It is probably the third time that we
explain you we can't break compatibility on this.

(also, many SSL sites are not covered by "default" CA certificates
shipped by most vendors, e.g. self-signed certificates or certificates
signed by CAcert)

Fine.
So if not in the ssl module what about for urllib etc.?

> Fine.
> So if not in the ssl module what about for urllib etc.?

The same answer really... We can't change default settings without
breaking compatibility.

So please close this bug.
Apparently making things secure by default is to much to ask.

Here is a patch that changes SSLContext into a context manager, allowing to write code like

    with ssl.SSLContext(ssl.PROTOCOL_SSLv23):
        ssl.SSLContext.current().set_default_verify_path()
        res = urllib.request.urlopen("https://www.microsoft.com")

If people like this approach, feel free to adjust it; I likely won't have much time to work on it further.

Thank you @loewis. However, I don't see where set_default_verify_path - is defined in the patch you have provided.

It would also be nice to do something like this:
import ssl
...
ssl._FORCE_VERIFICATION = True

and even better would be to determine the CA path as @pitrou was suggesting and incorporate this into the ssl module somehow.

> Thank you @loewis. However, I don't see where 
> set_default_verify_path - is defined in the patch you have provided.

It's not defined in the patch, as it is already committed to Python.

and what does it do ?

We tend to spend some time on documentation: http://docs.python.org/dev/library/ssl#ssl.SSLContext.set_default_verify_paths

Cool yeah. The documentation is good I asked the question because I
wasn't sure if it was in a pending patch elsewhere in the bug tracker
or was accepted. I guess I should have googled for it.

Thank you.

Hello,


Your patch about SSLContext is great! But what can we do about python 2.x?

I understand that we have to keep backward compatibility but something has to be done for improving current situation, even Paypal API (*) recommends using urllib.urlopen() to check sensitive things on HTTPS :(

If backporting SSLContext into python 2.x is not an option, what do you think about a DeprecationWarning at runtime?

Thanks,

(*) https://www.x.com/devzone/articles/getting-started-paypal-django
    And even the popular Django paypal module uses urllib2 
    https://github.com/johnboxall/django-paypal

> Your patch about SSLContext is great! But what can we do about python 2.x?

Adding new features to Python 2.7 is certainly not an option.

> what do you think about a DeprecationWarning at runtime?

What API exactly should this deprecate?

Martin v. Löwis <report@bugs.python.org> writes:
>> what do you think about a DeprecationWarning at runtime?
>
> What API exactly should this deprecate?

Ooops, lapsus. I was thinking about a RuntimeWarning raised on HTTPS
request (in httplib.HTTPSConnection.connect for instance).

I would like to reject this approach. I am still adding Christian to the CC list, in case he has something to say about it :)

I agree with Antoine. I'm not sure what is going to happen if you use a single SSLContext for unrelated services and different hosts -- not to mention SNI.

A while ago I tried a similar patch but I was stopped by the fact that OpenSSL doesn't provide an API to clone / copy a SSLContext object. Perhaps we have to introduce a factory that creates SSLContext object from a global configuration?