urllib currently blindly accepts bad certificates when passed an https address. This behavior, clearly not desirable for many users, is also not documented. I propose one of two changes:

1) add mechanisms for enforcing correct behavior to urllib, or
2) change the documentation for that module to include something akin to the following warning:

"Warning: urllib does not perform certificate checks if passed an HTTPS url! This permits remote machines to masquerade as your intended destination."

A big warning is now present (*) in the urllib and httplib documentation pages. Also, once issue1589 is fixed, we can go forward and make {http.client,urllib.request} check hostname and cert if the user gives the location of a bunch of CA certs.

(*) see e.g. http://docs.python.org/dev/library/urllib.request.html

Here is the API addition I would suggest for the http.client module:

Add two new keyword arguments `context` and `check_hostname` to HTTPSConnection; `context` would allow to pass a SSLContext instance for certificate checking and other options (default None, meaning no checking); `check_hostname` would specify whether to check the hostname against the URL (default to check only if context is present and context.verify_mode != CERT_NONE).

Here is the API addition I would suggest for the urllib.request module:

- Add constructor arguments `context` and `check_hostname` to HTTPSHandler. They will be passed to the underlying HTTPSConnection.

- Add `ssl_ca_file` and `ssl_ca_path` arguments to the high-level function urlopen(); if at least one of them is present, a custom opener with a custom HTTPSHandler will be created, mandating the checking of server certificates

Here is a preliminary patch for http.client. I think it would be good to have local tests using a custom HTTPS server, too.

Here is another patch for http.client containing more tests, including with a mismatching cert. Comments welcome.

Any chance on folding the HTTPSServer class into http.server?

Geremy Condra

> Any chance on folding the HTTPSServer class into http.server?

Its API and implementation would first have to be cleaned up.
I'd prefer if it were the subject of a separate issue.

Here is a patch which also adds 'cafile' and 'capath' keyword arguments to urlopen().

Here is a new patch with doc updates for urllib.request.

This patch should fix the test hanging issues witnessed on some machines.

Yes, it does solve the problem of httplib and urllib2_localnet tests which
were hanging with the earlier patch on certain machines..

Patch committed in r85408. I believe this fixes, at last, the whole issue people were complaining about.

New changeset 1882157b298a by Benjamin Peterson in branch '2.7':
allow passing cert/ssl information to urllib2.urlopen and httplib.HTTPSConnection
https://hg.python.org/cpython/rev/1882157b298a