Proposed patch adds possibility to opt-out certificate verification. 
Disclaimer: it is just proof of concept as the config value is hard-coded.

How it works?
This patch depends on existence of config file which holds information about the protocol settings.

> $ cat cert-verification.conf
[https]  # each protocol affected by cert-verification got its own section
verify=platform_default

Possible values for verify are:
enable - to enable certificate verification
disable - to disable certificate verification
platform_default - to use default (platform-specific) settings

Why platform_default?
This choice is for users who don't care about the security settings so they put the decision into their platform (distro) from which they get python. In rpm we can set package to not replace user edited configs when rpm is updated, so if user change the default value of config the config will remain the same.

Python example:

 >>> import http.client
 >>> cn = http.client.HTTPSConnection('www.google.com')
 >>> cn._context.verify_mode
 0L  # CERT_NONE
 >>> # config changed to verify=enable, still same interpreter
 >>> cn2 = http.client.HTTPSConnection('www.google.com')
 >>> cn2._context.verify_mode
 2L  # CERT_REQUIRED

This is how currently works patch attached, but I guess it would make more sense make this behave consistent within the same interpreter even when config is changed and the change will be propagated in the next interpreter run/service restart.

Also the patch could be changed to instead of being protocol based to be module based, but this would need also patching the affected modules.

I open the RFE mainly to see if there is a will to implement optionable certificate verification in upstream as it is in downstream [citation needed].

I've added some people to nosy list based on https://docs.python.org/devguide/experts.html

1) there is no patch attached
2) certificate verification is optional already, is it not? That is, it can be turned off in your code, it is just on by default.
3) what downstream are you talking about?

Supposing there is sufficient utility here, the level of change proposed would need to go through python-ideas first, IMO.

This issue is related to the PEP 476 which made the SSL certification checks mandatory by default.

The PEP contains a section to explain how to opt-out, but the solution is global:
https://www.python.org/dev/peps/pep-0476/#opting-out

I understand that Robert wants a finer grain.

> [https]  # each protocol affected by cert-verification got its own section

I'm not sure that the configuration should be made on the protocol. We may configure it per Python module (if we choose to accept the enhancement, I'm not convinced that it's good idea). Maybe it's a stupid idea, for example urllib and httplib are both used for HTTPS. But what about xmlrpclib? Should it follow the same policy?

> 2) certificate verification is optional already, is it not? That is, it can be turned off in your code, it is just on by default.

It requires to modify applications. Robert wants something to keep the Python 2.7.8 behaviour on Python 2.7.9 and newer. (Python 2.7.9 made SSL check mandatory by default, or is it only scheduled for Python 2.7.10? I don't remember :-()

> 3) what downstream are you talking about?

Robert and me are working for Red Hat.

1) patch attached, dunno how I missed it, thank you.

3) I work for Red Hat
additional interest for example here http://seclists.org/oss-sec/2015/q1/785

2) It exists but it is not system wide, I would like to provide users option to opt-in or opt-out without interfering with code. There are many of users who rely on python and not all of them are programmers.

> 2) It exists but it is not system wide,

You can hack site.py to disable SSL checks system-wide. It was also discussed to support an optional "sslcustomize" module, but the idea was rejected if I remember correctly. Anyway, did you read the discussion on the PEP 476? Options to disable SSL checks have been discussed there.

Examples:
https://mail.python.org/pipermail/python-dev/2014-August/136034.html
https://mail.python.org/pipermail/python-dev/2014-September/136102.html

(capturing these details here for now, we should at least have a python-dev discussion before going ahead with any changes in this area)

The additional background here is that we started looking closely at what would be involved in applying PEP 476 to systems where there are 3 parties involved and the aim is to tweak the system Python to require explicitly opting in to certificate verification (at least for the time being):

* CPython upstream verifies certificates by default
* Platform vendor configures Python to use legacy mode by default
* System administrator responsible for the machine can easily opt in to global verification and have that setting persist through Python updates

The opt out solution in PEP 476 works for the "direct consumption of upstream" case, as the system administrator responsible for the machine can opt out globally in sitecustomize.py.

It turns out that solution *doesn't* work at the platform vendor level, as sitecustomize.py is intended for use by the system administrator responsible for the machine - providing it as a platform vendor would be the wrong thing to do from a packaging perspective, as that usage would conflict with the intended use case of local site customisation.

We suspected this would be a problem when we were discussing PEP 476 (hence the discussions of configuration files that Victor linked), but decided to postpone further consideration until after the distros had had a chance to consider the problem in more detail.

Patching httplib to have a different default from CPython upstream isn't desirable, as that would perpetuate the problem that PEP 476 was designed to fix with no clear migration path to more secure defaults as the platform level.

The idea of this patch is to provide a middle ground where a configuration file can be provided with the system python package that preserves the pre-PEP-476 status quo by setting "verify=disable" for https verification. If the configuration file is missing entirely then the upstream default of verifying certificates would still be used.

The "verify=enable" setting would then let system administrators explicitly opt in to certificate verification, while "verify=platform_default" would mean "verify=disable" while the default package configuration still did that, but would potentially switch to meaning "verify=enable" at some point in the future (the exact meaning of the "platform_default" setting would likely be controlled by a distro level patch anyway, so it could potentially be omitted from the upstream change proposal).

From an implementation perspective, the reason the proposal is for a global HTTPS switch rather than a fine-grained per-module setting is back the follow-up proposal would be to write a PEP to backport it to Python 2.7, so something that's closely aligned with the existing opt-out mechanism in PEP 476 is considered desirable.

If you mean hack site.py to be sitecustomize  I don't find it as a sufficient solution because users may use their own sitecustomize and this way we would replace theirs.

Sslcustomize solution could be another option how to handle this but the config idea seems to me much more easier (also from the linux user POV).

( ^ I was replying to Victor)

The other goal worth noting here is that we'd like to facilitate easy system auditing/monitoring such that machines that still have Python certificate verification off by default can easily be flagged by checks in tools like Nagios, as well as being easy to adjust using configuration management and system orchestration tools (Ansible, Salt, Chef, Puppet, etc).

Clarified the issue heading a bit, and cc'ed in the main Debian/Ubuntu folks.

Matthias, Barry - the attached patch here is aimed at making PEP 476 a bit more distro friendly by moving the "opt out" to a configuration file rather than requiring monkeypatching in sitecustomize.

For upstream, the key components of the proposal are to have a simple ini-style config file that makes it possible to toggle the behaviour of the "ssl._create_default_https_context" function:

$ cat cert-verification.conf
[https]
verify=disable

=> ssl._create_default_https_context = ssl._create_unverified_context


$ cat cert-verification.conf
[https]
verify=enable

=> ssl._create_default_https_context = ssl.create_default_context

If the config file is missing entirely, there's no https section in the file, or the "verify" setting is missing, then it would default to verifying HTTPS certificates.

As more protocols were moved over to verifying certificates by default, they could follow the same pattern of having a private helper function in the ssl module that referred to either _create_unverified_context() or create_default_context() based on whether certification verification was enabled or not.

It would also be possible to define a true overall ssl/tls default behaviour using this scheme, but I think that's out of scope for this particular proposal.

I do not understand why the vendors want to re-introduce a security hole.

I understand that it causes issues using legacy software to communicate with sites that don't verify, but I think that the correct solution to this is disabling verification on a per-transaction basis, similar to how wget and curl have command line options for.   For Python I think this would mean an environment variable.  I believe I suggested or supported this before and it was rejected (I don't particularly remember why).

If you want to make it config file driven it ought to be keyed by site, not by protocol, IMO, and that seems like a suspect thing to put in a global configuration file.

Introducing a global config file for Python is a significant architectural change, and merits a careful discussion (and probably a PEP).

I don't think it is particularly useful to have this as a tracker issue at this stage.

Changing the title to be specific to the proposed patch.

+1  for keyed by site

There have been a number of issues over the years for which a configuration file (or files) would have been useful.  I think a discussion over on python-ideas is the right way to move forward on this point.

I'd really rather not add this to Python itself. If downstream wants to patch their Pythons to do it that is their prerogative. There's some legacy at play here of course, however I don't think that Python upstream is the right place to deal with that.

One particular problem with this, is it becomes a lot harder to figure out if accessing a https URL is going to be secured or not since you have to also figure out what additional settings have been put into place. It also feels like a really weird setting. You don't see this kind of thing in any other languages or tool that I'm aware of except for single purpose tools.

I agree with Donald on all points. This shouldn't be done at the language level at all (why should it apply only to Python-written tools?). Having a centralized setting saying "I relinquish security on HTTPS accesses" sounds like a bad idea. And if this is solely for the "support legacy systems" business of some vendors, then it sounds like it may be close to Alex's post here :-)
https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/

It's already possible to disable HTTPS certificate checking by using the right SSLContext options, at least with urllib and http.client.

The discussion isn't on python-ideas yet because I wanted to get a better sense of what might be politically feasible before putting this question to a broader audience. I agree it needs to move there eventually (likely during or after PyCon), and will almost certainly lead to a PEP (3.5b1 is slated for late May, so we have 6-7 weeks to resolve the question in time for that if anything is going to change for 3.5)

To be absolutely clear, nobody is thinking of reintroducing silent security failures anywhere - the ultimate aim of posting this draft patch is to start down the path to defining a new Python 3.5 feature that could then by pitched for a PEP 466 style backport to Python 2.7 to provide a potentially smoother upgrade path from the pre-PEP-476 status quo to the shiny new PEP 476 future for the benefits of folks that take both security concerns and backwards compatibility concerns at least as seriously as python-dev do, but are serving a very different audience and hence may need to make different trade-offs between these considerations.

The "use sitecustomize.py to monkeypatch in the old behaviour" section in PEP 476 was *intended* to provide that upgrade path, but it turned out not to work as well as I hoped it would as it turns out that approach effectively requires forking the standard library to let a vendor manage the migration on behalf of their customers by offering a bridging "opt-in" period. Changing the standard library's behaviour to this degree would be a genuinely drastic option, so I consider it vastly superior to backport a supported behaviour from a later version of Python (along the lines of the network security backports in PEP 466) than it would be to invent something custom that has no upstream support.

This does mean spending more time upfront coming up with a way of designing the feature that the core development community considers to be useful independently of backporting considerations (e.g. bringing the STARTTLS migration into the framework could be useful, as the sad state of email server certificate validity means that even upstream CPython is going to need to leave that off by default for the time being). That additional time investment is likely to be worthwhile when the pay-off is avoiding a long-lived behavioural fork.

As for *why* such an opt-in bridging period might be needed by some organisations, one of the key issues to consider is the likely desire to do a global upgrade to an updated Python version as soon as possible, *without* risking breaking currently "working" services in an end-user visible way, and then handling the security configuration change on a service-by-service basis as a subsequent step, in conjunction with any necessary upgrades to the related security infrastructure.

Splitting the two activities (Python upgrade, service network security upgrade) this way is potentially desirable even if you have control of all of the affected Python applications, but it may be absolutely essential if you're running a proprietary bytecode-only Python application in the system Python, or simply aren't authorised to make application level changes to an affected service.

The rationale for introducing a configuration or marker file for this is to allow the *default* behaviour in the absence of such a file to be the standard PEP 476 behaviour. An opt-in bridging period can then be implemented by publishing a default configuration file that globally opts out, with system administrators selectively opting in.

Eventually the default configuration can potentially be changed or removed such that certificates are verified by default, by which time services that genuinely need to be opted out should already have the appropriate configuration settings set.

As far as Alex's post goes, it's simply wrong, and I wish he had spoken to me about his frustrations with the significant challenges of infrastructure maintenance in large established organisations before posting it. Red Hat's been fighting the battle for better enterprise infrastructure management for 20 years at this point (including in the US public sector: https://www.redhat.com/en/technologies/industries/government), but like almost all institutional reform, it's very slow going.

We offer plenty of options for folks to upgrade faster, and it's much easier for us when they do: http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html

So if you care about getting security enhancements rolled out in a way that means people responsible for infrastructure management in large organisations will actually adopt them, rather than dismissing them out of hand as "too risky", please take a moment to consider that we might have some idea what we're talking about.

On it's own I think this switch is a bad idea because it's too big of a hammer. Someone shouldn't accidentally disable TLS verification in pip for instance because they wanted to disable TLS verification for some random tool that only hit internal TLS but which didn't have it's own off switch written into it. A lot of tools are written in Python and it's hard for a user to really know what the full extent of toggling this switch on their system will be, especially given that they may have no idea which other tools are incidentally written in python (pip is not a good example of this, but there are lots of tools that are written in Python but which the fact they are written in Python isn't important or maybe even obvious).

I think keyed by site is wrong too, again because the scope is wrong. Opting out of security at the Python level filters down into tons of random applications that the end user may or may not be aware is even written in Python.

Part of the benefit of the current "opt out" mechanism is that it feels a little dirty to opt in in that fashion, and it should because globally opting out is breaking the security expectations that any application has now with the latest versions of Python, and adding a "cleaner" way of doing this is essentially giving people a nicer footgun (in the long term).

Now, I recognize that there is legacy systems at play here that are going to be around for a long time and that who this proposal is really being aimed to helping. My question would be, why can't those downstreams simply carry this patch? The attached patch is relatively tiny so it shouldn't be a very large burden, the largest being documenting and making people aware that such a thing exists on that platform. If there's enough downstreams who would reasonably have a reason to apply said patch maybe an addendum (or a new PEP) can be added suggesting that downstreams should apply said patch.

The tl;dr of my opinion is that if it weren't for legacy systems, I don't think anyone would propose this feature, and given the security sensitive nature of it I think it's best to treat this feature as a quirk of those legacy systems rather than a fully supported API of Python.

Le 05/04/2015 12:25, Nick Coghlan a écrit :
> 
> This does mean spending more time upfront coming up with a way of
> designing the feature that the core development community considers to
> be useful independently of backporting considerations (e.g. bringing the
> STARTTLS migration into the framework could be useful, as the sad state
> of email server certificate validity means that even upstream CPython is
> going to need to leave that off by default for the time being).

I'm curious about statistics about e-mail servers, even though unrelated
to this issue.

> Splitting the two activities (Python upgrade, service network
> security
> upgrade) this way is potentially desirable even if you have control of
> all of the affected Python applications, but it may be absolutely
> essential if you're running a proprietary bytecode-only Python
> application in the system Python, or simply aren't authorised to make
> application level changes to an affected service.

True, but this is a repeat of the PEP 476 discussion. Something has
changed in the meantime: PEP 476 was accepted and its code has shipped
in an official release. There hasn't been any major (or even minor) outcry.

Speaking as someone who opposed PEP 476, I now support us moving forward
instead of trying to eschew the PEP's deliberate effects.

By the way, if a vendor wants vendor-specific behaviour, forking the standard library is a normal price to pay.
(in this case, the diff wouldn't be large, and it's made against an extremely stable upstream branch)

FWIW: I just ran into a situation where the new approach resulted
in pip, setuptools and zc.buildout not working anymore.

This was on an AIX system which did come with CA root certificates
at all.

Now, I knew how to fix this, but the solution was not
an obvious one. I had to use truss to figure out where OpenSSL
was looking for certificates and the added the Mozilla cert
bundle from our egenix-pyopenssl package to make things work
again.

This was on a system where Python 2.7.3 had been installed
previously. After the upgrade to Python 2.7.9 nothing worked
anymore.

Again: Please let the users decide what level of security they
want to apply. We can point users to solutions, but in the end
have to respect their own decisions. Note that staying with
Python 2.7.8 is a much worse approach than disabling the checks.

> This was on a system where Python 2.7.3 had been installed
> previously. After the upgrade to Python 2.7.9 nothing worked
> anymore.

Who did the upgrade and with which binaries?
If you're compiling Python from source, especially for an exotic system, well, you're supposed to read the release notes :-)

PEP 476 *has* a mechanism in it that was supposed to deal with this problem, thus leaving *end users* in full control of the decision on when they upgrade their security infrastructure rather than having that decision arbitrarily imposed on them by a vendor or an upstream community project regardless of whether or not it's appropriate for their particular situation. Unfortunately, it turned out I was wrong about the viability of the approach in PEP 476, hence this suggestion to revisit the question.

There is *no* suggestion of changing the default behaviour away from that defined in PEP 476, the part I would like to revisit is merely the section on configurability, where the goal is to be able to deploy "All of PEP 476 *except* the change in default certificate verification behaviour". The approach in the PEP works for folks deploying upstream Python directly, and I *thought* it would work for the redistributor case as well. It's the latter point I was wrong about.

This is a level of consideration of their needs that folks are willing to pay for, but it's also an expensive one to provide, so it doesn't make sense for upstream to provide it for free. Rather, I am asking the upstream development community to work with commercial redistributors to come to an accommodation that actually meets end users upgrade needs, rather than leaving them stuck on a legacy Python version with no viable path forward to more secure infrastructure. (Telling end users "just upgrade anyway" when complex systems and large scale deployments are involved doesn't work - this is why Microsoft ended up having to support Windows XP for 12 years)

I thought proposing a useful new feature for Python 3.5 and then proposing a subsequent backport would be the easiest path forward, but I now suspect a PEP specifically targeting an improved network security transition plan for the benefit of folks managing infrastructure upgrades in the 2.7.x series may be a better option.

> Now, I knew how to fix this, but the solution was not
> an obvious one. I had to use truss to figure out where OpenSSL
> was looking for certificates and the added the Mozilla cert
> bundle from our egenix-pyopenssl package to make things work
> again.

You also could have passed the --cert flag to pip to tel pip specifically where
to look for them (also available via environment variable and config file)
though I'm guessing it wasn't actually pip itself that had a problem because
we ship our own CA file and we don't actually rely on the stdlib to have
validated TLS. Unless you were using an old pip I guess.

> Again: Please let the users decide what level of security they
> want to apply. We can point users to solutions, but in the end
> have to respect their own decisions. Note that staying with
> Python 2.7.8 is a much worse approach than disabling the checks.

Sure, and nobody has ever advocated to make it impossible to disable the TLS
verification. For me it's entirely about the scope of the setting. I don't
think that a Python wide setting is the right scope. That's a knob that has
an extremely large scope of which end users are most likely not going to be
completely aware of the total impact of adjusting that knob. This isn't even
something that they could reasonably audit their system with _today_ and then
say "OK I've looked at everything that uses Python and I'm happy for it not to
verify" because if they every install anything else that uses Python (whether
they know it uses Python or not) they have to re-evaluate that decision they
made all over again, but with no indicator that they need to do that.

On 05.04.2015 18:28, Donald Stufft wrote:
> 
> Donald Stufft added the comment:
> 
>> Now, I knew how to fix this, but the solution was not
>> an obvious one. I had to use truss to figure out where OpenSSL
>> was looking for certificates and the added the Mozilla cert
>> bundle from our egenix-pyopenssl package to make things work
>> again.
> 
> You also could have passed the --cert flag to pip to tel pip specifically where
> to look for them (also available via environment variable and config file)
> though I'm guessing it wasn't actually pip itself that had a problem because
> we ship our own CA file and we don't actually rely on the stdlib to have
> validated TLS. Unless you were using an old pip I guess.

I was working on a Zope installation using zc.buildout, so
basically setuptools, and yes, it was an older version as well.

But this is only an example of an application not working anymore
because the system's OpenSSL could not verify certificates.
In this case, no root CA certs were available. On older systems
with proper root CA certs, it's likely that the newer CA certs
needed to verify the PyPI certificates are not installed...
and yes: those system do exist and are in active use, simply because
they cannot be upgraded for other reasons :-)

>> Again: Please let the users decide what level of security they
>> want to apply. We can point users to solutions, but in the end
>> have to respect their own decisions. Note that staying with
>> Python 2.7.8 is a much worse approach than disabling the checks.
> 
> Sure, and nobody has ever advocated to make it impossible to disable the TLS
> verification. For me it's entirely about the scope of the setting. I don't
> think that a Python wide setting is the right scope. That's a knob that has
> an extremely large scope of which end users are most likely not going to be
> completely aware of the total impact of adjusting that knob. This isn't even
> something that they could reasonably audit their system with _today_ and then
> say "OK I've looked at everything that uses Python and I'm happy for it not to
> verify" because if they every install anything else that uses Python (whether
> they know it uses Python or not) they have to re-evaluate that decision they
> made all over again, but with no indicator that they need to do that.

I'd be fine with having a knob that says: don't check the certificates
but warn me about instances where the certificates are not checked
(using the warning framework).

Le 05/04/2015 21:26, Marc-Andre Lemburg a écrit :
> 
> But this is only an example of an application not working anymore
> because the system's OpenSSL could not verify certificates.
> In this case, no root CA certs were available. On older systems
> with proper root CA certs, it's likely that the newer CA certs
> needed to verify the PyPI certificates are not installed...
> and yes: those system do exist and are in active use, simply because
> they cannot be upgraded for other reasons :-)

Let's sum it up:

- the machine can't be upgraded, but you are upgrading Python by hand
(hand-compiled?)

- OpenSSL is installed but there are no root CA certs (?!)

- the machine probably isn't ever doing a single verified HTTPS access,
for the previous reason, and nobody cares about it

- you want to be able to use unauthenticated HTTPS to download and
install software from the Internet

And, since this is an AIX machine, I'm presuming this isn't a hobbyist's
setup, but an enterprise system with paid-for support and licenses,
right? And you want the python-dev community to care for that broken
situation by bearing the cost of additional maintenance and security
risk in implementing the new configuration options?

On 05.04.2015 21:36, Antoine Pitrou wrote:
> 
> And you want the python-dev community to care for that broken
> situation by bearing the cost of additional maintenance and security
> risk in implementing the new configuration options?

No, I want to be able to easily disable the newly added
checks in 2.7.9+ to get systems such as these behave the
same as with 2.7.8, since without this option, people
using these system are going to be forced to stick with
buggy 2.7.8 systems.

It's rather unusual that a patch level release completely
breaks an existing setup. I understand why this was done,
but in the light of backwards compatibility, it's a huge
issue for people.

PS: Python installations in Zope systems are often custom
installs, not system installs of Python. The AIX system
I'm referencing here still has Python 2.6 as system
Python version.

> No, I want to be able to easily disable the newly added
> checks in 2.7.9+ to get systems such as these behave the
> same as with 2.7.8, since without this option, people
> using these system are going to be forced to stick with
> buggy 2.7.8 systems.

Why is the monkeypatch in sitecustomize.py unacceptable? I understand why it's
unacceptable to Nick and rkuska, they are a vendor and they don't want to write
sitecustomize.py when the machine operator may want to use that file, however
you're the machine operator in this case.

On 05.04.2015 21:48, Donald Stufft wrote:
> 
> Donald Stufft added the comment:
> 
>> No, I want to be able to easily disable the newly added
>> checks in 2.7.9+ to get systems such as these behave the
>> same as with 2.7.8, since without this option, people
>> using these system are going to be forced to stick with
>> buggy 2.7.8 systems.
> 
> Why is the monkeypatch in sitecustomize.py unacceptable? I understand why it's
> unacceptable to Nick and rkuska, they are a vendor and they don't want to write
> sitecustomize.py when the machine operator may want to use that file, however
> you're the machine operator in this case.

I don't consider monkey patching a proper way to configure a Python
installation.

I could simply patch the Python installation directly, but this is just
me. I'm talking about sys admins out there who don't know enough about
Python to be able to patch Python or write a sitecutomize.py which
uses monkey patching to fix the issue.

I also cannot recommend to our customers to monkey patch
Python just to get it running again. This is not what folks
expect from a production quality system :-)

Really these arguments make it sound like 2.7.9 never should have happened.

Given that it did, Nick has not addressed the question of why the vendors maintaining this simple patch (given that it addresses what he sees as their need) is not a viable option.

I do *not* see the proposed patch as an acceptable "feature" for 3.5, and I think I'm far from alone, so I suspect that it is a non-starter for following Nick's proposed "path".

Could there be a related feature that would be both acceptable and worthwhile?  Yes.  But someone will have to figure out what it is and propose it :)

> I don't consider monkey patching a proper way to configure a Python
> installation.

The point is that that TLS validation on/off isn't conceptually a Python level
configuration option, that's going to be a per application configuration
option. The monkeypatching is simply an escape hatch to give people time to
update their applications (or pressure whoever wrote the application) to
support the configuration option that really belongs at the application
level. It *should* feel improper because the entire concept of a Python level
on/off switch isn't proper and making it feel more proper by adding an official
API or config file for doing it is only giving footguns out to people.

On 05.04.2015 22:49, Donald Stufft wrote:
> 
> Donald Stufft added the comment:
> 
>> I don't consider monkey patching a proper way to configure a Python
>> installation.
> 
> The point is that that TLS validation on/off isn't conceptually a Python level
> configuration option, that's going to be a per application configuration
> option. The monkeypatching is simply an escape hatch to give people time to
> update their applications (or pressure whoever wrote the application) to
> support the configuration option that really belongs at the application
> level. It *should* feel improper because the entire concept of a Python level
> on/off switch isn't proper and making it feel more proper by adding an official
> API or config file for doing it is only giving footguns out to people.

People upgrading to a new patch level Python release will *not*
expect or want to have to change their application to adapt to
it. That's simply not within the scope of a patch level release.

Furthermore, old applications such as Zope will (most likely) not
receive such updates.

Please accept that there's a whole universe out there where people
don't continually update their applications, but still want to
benefit from bug fixes to their underlying libs and tools. The
world is full of legacy systems, regardless of whether we like it
or not. There's no good or bad about this. It's just a fact of
life.

What I'm arguing for is a way for admins of such older systems
to be able to receive bug fixes for Python 2.7.x *without*
having to change the applications.

Using an environment setting and adding that to the application's
user account settings is an easy way to resolve the issue in
situations where other options are not feasible or simply not
deemed needed (Zope has been working without any egg verification
for years).

MAL: then what you are arguing for is that the SSL changes in 2.7.9 should not have happened.  Which is an argument that Antoine and I at least are sympathetic to.... :)

On 05.04.2015 23:25, R. David Murray wrote:
> 
> MAL: then what you are arguing for is that the SSL changes in 2.7.9 should not have happened.  Which is an argument that Antoine and I at least are sympathetic to.... :)

I think those changes were probably fine for many Python users,
just not all of them. I'm only arguing to get some easy way to
disable these enforced checks which doesn't require patching Python.

(So I guess I'm kind of standing in the middle between Antoine and
you on one side and Donald on the other side ;-))

Actually I was in favor of an environment variable (or something like that) from the start, because it could be set per-process (making it as close to per-application as we can get from upstream).  But a global config file I think is a bad idea (at least in the form so far suggested).

The change in 2.7.9 upstream was *absolutely* the right thing for the upstream CPython community to do. The problem was real, it needed to be fixed, and the community fixed it in a way that works just fine for folks in the earlier parts of the technology adoption curve.

Change management for the folks in the latter half of the technology adoption curve is a key part of what commercial redistributors get paid for. Delaying PEP 476 while we figured out the details of how that was going to work would have been a bad plan from a community perspective, so I took a speculative shot at providing a very simple solution for the redistributor case and unfortunately missed the target.

The reason I still want to negotiate the technical details of the feature upstream (despite missing the mark in PEP 476 itself) is so that all of us that need this functionality can provide the *same* behaviour to our respective customers, rather than having Red Hat do one thing, Suse another, Canonical a third, and then cross-platform Python redistributors like eGenix and ActiveState also needing to figure out their own scheme. It's akin to the problem faced by Linux redistributors that independently provide stable ABI guarantees, but also aim to collaborate on backporting fixes to the *same* stable ABI to reduce duplicated effort: http://crunchtools.com/deep-dive-rebase-vs-backport/#Brief_History

So while this isn't a feature upstream itself needs, it's one potentially needed by multiple *downstreams*, so in my view it makes sense for us to work with upstream to come up with the "one obvious way" for redistributors to handle the problem (now that we know that my initial attempt at providing such a way doesn't work in practice).

Probably the closest precedents to this idea are PEP 394 (regarding management of the unqualified python symlink) and the section with recommendations for downstream redistributors in PEP 453 (bundling pip).

Le 06/04/2015 13:29, Nick Coghlan a écrit :
> 
> So while this isn't a feature upstream itself needs, it's one
potentially needed by multiple *downstreams*, so in my view it makes
sense for us to work with upstream to come up with the "one obvious way"
for redistributors to handle the problem (now that we know that my
initial attempt at providing such a way doesn't work in practice).

So would it be possible for the actual implementation to be done outside
of CPython? (in a dedicated fork, for example)

>Le 06/04/2015 13:29, Nick Coghlan a écrit :
>> 
>> So while this isn't a feature upstream itself needs, it's one
potentially needed by multiple *downstreams*, so in my view it makes
sense for us to work with upstream to come up with the "one obvious way"
for redistributors to handle the problem (now that we know that my
initial attempt at providing such a way doesn't work in practice).
>
>So would it be possible for the actual implementation to be done outside
of CPython? (in a dedicated fork, for example)

Yes it would and most likely will be, but as Nick pointed out, it is important to come up with the "one obvious way".


I understand why my patch is not acceptable for the upstream, it was my first shot (yet suitable for us) to start a discussion about cert verification. 

From the proposed solutions mentioned I favour the ENV variable which would address also Donald concerns, using ENV variable per application to enable/disable cert verification instead of global enable/disable, (yet it could be also `export`ed for global settings), are there any real disadvantages of using this method?

I believe the original objection was that it made it too easy to globally (and in a not-obvious-to-the-end-user way) disable validation.  That argument seems to apply equally well to the proposed patch, so an environment var at least isn't worse; but it does make it less likely that it will be accepted as a 3.5 feature.

Environment variables are hidden state. It makes them rather dangerous from a security POV (even more so than a root-modifiable configuration file, since it is less well-defined who can set an environment variable that will be inherited by some process).

As long as the solution that is decided on isn't part of vanilla Python, I care a bit less, of course :-)

I like the idea of a separate "2.7-redistributor" branch to capture changes like this, as that would almost *exactly* duplicate the kernel maintenance model.

That approach would also mean that we don't have to figure out sensible upstream documentation for features like this that only make sense in the context of a redistributor that is responsible for ensuring they're used appropriately.

In issue 23955, Steve Dower has suggested introducing config file support for easier control of path configuration when a dedicated Python interpreter runtime is deployed as part of a larger application.

If that proposal goes ahead (I think it needs a PEP for us to proceed with it), then we could use that scheme as the basis for solving the PEP 476 backporting problem (without necessarily having to officially standardise the latter upstream).

I think this discussion is moving in the wrong direction or least one which won't help people not using some Linux distribution.

The use case here is very similar to the hash seed randomization which was also successfully handled using an environment variable setting, so why not do the same here ?

I don't really understand the objections mentioned against env vars. They can be set per process, per user, even globally and they are under control by whoever runs an application.

Note that this is about breaking backwards compatibility badly. Certificate verification is a good thing, but if it results in people no longer being able to easily upgrade to a new patch level release, something is wrong. If such a feature causes applications to fail working, admins won't go in a fix the application; instead they'll simply not upgrade to 2.7.9+, cutting people off of all the other fixes in 2.7.9+.

Folks being wary of upgrading to new maintenance releases is already the case - RHEL/CentOS selectively backport things, and other orgs like Google do extensive integration testing before deploying new versions. 

Folks that only use and write well behaved and well maintained software can readily upgrade to new point releases, large enough organisations where that assumption isn't necessarily valid end up having to work a bit harder :)

That said, I agree a hash randomisation style approach using environment variables should also work, I just expect it might be a little harder to check in a security auditing script.

After further consideration, I realised there's an important difference between this case and the hash randomisation case: having the "-E" switch imply hash randomisation was OK, but having it imply HTTPS certificate verification after the system administrator has explicitly turned it off is going to cause problems.

The system administrator controlled configuration file gets around that by not relying on the interpreter's environment variable based configuration support.

As a result, I've now recommended pursuing the configuration file based approach, with a PEP to standardise the precise name, format and semantics for the configuration file: https://bugzilla.redhat.com/show_bug.cgi?id=1173041#c8

Redistributors would opt-in by patching their system Python to implement that informational PEP, rather than the feature appearing in upstream CPython itself.

Changing the title to reflect that the solution to how to configure Python is still up in the air.

I also started a thread on python-dev to get some more feedback.

Please let me join the party. :)

Like Antoine and Donald I'm against an option to disable certificate validation. I truly believe it's the wrong approach for the problem. 

Users don't *want* to disable security checks either. They disable the check because a SSL verification error is disruptive and they want to get on with their lives. Because with Python they have no other easy option they take the quick and easy path. *Yoda's voice* If you end SSL verification now - if you choose the quick and easy path as others did - you will become an agent of evil.

I like to suggest a better way. Let's handle cert checks like Firefox or OpenSSH. Both give you the option to trust an unknown certificate for a specific host name and remember this trust, too. Let's add a feature to do the same with Python. Yes, it would require more work, additional features and careful engineering. But I strongly believe it's the better approach.

Rough design idea:

$ python ssl trustcert https://192.168.42.1

This command retrieves the cert from 192.168.42.1:443 and stores the mapping of 192.168.42.1 to SPKI sha512 hash in a file/directory relative sys.prefix.

When a ssl._create_stdlib_context() context gets a verification error, then it checks the file for the hostname and SPKI hash of the leaf certificate.

This features requires access to SPKI as DER and a proper verify_cb callback function. Disclaimer: I have code for the first feature and a plan for the second.

PS: It's also super easy to trust self-signed certificates. All you have to do is to grab the cert and set SSL_CERT_FILE env var:

$ openssl s_client -connect host:443 | openssl x509 > /path/to/selfsigned.pem
$ SSL_CERT_FILE=/path/to/selfsigned.pem python script.py

Those are nice ideas, but you are forgetting two important points:

 * browsers are typically only being used by single users,
   applications by potentially hundreds or thousands of users

 * how should the poor sys admin who's task it is to keep Python
   up to date know which SSL certs to add to the trust store ?

E.g. assume your application fetches user comments for sentiment
analysis from a few thousand sites, or gathers status updates
from a few hundred routers and switches you have installed
at your site, or even more difficult: an application which
tries to map your IT world of a few thousand network nodes,
scanning port 443 for useful information.

For eGenix PyRun we have now implemented an env var PYRUN_HTTPSVERIFY
which can be set to 0 to disable the checks and revert back to
Python 2.7.8 standards, if necessary, on a per process basis.

Right, the key here is to think like a system administrator, not a
developer. Most of those folks are downstream of redistributors (whether
commercial ones or community Linux distributions) and relying on one of two
things:

* tools using the system cert store for certificate validation (so they can
trust an internal CA automatically)
* tools not verifying certificates properly

The *right* answer on Linux is option one (which the system Python will be
configured to use by default), but even with tools like DogTag available as
open source, running your own internal CA properly is currently still a
pain, especially once you start accounting for all the hardware devices out
there with tragically bad certificate management. You can't just wave a
magic wand and suddenly have all your physical gear catch up to the modern
state of the art in SSL/TLS management as learned on the public internet -
it's a staged upgrade project where the risks of insider threats and other
perimeter compromises get traded off against the upgrade costs and
infrastructure stability risks. This *is* work that needs to be done given
the world we live in, but we also need to trust CIOs to appropriately
manage the upgrade plans for their own intranets.

It's not a coincidence that initiatives like Let's Encrypt are due to
launch this year, nor that Red Hat's started hiring people like Christian
to help integrate SSL certificate management directly into Linux identity &
authentication management - this stuff currently gets done badly because
it's *too hard* to do it right.

But in the meantime, admins upgrading Python *2.7* need a way to say "let
us decide what our highest priority infrastructure risks are, thank you
very much". They don't need that for Python 3, as it doesn't have the same
kind of large install base in environments where infrastructure
modernisation represents a major ongoing investment. That means "your
SSL/TLS certificate management must be in good order" can reasonably be a
gating criterion for Python 3, but we need a better solution for
redistributors and administrators when it comes to Python 2.7 upgrades.

First draft of a recommendations PEP: https://hg.python.org/peps/rev/85bc7f13b295 (PEP 493)

Re: platform_default - I'm not sure that's a good idea.  It hides what's actually happening in some hard to discover place (the code).  Probably EIBTI and just go with 'enable' and 'disable'.

The rationale behind "platform_default" relates to what we put in the default config file in the RPM. If enable/disable are the only options, then as soon as the first version ships with "disable" as the default, affected systems will *never* switch to being enabled by default unless the system administrator changes it.

By contrast, if we put "platform_default" in the default configuration, then inattentive sysadmins could theoretically eventually have their defaults switched to "enable" at some point.

We don't know yet if we'd ever upgrade the "platform_default" setting, but I think it's worthwhile to retain the option.

On Oct 14, 2015, at 03:20 PM, Nick Coghlan wrote:

>The rationale behind "platform_default" relates to what we put in the default
>config file in the RPM. If enable/disable are the only options, then as soon
>as the first version ships with "disable" as the default, affected systems
>will *never* switch to being enabled by default unless the system
>administrator changes it.
>
>By contrast, if we put "platform_default" in the default configuration, then
>inattentive sysadmins could theoretically eventually have their defaults
>switched to "enable" at some point.
>
>We don't know yet if we'd ever upgrade the "platform_default" setting, but I
>think it's worthwhile to retain the option.

We've consulted with the Ubuntu security team and have decided not to enable
it for Ubuntu 14.04 LTS.  For upgrades from there to newer releases, we won't
include the patch and will just enable it by default.  So for us,
platform_default doesn't make sense.

Yeah, it's the extra 5 years on RHEL 7 that makes me wary. For anything with a shorter life cycle, letting the legacy setting age out likely makes more sense.

Since PEP 493 is now a standards track PEP, the attached patch provides the reference implementation for the current PEP text.

The attached patch adds documentation (to the ssl module docs, the environment variable descriptions and the 2.7 What's New) and also fixes the PYTHONHTTPSVERIFY tests to use a subprocess so they can still run under -E.

Assuming the docs pass muster, then the only bit missing should be the NEWS entry.

Explicitly noting for anyone considering backporting this change (together with PEP 466 & 476) to a long term support release: watch out for https://bugs.python.org/issue22438

The RHEL/CentOS backport includes a reimplementation of sslwrap:

* https://git.centos.org/blob/rpms!python.git/f63228654ecef84a78c552dac832f4cd939cf584/SPECS!python.spec#L989
* https://git.centos.org/blob/rpms!python.git/f63228654ecef84a78c552dac832f4cd939cf584/SOURCES!00221-pep466-backport-sslwrap-c-ssl.patch

If test fail it will print out non-telling message which make debugging a little bit hard:

 FAIL: test__https_verify_envvar (test.test_ssl.ContextTests)                                        
 ----------------------------------------------------------------------                              
 Traceback (most recent call last):                                                                  
   File "/builddir/build/BUILD/Python-2.7.5/Lib/test/test_ssl.py", line 1143, in test__https_verify_envvar
     assert_python_ok("-c", https_is_verified, **extra_env)                                          
   File "/builddir/build/BUILD/Python-2.7.5/Lib/test/script_helper.py", line 55, in assert_python_ok
     return _assert_python(True, *args, **env_vars)                                                  
   File "/builddir/build/BUILD/Python-2.7.5/Lib/test/script_helper.py", line 47, in _assert_python  
     "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))                                    
 AssertionError: Process return code is 1, stderr follows:

Would be possible to change following code which is being executed:

+        https_is_verified = """import ssl, sys;\
+            sys.exit(ssl._create_default_https_context is not
+                     ssl.create_default_context)"""

into something like

https_is_verified = """import ssl, sys;\
value = ssl._create_default_https_context is not ssl.create_default_context;\
sys.exit('ssl._create_default_https_context should be set to verified' if value else value)"""

So traceback will look like this:
...
    "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is 1, stderr follows:
ssl._create_default_https_context should be set to verified

As Robert suggested, I tweaked the envvar tests to be more self-explanatory:

        https_is_verified = """import ssl, sys; \
            status = "Error: _create_default_https_context does not verify certs" \
                       if ssl._create_default_https_context is \
                          ssl._create_unverified_context \
                     else None; \
            sys.exit(status)"""
        https_is_not_verified = """import ssl, sys; \
            status = "Error: _create_default_https_context verifies certs" \
                       if ssl._create_default_https_context is \
                          ssl.create_default_context \
                     else None; \
            sys.exit(status)"""

I'm going to check this in as is and mark the PEP as Final. If anyone spots any minor cleanup issues, feel free to comment on them here, otherwise I'd suggest either opening a new issue or reopening this one for any larger concerns.

New changeset 7a9c9368d79e by Nick Coghlan in branch '2.7':
Issue #23857: Implement PEP 493
https://hg.python.org/cpython/rev/7a9c9368d79e