Python's hashlib and ssl modules currently leverage OpenSSL to provide developers with access to cryptographic hash and TLS routines, but encryption/decryption and signature/verification support are still missing. I propose the addition of an easy-to-use crypto module modeled after Evpy[0] to remedy this.

apologies, forgot the link:

[0] http://gitorious.org/evpy

Assuming you are willing to contribute evpy (and have the rights to do so, i.e. all of the code is truly yours): what's the user acceptance of the code?

In particular, what do authors of competing OpenSSL wrappers (like M2Crypto) or other Python crypto packages (like pycrypto) think about this idea?

> and have the rights to do so, i.e. all of the code is truly yours

Is it really required, or is a non-copyleft liberal license (MIT-like or BSD-like) enough?

>> and have the rights to do so, i.e. all of the code is truly yours
>
> Is it really required, or is a non-copyleft liberal license (MIT-like or BSD-like) enough?

The contributor would have to sign a contributor agreement, giving the 
PSF the right to relicense under the PSF license (or anything they 
please to relicense under). If the contributor only has a BSD license 
(from his contributors), he has no right to contribute the code under 
the contributor agreement (i.e. he, himself, wouldn't have the right
to relicense).

> The contributor would have to sign a contributor agreement, giving the 
> PSF the right to relicense under the PSF license (or anything they 
> please to relicense under). If the contributor only has a BSD license 
> (from his contributors), he has no right to contribute the code under 
> the contributor agreement (i.e. he, himself, wouldn't have the right
> to relicense).

I always forget about that :/

On Mon, Jun 14, 2010 at 3:09 PM, Martin v. Löwis <report@bugs.python.org> wrote:
>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
> Assuming you are willing to contribute evpy (and have the rights to do so, i.e. all of the code is truly yours): what's the user acceptance of the code?

I'd be willing to, but I see more utility in contributing specific
elements of its
functionality to the stdlib. Obviously the code is mine, and I can relicense
as needed if necessary.

As for your second question, I don't believe it sees much in the way of use.

> In particular, what do authors of competing OpenSSL wrappers (like M2Crypto) or other Python crypto packages (like pycrypto) think about this idea?

Evpy and M2Crypto have very different goals. M2Crypto seeks to be a
complete wrapper for OpenSSL, which we don't, and also uses SWIG,
which disqualifies it from consideration for the stdlib.

I don't know what the pycrypto folks would say about evpy, but I admit
to being very wary of that project- it appears to have been constructed
in a way which lends itself well to academic exercise rather than
practical use by nonexperts, and have had multiple occasions to correct
its dire misuse.

Geremy Condra

> Evpy and M2Crypto have very different goals. M2Crypto seeks to be a
> complete wrapper for OpenSSL, which we don't, and also uses SWIG,
> which disqualifies it from consideration for the stdlib.
>
> I don't know what the pycrypto folks would say about evpy, but I admit
> to being very wary of that project- it appears to have been constructed
> in a way which lends itself well to academic exercise rather than
> practical use by nonexperts, and have had multiple occasions to correct
> its dire misuse.

That isn't really my question; it's the other way 'round: what do *they*
(i.e. the respective authors) say about evpy? In the absence of actual
user input, support for inclusion of it by these experts would be a 
valuable indication that this specific library should be included. 
Likewise, objective resistance may lead to significant changes before 
inclusion, or to rejection. In the absence of both user support and 
expert opinions, I'd ask for a PEP.

On Mon, Jun 14, 2010 at 3:37 PM, Martin v. Löwis <report@bugs.python.org> wrote:
>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
>> Evpy and M2Crypto have very different goals. M2Crypto seeks to be a
>> complete wrapper for OpenSSL, which we don't, and also uses SWIG,
>> which disqualifies it from consideration for the stdlib.
>>
>> I don't know what the pycrypto folks would say about evpy, but I admit
>> to being very wary of that project- it appears to have been constructed
>> in a way which lends itself well to academic exercise rather than
>> practical use by nonexperts, and have had multiple occasions to correct
>> its dire misuse.
>
> That isn't really my question; it's the other way 'round: what do *they*
> (i.e. the respective authors) say about evpy? In the absence of actual
> user input, support for inclusion of it by these experts would be a
> valuable indication that this specific library should be included.
> Likewise, objective resistance may lead to significant changes before
> inclusion, or to rejection. In the absence of both user support and
> expert opinions, I'd ask for a PEP.

I have no idea, and as I said earlier in the mailing list, I'm
willing to contribute the code, make changes as requested,
and maintain it- but I have no interest in or skill with the
political footwork the process demands. I like to think that
if this is as widely desired as it is asked for on python-list
that a champion will sooner or later emerge.

Geremy Condra

Le lundi 14 juin 2010 à 22:48 +0000, geremy condra a écrit :
> 
> I have no idea, and as I said earlier in the mailing list, I'm
> willing to contribute the code, make changes as requested,
> and maintain it- but I have no interest in or skill with the
> political footwork the process demands. I like to think that
> if this is as widely desired as it is asked for on python-list
> that a champion will sooner or later emerge.

For the record, Gregory P Smith (current maintainer of hashlib -- if I'm
not mistaken :-)), Jean-Paul Calderone (maintainer of pyOpenSSL) and
Heikki Toivonen (maintainer of m2crypto) have been added to the nosy
list for this issue.
As for the built-in ssl module, I've been doing most of the maintenance
work on it lately.

On Mon, Jun 14, 2010 at 6:51 PM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Le lundi 14 juin 2010 à 22:48 +0000, geremy condra a écrit :
>>
>> I have no idea, and as I said earlier in the mailing list, I'm
>> willing to contribute the code, make changes as requested,
>> and maintain it- but I have no interest in or skill with the
>> political footwork the process demands. I like to think that
>> if this is as widely desired as it is asked for on python-list
>> that a champion will sooner or later emerge.
>
> For the record, Gregory P Smith (current maintainer of hashlib -- if I'm
> not mistaken :-)), Jean-Paul Calderone (maintainer of pyOpenSSL) and
> Heikki Toivonen (maintainer of m2crypto) have been added to the nosy
> list for this issue.
> As for the built-in ssl module, I've been doing most of the maintenance
> work on it lately.

Lot of people. If nobody minds I'm going to go ahead and post a
link to this on python-crypto, since a lot of the interface emerged
out of discussions that group had at pycon.

I'd also urge folks who are interested in this to be vocal about
whether they like the API and where they'd like to see changes-
I'm open to suggestions and, as noted in the mailing list, am
reimplementing in C, so this is a good time to be talking about
where you'd like to see things go.

Geremy Condra

I've taken a quick look at the source tree (there doesn't seem to be any separate docs) and here is my opinion:
- the evp.py API is too low-level (it's a one-to-one mapping to the OpenSSL C API); we would want at least some kind of object-oriented abstraction around the basic concepts (such as in the hashlib and ssl modules) rather than passing opaque pointers around
- the other APIs (cipher.py, envelope.py, signature.py) look conversely too high-level, since they focus on specific use cases and make some arbitrary choices for the user (for example, envelope.py imposes AES-192)

By the way, the use of function signature annotations to mirror C APIs as Python APIs through ctypes is nice, perhaps you should upload it as a separate library on PyPI :)

On Tue, Jun 15, 2010 at 9:21 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> I've taken a quick look at the source tree (there doesn't seem to be any separate docs) and here is my opinion:
> - the evp.py API is too low-level (it's a one-to-one mapping to the OpenSSL C API); we would want at least some kind of object-oriented abstraction around the basic concepts (such as in the hashlib and ssl modules) rather than passing opaque pointers around

evp.py is mostly for internal use (to map the openssl calls into
Python) and won't exist in the rewrite- most of the people who would
want to use that should really be using M2Crypto or similar.

> - the other APIs (cipher.py, envelope.py, signature.py) look conversely too high-level, since they focus on specific use cases and make some arbitrary choices for the user (for example, envelope.py imposes AES-192)

The goals of the library are simplicity and ease of use. I've
frequently found that out of fear of making incorrect choices, people
will simply decide not to use crypto at all, or that they make
incredibly stupid choices like using RSA without padding. I'd be
willing to add in the option to alter those options via keyword
arguments if it became a major point of contention, but in general I
think its better for those who "just want to encrypt something" to
have a lot of those decisions made for them. The specific decision
you're talking about was made because while AES-256 has a bigger
number at the end, its key schedule appears weaker in light of recent
attacks.

> By the way, the use of function signature annotations to mirror C APIs as Python APIs through ctypes is nice, perhaps you should upload it as a separate library on PyPI :)

I've posted them as recipes on ASPN ([0] and [1]). I used a similar
technique and the JNI to mechanically wrap the Android libraries
(Java) for access from Python, and it worked pretty well. Looking at
the data from pypi, ease-of-use things don't seem to see a lot of use,
but if you think I ought to then I could go ahead and do that.

Geremy Condra

[0] http://code.activestate.com/recipes/576731-c-function-decorator/
[1] http://code.activestate.com/recipes/576734-c-struct-decorator/

AFAIK, what the stdlib needs is a high-level crypto module, analogous to hashlib

Le mardi 15 juin 2010 à 14:49 +0000, geremy condra a écrit :
> The goals of the library are simplicity and ease of use. I've
> frequently found that out of fear of making incorrect choices, people
> will simply decide not to use crypto at all, or that they make
> incredibly stupid choices like using RSA without padding. I'd be
> willing to add in the option to alter those options via keyword
> arguments if it became a major point of contention, but in general I
> think its better for those who "just want to encrypt something" to
> have a lot of those decisions made for them. The specific decision
> you're talking about was made because while AES-256 has a bigger
> number at the end, its key schedule appears weaker in light of recent
> attacks.

While it's fine to perhaps detect and warn about insecure use, I don't
think the API should be too directive (for inclusion in the stdlib
anyway). Most (if not all) stdlib modules don't impose any specific
policy but instead provide building blocks for users to address their
specific needs. Directive APIs should probably be left to third-party
libraries (which can of course build on the primitives provided by the
stdlib). Also, some uses of crypto functions can be to interoperate with
existing cryptographic protocols, and for that you need a fine-grained
control over algorithmic options.

Do note that the docs can be as educating as needed; they can include
suggestions, warnings and even small recipes.

As for default argument values, the problem is that we're then stuck
with them (for compatibility). It means that if e.g. AES-192 gets
compromised, Python will promote an API which by default is insecure and
dangerous to use. Again, giving equal access to various ciphers and then
providing guidance in the documentation would be a better compromise.

On Tue, Jun 15, 2010 at 9:49 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Le mardi 15 juin 2010 à 14:49 +0000, geremy condra a écrit :
>> The goals of the library are simplicity and ease of use. I've
>> frequently found that out of fear of making incorrect choices, people
>> will simply decide not to use crypto at all, or that they make
>> incredibly stupid choices like using RSA without padding. I'd be
>> willing to add in the option to alter those options via keyword
>> arguments if it became a major point of contention, but in general I
>> think its better for those who "just want to encrypt something" to
>> have a lot of those decisions made for them. The specific decision
>> you're talking about was made because while AES-256 has a bigger
>> number at the end, its key schedule appears weaker in light of recent
>> attacks.
>
> While it's fine to perhaps detect and warn about insecure use, I don't
> think the API should be too directive (for inclusion in the stdlib
> anyway). Most (if not all) stdlib modules don't impose any specific
> policy but instead provide building blocks for users to address their
> specific needs. Directive APIs should probably be left to third-party
> libraries (which can of course build on the primitives provided by the
> stdlib). Also, some uses of crypto functions can be to interoperate with
> existing cryptographic protocols, and for that you need a fine-grained
> control over algorithmic options.

I'm not clear on how a crypto library is supposed to detect insecure
use short of simply not allowing suspicious things. Maybe you have
some ideas there?

As for building-block type systems, like I say they have their place,
particularly where interoperating with existing systems is a concern,
and I don't want to come across as though I don't respect projects
like M2Crypto- I just think that most developers don't need that
level of complexity and aren't prepared to invest the time to learn
how to get what they want out of it. That's where something like
evpy shines.

> Do note that the docs can be as educating as needed; they can include
> suggestions, warnings and even small recipes.

I'm reasonably sure that there aren't enough docs in the world to
stop people from using OpenSSL to live dangerously. Evpy you
could get people not to be completely stupid with, at least a large
portion of the time.

> As for default argument values, the problem is that we're then stuck
> with them (for compatibility). It means that if e.g. AES-192 gets
> compromised, Python will promote an API which by default is insecure and
> dangerous to use. Again, giving equal access to various ciphers and then
> providing guidance in the documentation would be a better compromise.

I would be enormously surprised if a weakness in AES-192 was found that
weakened it to the point where it would actually constitute bad advice,
assuming that you made all the other right decisions. Having said that, it
might be a good idea to put a version switch in that allowed you to specify
compatibility modes, just in case.

Geremy Condra

More or less random opinions on things presented before:

 * I prefer having secure defaults to over documentation, because, well, people don't read documentation.
 * If not secure defaults, then pointing out in documentation the secure way AND providing examples that always show the secure way of doing things.
 * I can't comment on aes 192 vs 256 as I have not really kept up with that, but it would be good to ask the opinion(s) of the real experts in this field before choosing the defaults/recommending them. Of course, if you can point to an article where the experts already voice their (recent) recommendations, fine.
 * When I have thought about Python crypto in the stdlib, I've considered modeling it after hashlib, so you would get cipher = cryptolib.AES(bits=192, ...) etc. (Caveat: haven't thought it through.)
 * I'd prefer if the crypto API didn't become OpenSSL specific (like the SSL one is), which would theoretically allow switching in other crypto provider(s).
 * The library should make it easy to do the most common operations with as few steps as practically possible.
 * It would be nice if the library could provide the means to tweak lower level things if you needed to. Unfortunately this has a tendency to get messy quick, because crypto stuff tends to have lots of options to tweak.

On Thu, Jun 17, 2010 at 8:01 PM, Heikki Toivonen <report@bugs.python.org> wrote:
>
> Heikki Toivonen <hjtoi-bugzilla@comcast.net> added the comment:
>
> More or less random opinions on things presented before:
>
>  * I prefer having secure defaults to over documentation, because, well, people don't read documentation.

Wholeheartedly agree.

>  * If not secure defaults, then pointing out in documentation the secure way AND providing examples that always show the secure way of doing things.

Not as big a fan, honestly. Most domain-specific projects can count on
those reading the documentation to have a good idea of what it is that
they actually want to do; in crypto this does not seem to be the case
very often, and that's a tricky problem to fix that in the scope of a
recipe or piece of documentation.

>  * I can't comment on aes 192 vs 256 as I have not really kept up with that, but it would be good to ask the opinion(s) of the real experts in this field before choosing the defaults/recommending them. Of course, if you can point to an article where the experts already voice their (recent) recommendations, fine.

http://eprint.iacr.org/2009/317.pdf
http://eprint.iacr.org/2009/374.pdf
http://eprint.iacr.org/2009/241.pdf

Bruce Schneier's take:
http://www.schneier.com/blog/archives/2009/07/another_new_aes.html

The only cryptosystem/padding/etc choice in evpy I'm uncomfortable
with (at the moment ;) ) is the use of ad-hoc padding rather than
OAEP, and I only do that because that's what evp does. Of course, if
you have any other concerns I'd appreciate hearing about them.

>  * When I have thought about Python crypto in the stdlib, I've considered modeling it after hashlib, so you would get cipher = cryptolib.AES(bits=192, ...) etc. (Caveat: haven't thought it through.)

I'm not opposed to this, but I suspect that focusing on what the
algorithms are for rather than what they are reduces the cognitive
load somewhat. Perhaps a two-tier api?

>  * I'd prefer if the crypto API didn't become OpenSSL specific (like the SSL one is), which would theoretically allow switching in other crypto provider(s).

I agree in theory, although I'm not sure how important this is likely
to be in practice.

>  * The library should make it easy to do the most common operations with as few steps as practically possible.
>  * It would be nice if the library could provide the means to tweak lower level things if you needed to. Unfortunately this has a tendency to get messy quick, because crypto stuff tends to have lots of options to tweak.

100% agree. If you have any ideas- or if anyone else does- on how best
to do this, I'd be very happy to discuss it.

Geremy Condra

>>   * I'd prefer if the crypto API didn't become OpenSSL specific (like the SSL one is), which would theoretically allow switching in other crypto provider(s).
>
> I agree in theory, although I'm not sure how important this is likely
> to be in practice.

I always wanted to drop OpenSSL from the Windows binaries, and use MS 
CryptoAPI instead.

>  * When I have thought about Python crypto in the stdlib, I've considered modeling it after hashlib, so you would get cipher = cryptolib.AES(bits=192, ...) etc. (Caveat: haven't thought it through.)

I think there is a relevant PEP: PEP 272 -- API for Block Encryption Algorithms v1.0 (http://www.python.org/dev/peps/pep-0272/ )
It describes an API somewhat similar to hashlib.

On Fri, Jun 18, 2010 at 2:19 AM, Martin v. Löwis <report@bugs.python.org> wrote:
>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
>>>   * I'd prefer if the crypto API didn't become OpenSSL specific (like the SSL one is), which would theoretically allow switching in other crypto provider(s).
>>
>> I agree in theory, although I'm not sure how important this is likely
>> to be in practice.
>
> I always wanted to drop OpenSSL from the Windows binaries, and use MS
> CryptoAPI instead.

My familiarity with the CryptoAPI is limited, but I think doing this
for something like evpy would be possible. I also suspect that doing
it for anything that exposed much more than evpy does would grow very
quickly in complexity where it was possible at all.

Geremy Condra

On Fri, Jun 18, 2010 at 2:39 AM, Daniel Urban <report@bugs.python.org> wrote:
>
> Daniel Urban <urban.dani+py@gmail.com> added the comment:
>
>>  * When I have thought about Python crypto in the stdlib, I've considered modeling it after hashlib, so you would get cipher = cryptolib.AES(bits=192, ...) etc. (Caveat: haven't thought it through.)
>
> I think there is a relevant PEP: PEP 272 -- API for Block Encryption Algorithms v1.0 (http://www.python.org/dev/peps/pep-0272/ )
> It describes an API somewhat similar to hashlib.

Again, I'm not entirely opposed to this, but I think it represents a
lower-level API than most developers can really be safely trusted to
handle.

Geremy Condra

Le vendredi 18 juin 2010 à 06:46 +0000, geremy condra a écrit :
> geremy condra <debatem1@gmail.com> added the comment:
> 
> On Fri, Jun 18, 2010 at 2:39 AM, Daniel Urban <report@bugs.python.org> wrote:
> >
> > Daniel Urban <urban.dani+py@gmail.com> added the comment:
> >
> >>  * When I have thought about Python crypto in the stdlib, I've considered modeling it after hashlib, so you would get cipher = cryptolib.AES(bits=192, ...) etc. (Caveat: haven't thought it through.)
> >
> > I think there is a relevant PEP: PEP 272 -- API for Block Encryption Algorithms v1.0 (http://www.python.org/dev/peps/pep-0272/ )
> > It describes an API somewhat similar to hashlib.
> 
> Again, I'm not entirely opposed to this, but I think it represents a
> lower-level API than most developers can really be safely trusted to
> handle.

If there is a contention or disagreement between different API styles,
it may be wise to seek opinions on python-dev or python-ideas.

I'd point out that the "ssl" module itself seems to have evolved from a
trivial wrapper API (in the 2.5 docs I can only find a single
3-parameter function, socket.ssl()) to a more comprehensive API in 3.2,
because people ultimately need the functionalities.
(and yet the ssl API in 3.2 is still much less featureful than M2Crypto
or pyOpenSSL are)

On Fri, Jun 18, 2010 at 3:09 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Le vendredi 18 juin 2010 à 06:46 +0000, geremy condra a écrit :
>> geremy condra <debatem1@gmail.com> added the comment:
>>
>> On Fri, Jun 18, 2010 at 2:39 AM, Daniel Urban <report@bugs.python.org> wrote:
>> >
>> > Daniel Urban <urban.dani+py@gmail.com> added the comment:
>> >
>> >>  * When I have thought about Python crypto in the stdlib, I've considered modeling it after hashlib, so you would get cipher = cryptolib.AES(bits=192, ...) etc. (Caveat: haven't thought it through.)
>> >
>> > I think there is a relevant PEP: PEP 272 -- API for Block Encryption Algorithms v1.0 (http://www.python.org/dev/peps/pep-0272/ )
>> > It describes an API somewhat similar to hashlib.
>>
>> Again, I'm not entirely opposed to this, but I think it represents a
>> lower-level API than most developers can really be safely trusted to
>> handle.
>
> If there is a contention or disagreement between different API styles,
> it may be wise to seek opinions on python-dev or python-ideas.

I'm not sure there's a disagreement here except what the top-level API
should be. If someone is really determined to use the lower-level API
I have no issue with it, and (within the bounds of time and ability)
am willing to write the code to support it.

> I'd point out that the "ssl" module itself seems to have evolved from a
> trivial wrapper API (in the 2.5 docs I can only find a single
> 3-parameter function, socket.ssl()) to a more comprehensive API in 3.2,
> because people ultimately need the functionalities.
> (and yet the ssl API in 3.2 is still much less featureful than M2Crypto
> or pyOpenSSL are)

I'm not sure I'm understanding what you mean. Are you saying it should
start as a comprehensive wrapper because that's what ssl is headed
towards or that it should start simply because such functionality will
evolve organically as the need arises?

Geremy Condra

> > I'd point out that the "ssl" module itself seems to have evolved from a
> > trivial wrapper API (in the 2.5 docs I can only find a single
> > 3-parameter function, socket.ssl()) to a more comprehensive API in 3.2,
> > because people ultimately need the functionalities.
> > (and yet the ssl API in 3.2 is still much less featureful than M2Crypto
> > or pyOpenSSL are)
> 
> I'm not sure I'm understanding what you mean. Are you saying it should
> start as a comprehensive wrapper because that's what ssl is headed
> towards or that it should start simply because such functionality will
> evolve organically as the need arises?

The former. Evolving organically has quite a few issues, because the
original API may be far from ideal to build on, and yet you have to
ensure compatibility with that API.
("comprehensive" doesn't have to equate "exhaustive" of course. But any
API which tries to simplify things too much might also be a roadblock
when it comes to exposing more features)

On Fri, Jun 18, 2010 at 3:28 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
>> > I'd point out that the "ssl" module itself seems to have evolved from a
>> > trivial wrapper API (in the 2.5 docs I can only find a single
>> > 3-parameter function, socket.ssl()) to a more comprehensive API in 3.2,
>> > because people ultimately need the functionalities.
>> > (and yet the ssl API in 3.2 is still much less featureful than M2Crypto
>> > or pyOpenSSL are)
>>
>> I'm not sure I'm understanding what you mean. Are you saying it should
>> start as a comprehensive wrapper because that's what ssl is headed
>> towards or that it should start simply because such functionality will
>> evolve organically as the need arises?
>
> The former. Evolving organically has quite a few issues, because the
> original API may be far from ideal to build on, and yet you have to
> ensure compatibility with that API.
> ("comprehensive" doesn't have to equate "exhaustive" of course. But any
> API which tries to simplify things too much might also be a roadblock
> when it comes to exposing more features)

Well, like I say, I'm willing to contribute what time and ability
allow. Are you thinking of adding a comprehensive wrapper to the ssl
module?

Geremy Condra

> Well, like I say, I'm willing to contribute what time and ability
> allow. Are you thinking of adding a comprehensive wrapper to the ssl
> module?

Hmm, no, I was just providing an existing datapoint to help us deciding
on a crypto API.  AFAICT this issue hasn't much to do with the ssl
module, except perhaps for (positive or negative) inspiration ;-)
(and except that it will also - most likely - interface with OpenSSL)

On Fri, Jun 18, 2010 at 4:53 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
>> Well, like I say, I'm willing to contribute what time and ability
>> allow. Are you thinking of adding a comprehensive wrapper to the ssl
>> module?
>
> Hmm, no, I was just providing an existing datapoint to help us deciding
> on a crypto API.  AFAICT this issue hasn't much to do with the ssl
> module, except perhaps for (positive or negative) inspiration ;-)
> (and except that it will also - most likely - interface with OpenSSL)

The question in my mind then is whether anybody willing to contribute
time knows enough about the CryptoAPI, or NSS, or what-have-you, to
help craft an API that makes the waterfall model look manageable. If
not, I would suggest that we focus on defining and building a
lower-level interface along the lines of the PEP noted earlier,
integrating that with evpy, and getting it in shape to go into the
stdlib. At that point, if demand arises for an even lower level API,
we already have the wrapping functions for a lot of the calls into
OpenSSL or whatever, and we can build on those in the aforementioned
evolutionary fashion. If somebody does, then perhaps a four-tiered
model makes more sense, with the bottom one being the raw wrappers
around the various libs, the second from the bottom being
compatibility shims, and the top two matching the other proposal.
Having said that, it's not something I could take on alone.

Geremy Condra

> I would suggest that we focus on defining and building a
> lower-level interface along the lines of the PEP noted earlier,
> integrating that with evpy, and getting it in shape to go into the
> stdlib.

That sounds reasonable to me.
(although I would be also content with the lower-level interface
alone :-))

> If somebody does, then perhaps a four-tiered
> model makes more sense, with the bottom one being the raw wrappers
> around the various libs, the second from the bottom being
> compatibility shims, and the top two matching the other proposal.

That sounds much too complicated.

On Fri, Jun 18, 2010 at 5:37 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
>> I would suggest that we focus on defining and building a
>> lower-level interface along the lines of the PEP noted earlier,
>> integrating that with evpy, and getting it in shape to go into the
>> stdlib.
>
> That sounds reasonable to me.

Great, I'm thinking more-or-less the API proposed in PEP 272- the
exception I'm thinking of is that 'strings' should be substituted for
'bytes'- for AES and DES. It gets trickier when talking about public
key crypto, though. Perhaps something along the lines of
RSA.new(public_key=None, private_key=None,...), with the resulting
object supporting encrypt/decrypt/sign/verify operations?

> (although I would be also content with the lower-level interface
> alone :-))

>> If somebody does, then perhaps a four-tiered
>> model makes more sense, with the bottom one being the raw wrappers
>> around the various libs, the second from the bottom being
>> compatibility shims, and the top two matching the other proposal.
>
> That sounds much too complicated.

If we're likely to have openssl taken out from under us it could save
us a lot of hassle to plan for that up front. If not, then why worry,
and ISTM we should go the simpler route.

Geremy Condra

> Great, I'm thinking more-or-less the API proposed in PEP 272- the
> exception I'm thinking of is that 'strings' should be substituted for
> 'bytes'- for AES and DES. It gets trickier when talking about public
> key crypto, though. Perhaps something along the lines of
> RSA.new(public_key=None, private_key=None,...), with the resulting
> object supporting encrypt/decrypt/sign/verify operations?

I don't have any opinion right now. I think a concrete proposal should
be initiated and we can iterate from that.
(that's assuming other people agree on the principle, of course)

> If we're likely to have openssl taken out from under us it could save
> us a lot of hassle to plan for that up front.

It doesn't seem very likely in the middle term. In particular, the ssl
module itself is quite tied to OpenSSL's socket wrapping semantics
(including error codes and non-blocking behaviour), so OpenSSL will
probably still be required for SSL sockets.

On Fri, Jun 18, 2010 at 6:05 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
>> Great, I'm thinking more-or-less the API proposed in PEP 272- the
>> exception I'm thinking of is that 'strings' should be substituted for
>> 'bytes'- for AES and DES. It gets trickier when talking about public
>> key crypto, though. Perhaps something along the lines of
>> RSA.new(public_key=None, private_key=None,...), with the resulting
>> object supporting encrypt/decrypt/sign/verify operations?
>
> I don't have any opinion right now. I think a concrete proposal should
> be initiated and we can iterate from that.
> (that's assuming other people agree on the principle, of course)

I assume that by "a concrete proposal" you're talking about code? Or
API docs? Also, what more needs to be done to ensure that other people
agree on the principle?

>> If we're likely to have openssl taken out from under us it could save
>> us a lot of hassle to plan for that up front.
>
> It doesn't seem very likely in the middle term. In particular, the ssl
> module itself is quite tied to OpenSSL's socket wrapping semantics
> (including error codes and non-blocking behaviour), so OpenSSL will
> probably still be required for SSL sockets.

I'm fine with doing it the simpler way and adding in support for other
systems PRN. Having said that, Martin, if this is high priority for
you let me know.

Geremy Condra

Le samedi 19 juin 2010 à 00:55 +0000, geremy condra a écrit :
> geremy condra <debatem1@gmail.com> added the comment:
> 
> On Fri, Jun 18, 2010 at 6:05 AM, Antoine Pitrou <report@bugs.python.org> wrote:
> >
> > Antoine Pitrou <pitrou@free.fr> added the comment:
> >
> >> Great, I'm thinking more-or-less the API proposed in PEP 272- the
> >> exception I'm thinking of is that 'strings' should be substituted for
> >> 'bytes'- for AES and DES. It gets trickier when talking about public
> >> key crypto, though. Perhaps something along the lines of
> >> RSA.new(public_key=None, private_key=None,...), with the resulting
> >> object supporting encrypt/decrypt/sign/verify operations?
> >
> > I don't have any opinion right now. I think a concrete proposal should
> > be initiated and we can iterate from that.
> > (that's assuming other people agree on the principle, of course)
> 
> I assume that by "a concrete proposal" you're talking about code? Or
> API docs? Also, what more needs to be done to ensure that other people
> agree on the principle?

I was thinking about a PEP. Of course, you are free to reuse existing
PEP content for that :)

On Sat, Jun 19, 2010 at 7:52 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Le samedi 19 juin 2010 à 00:55 +0000, geremy condra a écrit :
>> geremy condra <debatem1@gmail.com> added the comment:
>>
>> On Fri, Jun 18, 2010 at 6:05 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>> >
>> > Antoine Pitrou <pitrou@free.fr> added the comment:
>> >
>> >> Great, I'm thinking more-or-less the API proposed in PEP 272- the
>> >> exception I'm thinking of is that 'strings' should be substituted for
>> >> 'bytes'- for AES and DES. It gets trickier when talking about public
>> >> key crypto, though. Perhaps something along the lines of
>> >> RSA.new(public_key=None, private_key=None,...), with the resulting
>> >> object supporting encrypt/decrypt/sign/verify operations?
>> >
>> > I don't have any opinion right now. I think a concrete proposal should
>> > be initiated and we can iterate from that.
>> > (that's assuming other people agree on the principle, of course)
>>
>> I assume that by "a concrete proposal" you're talking about code? Or
>> API docs? Also, what more needs to be done to ensure that other people
>> agree on the principle?
>
> I was thinking about a PEP. Of course, you are free to reuse existing
> PEP content for that :)

Ok. I've gone ahead and put together kind of a map for what I think the
basic structure of the library is going to look like. Let me know what you
think, and once we're done with that we can proceed into PEP land.

crypto API
==========
Variables message, key, salt, iv, ciphertext, and signature are of type bytes.
Variables public_key and private_key are DER-encoded bytes.
Variable bitlength is an integer.

Note that we deviate from the standard in PEP 272 in several ways:

	* arguments are generally bytes rather than strings
	* ciphers do not accept the 'counter', 'rounds', or 'segment_size' args

Layer 1
-------

Symmetric Ciphers
	crypto.cipher.encrypt(message, key) -> (salt, iv, ciphertext)
		depends on:
			crypto.keys.strengthen_password
			crypto.AES.new
			crypto.AES.encrypt
		raises:
			crypto.cipher.EncryptionError

	crypto.cipher.decrypt(salt, iv, ciphertext, key) -> message
		depends on:
			crypto.AES.new
			crypto.AES.decrypt
		raises:
			crypto.cipher.DecryptionError

Envelope Encryption
	crypto.envelope.encrypt(message, public_key) -> (iv, aes_key, ciphertext)
		depends on:
			crypto.keys.random_key
			crypto.AES.new
			crypto.AES.encrypt
			crypto.RSA.new
			crypto.RSA.encrypt
		raises:
			crypto.envelope.EncryptionError

	crypto.envelope.decrypt(iv, aes_key, ciphertext, private_key) -> message
		depends on:
			crypto.AES.new
			crypto.AES.decrypt
			crypto.RSA.new
			crypto.RSA.decrypt
		raises:
			crypto.envelope.DecryptionError

Digital Signatures
	crypto.signature.sign(message, private_key) -> signature
		depends on:
			hashlib.SHA512.new
			hashlib.SHA512.update
			hashlib.SHA512.digest
			crypto.RSA.new
			crypto.RSA.sign
		raises:
			crypto.signature.SigningError

	crypto.signature.verify(message, signature, public_key)
		depens on:
			hashlib.SHA512.new
			hashlib.SHA512.update
			hashlib.SHA512.digest
			crypto.RSA.new
			crypto.RSA.verify

Layer 2
-------

Utilities
	crypto.keys.strengthen_password(password) -> key
		depends on:
			openssl: RAND_bytes, EVP_get_digest_by_name, EVP_bytes_to_key
		raises:
			crypto.keys.KeyGenerationError

Symmetric Encryption
	crypto._cipher_object

		crypto._cipher_object.CipherObject._ctx = openssl context | None
		crypto._cipher_object.CipherObject._cipher = openssl cipher | None
		crypto._cipher_object.CipherObject._key = bytes | None

		CipherObject.encrypt(self, data) -> ciphertext
			depends on:
				crypto._cipher_object.CipherObject.encrypt_init
				crypto._cipher_object.CipherObject.encrypt_update
				crypto._cipher_object.CipherObject.encrypt_finalize
			raises:
				crypto._cipher_object.EncryptError

		CipherObject.encrypt_init() -> None
			depends on:
				openssl: EVP_EncryptInit_ex
			raises:
				crypto._cipher_object.EncryptInitError

		CipherObject.encrypt_update
			depends on:
				openssl: EVP_EncryptUpdate_ex
			raises:
				crypto._cipher_object.EncryptUpdateError

		CipherObject.encrypt_finalize
			depends on:
				openssl: EVP_EncryptFinal_ex
			raises:
				crypto._cipher_object.FinalizeError

		CipherObject.decrypt(self, ciphertext) -> message
			depends on:
				crypto._cipher_object.CipherObject.decrypt_init
				crypto._cipher_object.CipherObject.decrypt_update
				crypto._cipher_object.CipherObject.decrypt_finalize
			raises:
				crypto._cipher_object.DecryptError

		CipherObject.decrypt_init() -> None
			depends on:
				openssl: EVP_DecryptInit_ex
			raises:
				crypto._cipher_object.DecryptInitError

		CipherObject.decrypt_update
			depends on:
				openssl: EVP_DecryptUpdate_ex
			raises:
				crypto._cipher_object.DecryptUpdateError

		CipherObject.decrypt_finalize
			depends on:
				openssl: EVP_DecryptFinal_ex
			raises:
				crypto._cipher_object.DecryptFinalizeError

	crypto.AES
		crypto.AES.new(key, mode, IV=None) -> cipher_object

	crypto.DES
		crypto.DES.new(key, mode, IV=None) -> cipher_object

Asymmetric Encryption
	crypto.RSA
		crypto.RSA.new(public_key=None, private_key=None, padding=4) -> crypto.RSA.RSA
			depends on:
				openssl: d2i_RSAPublicKey, d2i_RSAPrivateKey
			raises:
				crypto.RSA.KeyError
				crypto.RSA.InitializationError

		crypto.RSA.generate_keypair(bitlength) -> public_key, private_key
			depends on:
				openssl: RSA_generate_key, i2d_RSAPublicKey, RSA_free
			raises:
				crypto.RSA.KeygenError

	crypt.RSA.RSA
		crypto.RSA.RSA._public_key = openssl RSA key | None
		crypto.RSA.RSA._private_key = openssl RSA key | None
		crypto.RSA.RSA._padding_type = integer

		crypto.RSA.RSA.encrypt(self, data) -> ciphertext
			depends on:
				openssl: RSA_size, RSA_public_encrypt
			raises:
				crypto.RSA.EncryptionError

		crypto.RSA.RSA.decrypt(self, ciphertext) -> message
			depends on:
				openssl: RSA_size, RSA_private_decrypt
			raises:
				crypto.RSA.DecryptionError

		crypto.RSA.RSA.sign(self, hash) -> signature
			depends on:
				openssl: RSA_size, RSA_sign
			raises:
				crypto.RSA.SigningError

		crypto.RSA.RSA.verify(self, hash, signature) -> True | False
			depends on:
				openssl: RSA_size, RSA_verify
			raises:
				crypto.RSA.VerificationError

Geremy Condra

Le dimanche 20 juin 2010 à 06:30 +0000, geremy condra a écrit :
> crypto API
> ==========
> [...]

For presentation purposes, I would order layers by abstraction levem:
that is, "layer 1" should be the lower-level layer and "layer 2" the
upper-level.

I think all further discussion should happen on the PEP itself.

Apart from the question of API, please also include a section on the legal implications this move would have on Python in the PEP.

We currently only include OpenSSL in the Windows installers and (for some reason) don't pay much attention to the implications this has (the fact is not mentioned on the download page and the Windows installer doesn't show the required OpenSSL old-style BSD attribution).

If we are to require OpenSSL or some other crypto lib, possibly even our own (e.g. pycrypto) for all platforms, then we could no longer just ignore the fact that crypto code is subject to strong legislation in many countries of the world.

> If we are to require OpenSSL or some other crypto lib,

We already depend on OpenSSL for both hashlib and ssl, this proposal
wouldn't change anything in this regard.

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>> If we are to require OpenSSL or some other crypto lib,
> 
> We already depend on OpenSSL for both hashlib and ssl, this proposal
> wouldn't change anything in this regard.

hashlib can still works without OpenSSL and hash algorithms don't
fall under crypto laws. ssl doesn't work without OpenSSL, but also
doesn't require adding any crypto code to the stdlib.

The main point that needs to be addressed is shipping Python
with crypto code. If OpenSSL is optionally used, we're fine,
but if we start shipping crypto code, things are more contrived.

See http://rechten.uvt.nl/koops/cryptolaw/ for a survey.

We're hosting the Python software on servers in The Netherlands,
so have to follow the Wassenaar Arrangement if we include
crypto code. Fortunately, that agreement includes a clause which
pretty much exempts open source crypto code from export regulations.

However, users of Python downloading installers with crypto software
would import and use it in their resp. countries and that may get
them into trouble, so they need to be warned if we decide to
ship crypto code with Python.

They way I understand Geremy's suggestion is to just include a
wrapper for OpenSSL, so that's fine. The PEP should include a
mention of the above to argue against putting e.g. pycrypto
into the stdlib (not because it's poor software, much to the
contrary, only because it causes lots of problems for our
users and the developers).

Marc-Andre Lemburg wrote:
> 
> We currently only include OpenSSL in the Windows installers and (for some reason) don't pay much attention to the implications this has (the fact is not mentioned on the download page and the Windows installer doesn't show the required OpenSSL old-style BSD attribution).

I've opened issue9119 to address this part.

On Tue, Jun 29, 2010 at 2:25 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>
> Antoine Pitrou wrote:
>>
>> Antoine Pitrou <pitrou@free.fr> added the comment:
>>
>>> If we are to require OpenSSL or some other crypto lib,
>>
>> We already depend on OpenSSL for both hashlib and ssl, this proposal
>> wouldn't change anything in this regard.
>
> hashlib can still works without OpenSSL and hash algorithms don't
> fall under crypto laws. ssl doesn't work without OpenSSL, but also
> doesn't require adding any crypto code to the stdlib.

This won't change the status quo, as my code simply leverages OpenSSL
rather than being an independent implementation.

> The main point that needs to be addressed is shipping Python
> with crypto code. If OpenSSL is optionally used, we're fine,
> but if we start shipping crypto code, things are more contrived.

As I say, we're doing things exactly how they're already done. Python
would not be shipping any more crypto code with this module than it
already does.

> See http://rechten.uvt.nl/koops/cryptolaw/ for a survey.

I've looked over it before and didn't notice anything glaringly
applicable, outside of the Windows situation. IANAL, though.

> We're hosting the Python software on servers in The Netherlands,
> so have to follow the Wassenaar Arrangement if we include
> crypto code. Fortunately, that agreement includes a clause which
> pretty much exempts open source crypto code from export regulations.

Again, this seems to me something more relevant to the OpenSSL folks than to us.

> However, users of Python downloading installers with crypto software
> would import and use it in their resp. countries and that may get
> them into trouble, so they need to be warned if we decide to
> ship crypto code with Python.

Your suggestion about a warning for Windows downloads seems
appropriate. I'm not sure how much more than that needs to be done,
though.

> They way I understand Geremy's suggestion is to just include a
> wrapper for OpenSSL, so that's fine. The PEP should include a
> mention of the above to argue against putting e.g. pycrypto
> into the stdlib (not because it's poor software, much to the
> contrary, only because it causes lots of problems for our
> users and the developers).

I'll add mention of the concern over export laws, but it's probably
not feasible to get similar security properties out of any
reimplementation that could be crafted in a reasonable time anyway.

As a note, I intend to have prototype code ready at approximately the
same time as the PEP, so, time permitting, you should be able to play
with this before too long.

Geremy Condra

Geremy, could you kindly give a status update? Thanks

On Thu, Aug 26, 2010 at 3:49 PM, Éric Araujo <report@bugs.python.org> wrote:
>
> Éric Araujo <merwok@netwok.org> added the comment:
>
> Geremy, could you kindly give a status update? Thanks

The block and stream cipher parts of the library (RC4, AES, and DES)
are functionally complete. I'm putting the finishing touches on
envelope encryption this week, but would greatly appreciate assistance
in demonstrating the library's capabilities- one person is helping
with AES encryption in ziplib, but other examples would be very
helpful.

Geremy Condra

Thanks for the reply, the situation looks good!

I’m an interested outsider with practically no knowledge of encryption except from a high-level GPG user viewpoint, so I can’t help with tests, but I could give a hand to documentation.

May I recommend using libtomcrypt instead of openssl because of the advertising problem outlined here?

http://bugs.python.org/issue9119

In my opinion, libtomcrypt is easier to use and cleaner. It compiles on Windows without requiring Perl, and is free of the advertising clause in OpenSSL since it is public domain.

http://libtom.org/?page=features&newsitems=5&whatfile=crypt

> May I recommend using libtomcrypt instead of openssl because of the
> advertising problem outlined here?

Changing libraries because of an "advertising problem" doesn't sound
reasonable. The latter is much more easily solved than the former.

Besides, libtomcrypt doesn't seem to provide SSL or TLS support (at
least the Web page you linked to doesn't say so), so OpenSSL would still
be needed for the ssl module.

How about nss?  As a bonus, this would also avoid making more work for Fedora (<http://fedoraproject.org/wiki/FedoraCryptoConsolidation>).

> How about nss?  As a bonus, this would also avoid making more work for
> Fedora (<http://fedoraproject.org/wiki/FedoraCryptoConsolidation>).

Well, similar question: what will it bring and who will do the work? :)
(Fedora perhaps?)

On Fri, 2010-09-17 at 23:11 +0000, Antoine Pitrou wrote:
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> > How about nss?  As a bonus, this would also avoid making more work for
> > Fedora (<http://fedoraproject.org/wiki/FedoraCryptoConsolidation>).
> 
> Well, similar question: what will it bring and who will do the work? :)
> (Fedora perhaps?)

Possibly me - if you'll take my patches :)

What it will bring: APIs which aren't absolutely insane; full SSL support; RSA, DSA, ECDSA, Diffie-Hellman, EC Diffie-Hellman, AES, Triple DES, DES, RC2, RC4, SHA-1, SHA-256, SHA-384, SHA-512, MD2, MD5, HMAC: Common cryptographic algorithms used in public-key and symmetric-key cryptography; simplified FIPS 140 validation; better licensing (MPL).

I'm interested in stuff based on nss, but I definitely won't promise to do the work.  Fortunately dmalcolm seems to be on top of that. :)

I should note that I can't touch anything to do with Elliptic Curve crypto.  I don't know if I can comment on the reasons for that.

> I should note that I can't touch anything to do with Elliptic Curve crypto.  I don't know if I can comment on the reasons for that.

Hopefully anything ECC related can be done separately.  There's certainly no ECC APIs in Python now, so there would be no loss of functionality if any new crypto-related APIs didn't include ECC.  Later if someone is sufficiently interesting, they could add it to the base library.

Le vendredi 17 septembre 2010 à 23:14 +0000, Dave Malcolm a écrit :
> Dave Malcolm <dmalcolm@redhat.com> added the comment:
> 
> On Fri, 2010-09-17 at 23:11 +0000, Antoine Pitrou wrote:
> > Antoine Pitrou <pitrou@free.fr> added the comment:
> > 
> > > How about nss?  As a bonus, this would also avoid making more work for
> > > Fedora (<http://fedoraproject.org/wiki/FedoraCryptoConsolidation>).
> > 
> > Well, similar question: what will it bring and who will do the work? :)
> > (Fedora perhaps?)
> 
> Possibly me - if you'll take my patches :)

Converting/rewriting _ssl.c while remaining compatible is certainly the
most interesting part, so I suggest you start with that if you are
motivated :)
You should also explain the rationale on the mailing-list, so that
interested people have a chance to offer advice.

FWIW, one of my RH colleagues (John Dennis) has written a set of Python bindings for NSS:
  http://fedoraproject.org/wiki/Features/PythonNSS

(Though that seems to me to be a slightly different thing to a general-purpose crypto lib that happens to be written using NSS as an implementation detail)

libtomcrypt is a _great_ library.  That is what hashlib uses for the hash algorithms when OpenSSL is not available.

But the _primary_ reason for using OpenSSL is that it is the defacto open source location for the best architecture specific implementations of any hash and crypto algorithm.  OpenSSL outperforms libtomcrypt by a significant factor (easily 2x) in most cases.

The NSS everywhere effort mentioned in the fedora link sounds interesting.  I support having the ability to link against that instead of OpenSSL or copies of libtomcrypt but I am generally in favor of absolute performance per byte of all algorithms concerned being available.  (ie: don't force hashlib to stop using openssl, just provide an alternative).

On Fri, Sep 17, 2010 at 8:55 PM, Gregory P. Smith
<report@bugs.python.org> wrote:
>
> Gregory P. Smith <greg@krypto.org> added the comment:
>
> libtomcrypt is a _great_ library.  That is what hashlib uses for the hash algorithms when OpenSSL is not available.
>
> But the _primary_ reason for using OpenSSL is that it is the defacto open source location for the best architecture specific implementations of any hash and crypto algorithm.  OpenSSL outperforms libtomcrypt by a significant factor (easily 2x) in most cases.
>
> The NSS everywhere effort mentioned in the fedora link sounds interesting.  I support having the ability to link against that instead of OpenSSL or copies of libtomcrypt but I am generally in favor of absolute performance per byte of all algorithms concerned being available.  (ie: don't force hashlib to stop using openssl, just provide an alternative).

I'm open to working with other libraries, but realistically there
isn't a huge point in supporting every crypto library out there, and
it would be a bad idea all around to try. My suggestion would be that
we stick with OpenSSL until a replacement for _ssl.c exists; after
that we can revisit that decision and see where we stand.

Geremy Condra

> OpenSSL outperforms libtomcrypt by a significant factor (easily 2x) in most cases.

Gregory, do you have any evidence to substantiate this claim? Not that it isn't plausible, but I couldn't find any benchmarks, and here the author of libtomcrypt finds it to be 40% faster than OpenSSL concerning RSA operations.

http://www.adras.com/TomsFastMath-faster.t71-93.html

>  but I am generally in favor of absolute performance per byte of all algorithms concerned being available

Performance isn't all that matters, or else Python would have used GMP, as Guido discussed here:

http://mail.python.org/pipermail/python-3000/2007-September/010329.html

It is also not a convincing argument that new python libraries should use OpenSSL if possible just because that is what _ssl uses. Compiling Python with OpenSSL support has been optional because it puts additional restrictions on the PSF license. Spreading this restriction to the future crypto module (when we have a choice not to) doesn't make sense.

> It is also not a convincing argument that new python libraries should
> use OpenSSL if possible just because that is what _ssl uses. Compiling
> Python with OpenSSL support has been optional because it puts
> additional restrictions on the PSF license. Spreading this restriction
> to the future crypto module (when we have a choice not to) doesn't
> make sense.

It certainly makes more sense than making Python depend on *several*
crypto libraries.
As for the licensing restriction, it doesn't seem to disturb many Python
users. It's the first time I see someone complaining about it.

This isn't meant to discourage any such efforts (I don't care about
which crypto library we use), but any proposal of using another crypto
library than OpenSSL should IMO include a migration proposal for the
_ssl module.

> It certainly makes more sense than making Python depend on *several* crypto libraries.

Since libtomcrypt is public domain, you could incorporate the source into the tree without making it a binary dependency. The same cannot be said for OpenSSL. I certainly wouldn't mind having 1 dependency on NSS, but having 2 modules depend on OpenSSL is a step in the wrong direction.

> As for the licensing restriction, it doesn't seem to disturb many Python users. It's the first time I see someone complaining about it.

It took several years until someone like Marc-Andre Lemburg to find that the Python website might be violating that license. Perhaps the reason is because no one bothers to read licenses carefully. People are probably violating the license without knowing it.

If you take a look at the clause "All advertising materials mentioning features or use of this software must display the following acknowledgment", you will find at least 2 problems. 

One is that if you mention something like "base64" in whatever could be deemed "advertising", you will be subject to this clause because base64 is a feature of OpenSSL, even if you don't use their implementation. Another problem is the difference between the clause "features or use of this software", which is semantically different from "features of this software or use of this software".

Is it worth the risk to depend on Eric Young's proclivity to sue now that he works for RSA and produces competing software called BSAFE? Maybe it is for you, but certainly not for me.

> Since libtomcrypt is public domain, you could incorporate the source
> into the tree without making it a binary dependency.

And then we have to maintain our copy ourselves. I'm not sure why you
think this is better than depending on a system-wide install, because
it's certainly worse.

(we do have private copies of a couple of libraries: zlib, expat,
libffi. The first two are probably for historical reasons (the
system-wide versions are used by default), while the third is because
it's patched)

> I certainly wouldn't mind having 1 dependency on NSS, but having 2
> modules depend on OpenSSL is a step in the wrong direction.

Perhaps you wouldn't mind, but others would (especially packagers;
including ourselves since we build binary packages for Windows and Mac
OS X).

> It took several years until someone like Marc-Andre Lemburg to find
> that the Python website might be violating that license. Perhaps the
> reason is because no one bothers to read licenses carefully. People
> are probably violating the license without knowing it.

The solution to stop violating it is trivial, though: just add the
required mention(s).
Compare that to rewriting a lot of code and making sure it doesn't
change behaviour compared to previous Python versions.

> One is that if you mention something like "base64" in whatever could
> be deemed "advertising", you will be subject to this clause because
> base64 is a feature of OpenSSL, even if you don't use their
> implementation.

Unless "base64" is an OpenSSL trademark, this is FUD.

> The solution to stop violating it is trivial, though: just add the
required mention(s).

That only solves the problem for Python.org. It does not solve the problem for everyone else that has to write "This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/) This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)" in advertisement that mentions features.

> Unless "base64" is an OpenSSL trademark, this is FUD.

The license clearly states: "All advertising materials mentioning features or use of this software". Do you somehow disagree that base64 is a feature of the OpenSSL library?

http://www.openssl.org/docs/crypto/BIO_f_base64.html

This bug has turned into a bikeshed.

Lets stop that please.

I _DON'T_ care about performance when it comes to someone submitting an
actual working implementation of a crypto library for inclusion with the
standard library.  The first priority needs to be a useful stable API.

After that is settled, implementations backed by whatever libraries can be
written by anyone who wants to contribute them.  Lets not stop that from
happening by arguing about who's library is a prettier color.  The first one
may well be implemented in Python itself as a for all I care.

-gps

(comments below are to answer questions but really are not relevant to the
bug)

On Sat, Sep 18, 2010 at 3:54 PM, lorph <report@bugs.python.org> wrote:

>
> lorph <lorph1@gmail.com> added the comment:
>
> > OpenSSL outperforms libtomcrypt by a significant factor (easily 2x) in
> most cases.
>
> Gregory, do you have any evidence to substantiate this claim? Not that it
> isn't plausible, but I couldn't find any benchmarks, and here the author of
> libtomcrypt finds it to be 40% faster than OpenSSL concerning RSA
> operations.
>

I should not have said "most" cases but there are many cases where it does.
 Run your own microbenchmarks on various hardware if you're curious
("openssl speed algorithmname" on the command line is an existing openssl
benchmark).  The important thing to realize is that libtomcrypt is
intentionally written in very portable C.  That is great but it leaves a lot
on the table.  Optimizations for various platforms to take advantage of
enhanced instruction sets such as SSE2 and explicit hardware crypto
acceleration instructions such as
http://software.intel.com/en-us/articles/intel-advanced-encryption-standard-aes-instructions-set/are
not likely to be part of libtomcrypt, nor should they ever be part of
code included with and distributed as part of Python.  That should come from
a dynamic library that is part of the OS.  That may be NSS, that may be
OpenSSL, that may be any number of other things.  Its beyond the scope of
this bug.

I'm sorry for bringing the performance angle up even if it motivate me to
make hashlib happen.

An example benchmark of hashlib using libtomcrypt sha1 vs openssl sha1 is
here http://bugs.python.org/issue1121611#msg47779 if you're curious but
please make any responses to that outside of this bug.

-gps

> The license clearly states: "All advertising materials mentioning
> features or use of this software". Do you somehow disagree that base64
> is a feature of the OpenSSL library?
>
> http://www.openssl.org/docs/crypto/BIO_f_base64.html

That's funny. Do you think that if OpenSSL provided its own implementation of strlen(), every text that mentions strlen() needs to acknowledge OpenSSL? Do you realize how ridiculous that is?

Just another data point for the discussion:

The PSF is currently funding the effort to port pyOpenSSL to
Python 3.x and the port is nearly finished.

It may be worthwhile investigating adding the EVP interface
from evpy (with the ctypes bindings converted to real C wrappers)
to pyOpenSSL and then adding the pyOpenSSL package to the stdlib.

Note that going for other crypto libs than OpenSSL is currently
not an option, since those are not widely available on the OSes and
we cannot easily add crypto code itself to Python's stdlib due to the
issues with crypto import/export/use restrictions which would
limit the use of Python in various countries.

> It may be worthwhile investigating adding the EVP interface
> from evpy (with the ctypes bindings converted to real C wrappers)
> to pyOpenSSL and then adding the pyOpenSSL package to the stdlib.

pyOpenSSL being LGPL'ed, I'm not sure this is possible.
On the other hand, gradually adding some of pyOpenSSL's most useful functionalities to the ssl module would be worthwhile, and I know Jean-Paul would be interested in this.
This has already begun in 3.2, but I've been alone in doing it and it would be nice if other people contributed:
http://docs.python.org/dev/library/ssl.html#ssl-contexts

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>> It may be worthwhile investigating adding the EVP interface
>> from evpy (with the ctypes bindings converted to real C wrappers)
>> to pyOpenSSL and then adding the pyOpenSSL package to the stdlib.
> 
> pyOpenSSL being LGPL'ed, I'm not sure this is possible.

Changing the license should be possible, since we know the
copyright owners and both are PSF members (AB Strakt and Jean-Paul).

> On the other hand, gradually adding some of pyOpenSSL's most useful functionalities to the ssl module would be worthwhile, and I know Jean-Paul would be interested in this.
> This has already begun in 3.2, but I've been alone in doing it and it would be nice if other people contributed:
> http://docs.python.org/dev/library/ssl.html#ssl-contexts

I don't think we should redo this effort in the context
of the ssl module.

pyOpenSSL is stable, in production use and
has a decent API. The ssl module is good enough for HTTPS client
use. pyOpenSSL provides a robust server side implementation with
all the required certificate and context handling needed for this.

We could tell people to use the ssl module for clients and
pyOpenSSL for the server side and perhaps integrate the OpenSSL
package into the ssl namespace.

> pyOpenSSL is stable, in production use and
> has a decent API. The ssl module is good enough for HTTPS client
> use. pyOpenSSL provides a robust server side implementation with
> all the required certificate and context handling needed for this.
> 
> We could tell people to use the ssl module for clients and
> pyOpenSSL for the server side and perhaps integrate the OpenSSL
> package into the ssl namespace.

In this case, this should be decided early, so that I know if I should
continue caring about the ssl module or not. I'm not interested in
maintaining potentially obsolete code.

>> Unless "base64" is an OpenSSL trademark, this is FUD.
> 
> The license clearly states: "All advertising materials mentioning
> features or use of this software". Do you somehow disagree that
> base64 is a feature of the OpenSSL library?

What specific "advertising material" mentions "base64" but fails to
mention "OpenSSL"? In any case, this is off-topic for this issue.

> Do you think that if OpenSSL provided its own implementation of strlen(), every text that mentions strlen() needs to acknowledge OpenSSL? Do you realize how ridiculous that is?

If that text is deemed to be advertising by Eric Young and a court of law, then absolutely yes. The license is short, clear, and does not make any exceptions for features that you might deem to be commonplace.

http://www.openssl.org/source/license.html

I also agree that this is ridiculous, but believing something is ridiculous does not make it any less real as Barnes and Nobles learned the hard way by using Amazon's one-click patent.

> The important thing to realize is that libtomcrypt is
intentionally written in very portable C.  That is great but it leaves a lot on the table. Optimizations for various platforms to take advantage of enhanced instruction sets such as SSE2 and explicit hardware crypto acceleration instructions such as [...] are not likely to be part of libtomcrypt, 

A quick glance at libtomcrypt tells me otherwise. It is portable C, but it still has inline assembler. It has optimizations using SSE2, x86, x86_64, and PPC32. Even though it might not have that new Intel AES instruction yet, this is the same argument people had for using GMP for python's integers.

If there is a problem with multiple libraries, I'd like to reiterate my support for migrating to NSS.

> If there is a problem with multiple libraries, I'd like to reiterate
> my support for migrating to NSS.

Will your support go so far as to investigate feasibility and provide a
patch?

On Mon, Sep 20, 2010 at 3:44 PM, lorph <report@bugs.python.org> wrote:
>
> lorph <lorph1@gmail.com> added the comment:
>
>> Do you think that if OpenSSL provided its own implementation of strlen(), every text that mentions strlen() needs to acknowledge OpenSSL? Do you realize how ridiculous that is?
>
> If that text is deemed to be advertising by Eric Young and a court of law, then absolutely yes. The license is short, clear, and does not make any exceptions for features that you might deem to be commonplace.
>
> http://www.openssl.org/source/license.html
>
> I also agree that this is ridiculous, but believing something is ridiculous does not make it any less real as Barnes and Nobles learned the hard way by using Amazon's one-click patent.

I get called paranoid a lot, but I don't find this terribly
convincing. Having said that, I'm happy to have your help in making
sure that the interface we create is agnostic with respect to openssl,
libtomcrypt, NSS, etc. That would mean that if Eric Young did decide
to try and force his hand we would be able to switch away the very
next day with no muss and no fuss. Are you willing to provide that
help?

>> The important thing to realize is that libtomcrypt is
> intentionally written in very portable C.  That is great but it leaves a lot on the table. Optimizations for various platforms to take advantage of enhanced instruction sets such as SSE2 and explicit hardware crypto acceleration instructions such as [...] are not likely to be part of libtomcrypt,
>
> A quick glance at libtomcrypt tells me otherwise. It is portable C, but it still has inline assembler. It has optimizations using SSE2, x86, x86_64, and PPC32. Even though it might not have that new Intel AES instruction yet, this is the same argument people had for using GMP for python's integers.

I'm not worried about speed. In fact, if fast means "vulnerable to
side-channel cryptanalysis" I'm firmly opposed to it, and I don't know
if that's the case here. OpenSSL has at least been subject to
significant attention in that regard.

> If there is a problem with multiple libraries, I'd like to reiterate my support for migrating to NSS.

I'm happy to do this work if you (or someone else) will step forward
and help me identify and fix compatibility concerns. Otherwise, this
is just a waste of time all around.

Geremy Condra

> If that text is deemed to be advertising by Eric Young and a court of law

The license of a software product cannot affect software that is not even aware of that said product.  (A patent, or a trademark can.)  It governs the use of that product, not of others.  "Features of this library" means really features of *that* library, as specifically implemented in that library.

If a license for one specific software product could affect all other software with such a primitive clause, I don't want to think how Richard Stallman would have worded the GPL :)

> The license of a software product cannot affect software that is not even aware of that said product.

I never claimed that the clause triggered for all software in existence. We are talking about OpenSSL being bundled with Python where Python is very much aware of OpenSSL. Provided the following 3 circumstances are met, the advertisement clause applies:

1. You are distributing Python with OpenSSL
2. You are "advertising".
3. Your advertising mentions features.

By mentioning features of Python, or even a feature of OpenSSL you don't even use (base64), by the wording of the license you are obligated to also advertise OpenSSL and Eric Young. This obviously has a chilling effect on the distribution and advertising of Python apps. Think about the 100 char blurb on every small web banner.

> In fact, if fast means "vulnerable to side-channel cryptanalysis" I'm firmly opposed to it, and I don't know if that's the case here. OpenSSL has at least been subject to significant attention in that regard.

LTC does address side-channel attacks, but this is a moot point since by using a high level language like Python, you are vulnerable to memory scanning since you cannot normally zero out Python strings (something you may wish to consider in the crypto API).

I'd also add that the "rounds" option should be left in for compatibility reasons. For easy usage, a default such as CBC could be specified. Otherwise, I don't think there is anything wrong with the API.

> I never claimed that the clause triggered for all software in
> existence. We are talking about OpenSSL being bundled with Python
> where Python is very much aware of OpenSSL. Provided the following 3
> circumstances are met, the advertisement clause applies:

Can we please stop getting license interpretations from laymen on
a bug tracker? If somebody is really concerned that the PSF might
violate some license, the PSF lawyer should be asked to evaluate
the situation.

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>> pyOpenSSL is stable, in production use and
>> has a decent API. The ssl module is good enough for HTTPS client
>> use. pyOpenSSL provides a robust server side implementation with
>> all the required certificate and context handling needed for this.
>>
>> We could tell people to use the ssl module for clients and
>> pyOpenSSL for the server side and perhaps integrate the OpenSSL
>> package into the ssl namespace.
> 
> In this case, this should be decided early, so that I know if I should
> continue caring about the ssl module or not. I'm not interested in
> maintaining potentially obsolete code.

I'll ask Jean-Paul and AB Strakt if they are up to contributing
the pyOpenSSL code to the Python stdlib based on a contributor
agreement. This would enable us to relicense the code undert
the PSF license even if the original code's license is not
changed.

Once that's a done deal, we can then consider moving in the above
direction.

On Tue, Sep 21, 2010 at 4:04 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>
> Antoine Pitrou wrote:
>>
>> Antoine Pitrou <pitrou@free.fr> added the comment:
>>
>>> pyOpenSSL is stable, in production use and
>>> has a decent API. The ssl module is good enough for HTTPS client
>>> use. pyOpenSSL provides a robust server side implementation with
>>> all the required certificate and context handling needed for this.
>>>
>>> We could tell people to use the ssl module for clients and
>>> pyOpenSSL for the server side and perhaps integrate the OpenSSL
>>> package into the ssl namespace.
>>
>> In this case, this should be decided early, so that I know if I should
>> continue caring about the ssl module or not. I'm not interested in
>> maintaining potentially obsolete code.
>
> I'll ask Jean-Paul and AB Strakt if they are up to contributing
> the pyOpenSSL code to the Python stdlib based on a contributor
> agreement. This would enable us to relicense the code undert
> the PSF license even if the original code's license is not
> changed.
>
> Once that's a done deal, we can then consider moving in the above
> direction.

I'm not sure I understand the relevance of pyopenssl here- it's pretty
clearly focused on SSL/TLS rather than on crypto. Maybe someone can
clarify?

Geremy Condra

geremy condra wrote:
> 
> geremy condra <debatem1@gmail.com> added the comment:
> 
> On Tue, Sep 21, 2010 at 4:04 AM, Marc-Andre Lemburg
> <report@bugs.python.org> wrote:
>>
>> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>>
>> Antoine Pitrou wrote:
>>>
>>> Antoine Pitrou <pitrou@free.fr> added the comment:
>>>
>>>> pyOpenSSL is stable, in production use and
>>>> has a decent API. The ssl module is good enough for HTTPS client
>>>> use. pyOpenSSL provides a robust server side implementation with
>>>> all the required certificate and context handling needed for this.
>>>>
>>>> We could tell people to use the ssl module for clients and
>>>> pyOpenSSL for the server side and perhaps integrate the OpenSSL
>>>> package into the ssl namespace.
>>>
>>> In this case, this should be decided early, so that I know if I should
>>> continue caring about the ssl module or not. I'm not interested in
>>> maintaining potentially obsolete code.
>>
>> I'll ask Jean-Paul and AB Strakt if they are up to contributing
>> the pyOpenSSL code to the Python stdlib based on a contributor
>> agreement. This would enable us to relicense the code under
>> the PSF license even if the original code's license is not
>> changed.
>>
>> Once that's a done deal, we can then consider moving in the above
>> direction.
> 
> I'm not sure I understand the relevance of pyopenssl here- it's pretty
> clearly focused on SSL/TLS rather than on crypto. Maybe someone can
> clarify?

Yes, but it provides a decent platform for adding other crypto APIs
as well and then we could have these as C APIs rather than wrappers
using ctypes.

There's already a patch available from Keyphrene adding all those bits:
http://www.keyphrene.com/products/pyOpenSSL-extended/index.php?lng=en

The patch would need to be updated for the new pyOpenSSL version,
but that's certainly within range. And we'd need to get their permission
to relicense as well.

On Tue, Sep 21, 2010 at 10:33 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>
> geremy condra wrote:
>>
>> geremy condra <debatem1@gmail.com> added the comment:
>>
>> On Tue, Sep 21, 2010 at 4:04 AM, Marc-Andre Lemburg
>> <report@bugs.python.org> wrote:
>>>
>>> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>>>
>>> Antoine Pitrou wrote:
>>>>
>>>> Antoine Pitrou <pitrou@free.fr> added the comment:
>>>>
>>>>> pyOpenSSL is stable, in production use and
>>>>> has a decent API. The ssl module is good enough for HTTPS client
>>>>> use. pyOpenSSL provides a robust server side implementation with
>>>>> all the required certificate and context handling needed for this.
>>>>>
>>>>> We could tell people to use the ssl module for clients and
>>>>> pyOpenSSL for the server side and perhaps integrate the OpenSSL
>>>>> package into the ssl namespace.
>>>>
>>>> In this case, this should be decided early, so that I know if I should
>>>> continue caring about the ssl module or not. I'm not interested in
>>>> maintaining potentially obsolete code.
>>>
>>> I'll ask Jean-Paul and AB Strakt if they are up to contributing
>>> the pyOpenSSL code to the Python stdlib based on a contributor
>>> agreement. This would enable us to relicense the code under
>>> the PSF license even if the original code's license is not
>>> changed.
>>>
>>> Once that's a done deal, we can then consider moving in the above
>>> direction.
>>
>> I'm not sure I understand the relevance of pyopenssl here- it's pretty
>> clearly focused on SSL/TLS rather than on crypto. Maybe someone can
>> clarify?
>
> Yes, but it provides a decent platform for adding other crypto APIs
> as well and then we could have these as C APIs rather than wrappers
> using ctypes.

The intention all along has been that we use the C API, and in fact
I'm pretty far along on writing that. Assuming there won't be an issue
with porting pyopenssl to python3, this seems like a pretty good idea
to me though.

> There's already a patch available from Keyphrene adding all those bits:
> http://www.keyphrene.com/products/pyOpenSSL-extended/index.php?lng=en
>
> The patch would need to be updated for the new pyOpenSSL version,
> but that's certainly within range. And we'd need to get their permission
> to relicense as well.

Bits and pieces of this look useful but it also looks like I'd be
rewriting a lot of it to move away from the RSA_* routines, etc. I
suspect it would take more time to get it into line than to just
cherrypick out of it.

Geremy Condra

geremy condra wrote:
> 
>>>> I'll ask Jean-Paul and AB Strakt if they are up to contributing
>>>> the pyOpenSSL code to the Python stdlib based on a contributor
>>>> agreement. This would enable us to relicense the code under
>>>> the PSF license even if the original code's license is not
>>>> changed.
>>>>
>>>> Once that's a done deal, we can then consider moving in the above
>>>> direction.
>>>
>>> I'm not sure I understand the relevance of pyopenssl here- it's pretty
>>> clearly focused on SSL/TLS rather than on crypto. Maybe someone can
>>> clarify?
>>
>> Yes, but it provides a decent platform for adding other crypto APIs
>> as well and then we could have these as C APIs rather than wrappers
>> using ctypes.
> 
> The intention all along has been that we use the C API, and in fact
> I'm pretty far along on writing that. Assuming there won't be an issue
> with porting pyopenssl to python3, this seems like a pretty good idea
> to me though.

Ah, sorry, I must have missed that turn in the discussion :-)

The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul
is planning for an alpha release next month.

>> There's already a patch available from Keyphrene adding all those bits:
>> http://www.keyphrene.com/products/pyOpenSSL-extended/index.php?lng=en
>>
>> The patch would need to be updated for the new pyOpenSSL version,
>> but that's certainly within range. And we'd need to get their permission
>> to relicense as well.
> 
> Bits and pieces of this look useful but it also looks like I'd be
> rewriting a lot of it to move away from the RSA_* routines, etc. I
> suspect it would take more time to get it into line than to just
> cherrypick out of it.

If you are writing new code for this anyway, it may be better to
avoid the license question of the Keyphrene patch and just use
their code as reference.

On Tue, Sep 21, 2010 at 11:29 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>
> geremy condra wrote:
>>
>>>>> I'll ask Jean-Paul and AB Strakt if they are up to contributing
>>>>> the pyOpenSSL code to the Python stdlib based on a contributor
>>>>> agreement. This would enable us to relicense the code under
>>>>> the PSF license even if the original code's license is not
>>>>> changed.
>>>>>
>>>>> Once that's a done deal, we can then consider moving in the above
>>>>> direction.
>>>>
>>>> I'm not sure I understand the relevance of pyopenssl here- it's pretty
>>>> clearly focused on SSL/TLS rather than on crypto. Maybe someone can
>>>> clarify?
>>>
>>> Yes, but it provides a decent platform for adding other crypto APIs
>>> as well and then we could have these as C APIs rather than wrappers
>>> using ctypes.
>>
>> The intention all along has been that we use the C API, and in fact
>> I'm pretty far along on writing that. Assuming there won't be an issue
>> with porting pyopenssl to python3, this seems like a pretty good idea
>> to me though.
>
> Ah, sorry, I must have missed that turn in the discussion :-)
>
> The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul
> is planning for an alpha release next month.

Do you know if he's looking for help with that? There's been some talk of
a porting sprint here and I'd be happy to put that at the top of the list.

>>> There's already a patch available from Keyphrene adding all those bits:
>>> http://www.keyphrene.com/products/pyOpenSSL-extended/index.php?lng=en
>>>
>>> The patch would need to be updated for the new pyOpenSSL version,
>>> but that's certainly within range. And we'd need to get their permission
>>> to relicense as well.
>>
>> Bits and pieces of this look useful but it also looks like I'd be
>> rewriting a lot of it to move away from the RSA_* routines, etc. I
>> suspect it would take more time to get it into line than to just
>> cherrypick out of it.
>
> If you are writing new code for this anyway, it may be better to
> avoid the license question of the Keyphrene patch and just use
> their code as reference.

Yeah, I think that makes the most sense.

Geremy Condra

geremy condra wrote:
> 
>>> The intention all along has been that we use the C API, and in fact
>>> I'm pretty far along on writing that. Assuming there won't be an issue
>>> with porting pyopenssl to python3, this seems like a pretty good idea
>>> to me though.
>>
>> Ah, sorry, I must have missed that turn in the discussion :-)
>>
>> The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul
>> is planning for an alpha release next month.
> 
> Do you know if he's looking for help with that? There's been some talk of
> a porting sprint here and I'd be happy to put that at the top of the list.

I don't know. You might want to contact him directly.

>>> The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul
>>> is planning for an alpha release next month.
>>
>> Do you know if he's looking for help with that? There's been some talk of
>> a porting sprint here and I'd be happy to put that at the top of the list.
>
> I don't know. You might want to contact him directly.

Sent.

By the way, if you've used the keyphrene API and wouldn't mind sharing
how it compares with the proposed crypto API I'd appreciate it, I
don't think it came up in any of the initial conversations about this.

Geremy Condra

geremy condra wrote:
> 
> geremy condra <debatem1@gmail.com> added the comment:
> 
>>>> The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul
>>>> is planning for an alpha release next month.
>>>
>>> Do you know if he's looking for help with that? There's been some talk of
>>> a porting sprint here and I'd be happy to put that at the top of the list.
>>
>> I don't know. You might want to contact him directly.
> 
> Sent.
> 
> By the way, if you've used the keyphrene API and wouldn't mind sharing
> how it compares with the proposed crypto API I'd appreciate it, I
> don't think it came up in any of the initial conversations about this.

No, I haven't used the APIs. I do have some experience with mxCrypto,
though, which is a low-level wrapper for the ciphers and hashes in
OpenSSL:

    http://www.egenix.com/www2002/python/mxCrypto.html

I've been in touch with the copyright holders of pyOpenSSL and they all were positive about contributing the code to the PSF under a contributor agreement.

The idea would then be to add the crypto routines to pyOpenSSL and have that added to the stdlib as say openssl package.

So how should we go about this ? Open a new ticket ?

> I've been in touch with the copyright holders of pyOpenSSL and they
> all were positive about contributing the code to the PSF under a
> contributor agreement.
> So how should we go about this ? Open a new ticket ?

I would like to see public discussion about this, especially in the
light of the ssl module improvements in Python 3.2. It is not obvious
that duplicate APIs in the stdlib are a good idea, especially when they
are not compatible with each other. It also means that the current
pyOpenSSL maintainer (Jean-Paul) should agree to do maintenance directly
in the stdlib rather than in a separate repo.

> The idea would then be to add the crypto routines to pyOpenSSL and
> have that added to the stdlib as say openssl package.

This sounds a bit ridiculous. Why not add the crypto routines directly
to the stdlib?

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>> I've been in touch with the copyright holders of pyOpenSSL and they
>> all were positive about contributing the code to the PSF under a
>> contributor agreement.
>> So how should we go about this ? Open a new ticket ?
> 
> I would like to see public discussion about this, especially in the
> light of the ssl module improvements in Python 3.2. It is not obvious
> that duplicate APIs in the stdlib are a good idea, especially when they
> are not compatible with each other. It also means that the current
> pyOpenSSL maintainer (Jean-Paul) should agree to do maintenance directly
> in the stdlib rather than in a separate repo.
> 
>> The idea would then be to add the crypto routines to pyOpenSSL and
>> have that added to the stdlib as say openssl package.
> 
> This sounds a bit ridiculous. Why not add the crypto routines directly
> to the stdlib?

To make those routines available to a broader audience and to
get more user feedback.

I don't think we can add pyOpenSSL to Python 3.2, so it's better
to use the available time to hash out the details outside the
stdlib. Once it's in the stdlib, changing APIs is very difficult.

> > This sounds a bit ridiculous. Why not add the crypto routines directly
> > to the stdlib?
> 
> To make those routines available to a broader audience and to
> get more user feedback.

Sure. But it can be any standalone package, not necessarily pyOpenSSL.
Then, if we want to add them to the stdlib, we don't have to pull in the
whole pyOpenSSL package.

> I don't think we can add pyOpenSSL to Python 3.2,

Right, it's too late.

> so it's better
> to use the available time to hash out the details outside the
> stdlib. Once it's in the stdlib, changing APIs is very difficult.

Then I think the discussion about API and process should move to
python-ideas.

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>>> This sounds a bit ridiculous. Why not add the crypto routines directly
>>> to the stdlib?
>>
>> To make those routines available to a broader audience and to
>> get more user feedback.
> 
> Sure. But it can be any standalone package, not necessarily pyOpenSSL.
> Then, if we want to add them to the stdlib, we don't have to pull in the
> whole pyOpenSSL package.

pyOpenSSL has the advantage of already providing all the other
bits and pieces needed to interface and build against OpenSSL,
so it's a good ecosystem for such a development.

Besides there are already patches available which do add the
ciphers and hashs to pyOpenSSL, so the development could be
sped up by using those as references.

>> I don't think we can add pyOpenSSL to Python 3.2,
> 
> Right, it's too late.
> 
>> so it's better
>> to use the available time to hash out the details outside the
>> stdlib. Once it's in the stdlib, changing APIs is very difficult.
> 
> Then I think the discussion about API and process should move to
> python-ideas.

The APIs should probably be discussed on the Python crypto or
pyOpenSSL list and the discussion about its integration into the
stdlib on either the python-dev or the stdlib list.

https://lists.sourceforge.net/lists/listinfo/pyopenssl-list
http://mail.python.org/mailman/listinfo/python-crypto
http://mail.python.org/mailman/listinfo/python-dev
http://mail.python.org/mailman/listinfo/stdlib-sig

python-ideas is not really meant for such discussions.

> The APIs should probably be discussed on the Python crypto or
> pyOpenSSL list and the discussion about its integration into the
> stdlib on either the python-dev or the stdlib list.

If the target goal is stdlib inclusion, the APIs should be discussed on
python-ideas too.

> python-ideas is not really meant for such discussions.

Yes, it is.

> Besides there are already patches available which do add the
> ciphers and hashs to pyOpenSSL, so the development could be
> sped up by using those as references.

I don't think that's the case. I admit that development on this has
been very slow (curse you, Real Life!) but it is nearing a release,
and I doubt that this would save any time even if I didn't have to
rewrite half of the keyphrene code.

Geremy Condra

It sounds like you may already have an idea of how you want the API structured, but just in case you're still thinking about it here's another API to look at that I think focuses on exactly what you were highlighting as priorities (sane defaults, easy to use): keyczar.

I use keyczar quite a bit and really like it, although I tend to only use the key classes directly instead of using the generic "Crypter" etc. classes.

http://code.google.com/p/keyczar/

This issue may have been dead for 3+ years, but perhaps it's time its brought back to the surface.  Aside from simple being convenient for general security practices, a stdlib module for crypto routines would enable python to handle encrypted zipfiles and resolve issues like issue9170. Currently, only hashlib and hmac are available to users (see http://docs.python.org/3/library/crypto.html).  

I'd imagine that collaboration with the likes of Dwayne from PyCrypto or further collaboration with Geremy from Evpy would be possible, and perhaps the stdlib could build on a 3rd party crypto library.  Is anyone else interested in working to make this happen (or does anyone have a good argument against a stdlib crypto library)?  I imagine it would be difficult to maintain, but perhaps its at least worth giving this issue a second chance.

Madison May wrote:
> This issue may have been dead for 3+ years, but perhaps it's time its brought back to the surface.  Aside from simple being convenient for general security practices, a stdlib module for crypto routines would enable python to handle encrypted zipfiles and resolve issues like issue9170. Currently, only hashlib and hmac are available to users (see http://docs.python.org/3/library/crypto.html).  
> 
> I'd imagine that collaboration with the likes of Dwayne from PyCrypto or further collaboration with Geremy from Evpy would be possible, and perhaps the stdlib could build on a 3rd party crypto library.  Is anyone else interested in working to make this happen (or does anyone have a good argument against a stdlib crypto library)?  I imagine it would be difficult to maintain, but perhaps its at least worth giving this issue a second chance.

IMO, it's better to have crypto routines not be part of the stdlib.
Not because there's a technical reason. Making crypto code part of
the stdlib would create legal issues, e.g. make it illegal to use
in some parts of the world.

See http://www.cryptolaw.org/ for details.

Yeah, that definitely qualifies as a good argument.  I didn't consider the legal issues that would create.  Let's let this issue rest in peace, then.

The interest is there but nobody is designing or implementing it. Realistically, make a module sporting the proposed API and put it up on pypi and after success there: raise the issue on python-ideas.

Read the entire thread here first before designing such a thing.

We will always have unsolvable problems in any crypto shipped with Python such as side channel attacks. Those are outside the scope of what the CPython project should ever attempt to solve. Document them as weaknesses and suggest people use C libraries dedicated to avoiding those problems if their security needs require it.

Python already ships with SSL support so the legal ship might have sailed ages ago if thats what PSF lawyers say. (IANAL)