classification
Title: Support LibreSSL (instead of OpenSSL): make RAND_egd optional
Type: Stage:
Components: Build Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Edd.Barrett, christian.heimes, dstufft, flox, giampaolo.rodola, janssen, koobs, mgorny, oberstet, pitrou, polymorphm, python-dev, rpointel, spil, vstinner
Priority: normal Keywords:

Created on 2014-04-26 14:21 by Edd.Barrett, last changed 2015-01-06 13:01 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
patch-Lib_ssl.py spil, 2014-11-28 08:47 Make RAND_egd support automatic
patch-Modules__ssl.c spil, 2014-11-28 08:47 Make RAND_egd support automatic
patch-configure.ac spil, 2014-11-28 09:56 Make RAND_egd support automatic
test_ssl.log spil, 2014-11-28 13:29 Output of test_ssl.py
Messages (25)
msg217198 - (view) Author: Edd Barrett (Edd.Barrett) Date: 2014-04-26 14:21
Hi,

I'm sure you have heard about OpenBSD's LibreSSL fork of OpenSSL. There has been a lot of code reorganisation and removal. One function which was removed `RAND_egd()` breaks the CPython build. CPython no longer builds on OpenBSD.

I have submitted a patch against PyPy already. The application library part of the change can probably be re-used since PyPy borrows CPython's application-level standard library (including the `ssl` and `socket` module). However, for the interpreter level change, the build system will probably have to be hacked. We need to check for the existence of `RAND_egd()` at configure time and only build in support if the function is found.

The PyPy patch (and some discussion) is here:
https://bitbucket.org/pypy/pypy/pull-request/233/fix-translation-for-libressl-and-fix-ssl/diff#comment-1744605

I may have a go at doing this myself (for Python-2.7 at least) if no-one steps up in the meantime; for now just making the CPython devs aware.

Thanks
msg217199 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-26 14:48
This should wait until the LibreSSL API stabilizes.

Regardless, I think we should consider deprecating RAND_egd(). The Entropy Gathering Daemon doesn't seem to have seen a release for more than 10 years... (http://sourceforge.net/projects/egd/files/)
msg226355 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-09-04 10:36
> The PyPy patch (and some discussion) is here:

Your patch checks at runtime if libssl comes with RAND_egd:

   HAVE_OPENSSL_RAND_EGD = rffi_platform.Has('RAND_egd')

In CPython, the _ssl module is compiled in C. How can we check if libssl provides RAND_egd() or not at compile time?

Is there a way to check if libssl is OpenSSL or LibreSSL?
msg226357 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-09-04 10:44
Related discussion:
http://marc.info/?l=openbsd-tech&m=140512043210089&w=2

The answer for Python is:
"your package maintainers and ask them to configure these software without egd support."
msg226819 - (view) Author: Michał Górny (mgorny) * Date: 2014-09-12 13:50
> In CPython, the _ssl module is compiled in C. How can we check if libssl provides RAND_egd() or not at compile time?

How about... checking whether the function is provided? Unless I'm missing some major point, AC_CHECK_FUNC should be good enough.

> Is there a way to check if libssl is OpenSSL or LibreSSL?

Why would you want to do that? Do you want to make silly assumptions on API depending on provider name, and then add extra conditionals for versions?
msg226832 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-12 18:05
> Unless I'm missing some major point, AC_CHECK_FUNC should be good enough.

Building extension modules such as ssl doesn't involve autoconf.

> Do you want to make silly assumptions on API depending on provider name, and then add extra conditionals for versions?

Arguably it would be better if LibreSSL exposed the same API as OpenSSL. We're not responsible for the discrepancy here.
msg231426 - (view) Author: Bernard Spil (spil) * Date: 2014-11-20 12:15
EGD was only necessary for some commercial UNIX systems, versions that needed it all reached end of life. It no longer makes sense to have any code referring to it.

        EGD needed until        OS release date
IRIX	6.5.19			feb 2003
Solaris 2.6                     jul 1997
AIX     5.2                     oct 2002
Tru64   5.1B                    sep 2002
HP-UX   11i v2                  sep 2003

Please check OpenBSD's patches to remove EGD support from Python for many versions.
http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/lang/python/2.7/patches/patch-Lib_ssl_py
http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/lang/python/3.4/patches/patch-Lib_ssl_py
http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/lang/python/3.4/patches/patch-Lib_ssl_py

Alternatively see Gentoo's LibreSSL changes https://github.com/Sp1l/libressl/tree/master/dev-lang/python
msg231428 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-11-20 13:23
We don't drop feature in minor releases, we are working hard to maintain the backward compatibility.

We may only disable RAND_egd if Python is compiled/linked to LibreSSL. So the check should probably be dynamic.
msg231462 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-11-21 01:08
We're still willing to fix this if someone tells us how to test for LibreSSL in C code.
msg231471 - (view) Author: Bernard Spil (spil) * Date: 2014-11-21 10:47
Hi, 

I think this can be found in LibreSSL's opensslv.h 
An ifdef LIBRESSL_VERSION_NUMBER should work

See https://github.com/libressl-portable/openbsd/blob/master/src/lib/libssl/src/crypto/opensslv.h

_ssl.c includes crypto.h which in turn includes opensslv.h so checking for LIBRESSL_VERSION_NUMBER should provide the correct check.

Attached patch does this in C whereas it should be checked for in configure and disabled with a HAS_RAND_egd
Have not figured out how to do this conditionally in Lib/ssl.py yet
msg231797 - (view) Author: Bernard Spil (spil) * Date: 2014-11-28 08:46
When configure is called with correct LDFLAGS and CPPFLAGS for LibreSSL these patches to configure, Modules/_ssl.c and Lib/_ssl.py will detect not having RAND_egd support in OpenSSL and make the build succeed.
msg231798 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-11-28 08:49
patch-configure.ac:
-AC_DEFINE(__BSD_VISIBLE, 1, [Define on FreeBSD to activate all library features])

Why do you remove this define?
msg231799 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-11-28 08:52
I thikn RAND_egd() should probably raise NotImplementedError if the function isn't exposed by the ssl library.
msg231801 - (view) Author: Bernard Spil (spil) * Date: 2014-11-28 09:50
Victor: That is a change that has been implemented in the downstream port to fix wxPython, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192365 this ended up in this patch as my primary objective was to fix it for the FreeBSD port.

Antoine: Sorry, I'm not a python dev... I'm willing to do the work if you can provide the guidance... This was merely a "works-for-me(TM)" patch. Since nothing actually uses egd any longer I would not spend to much effort on it. The odds of anyone requiring EGD support _and_ using LibreSSL are negligable. EGD is last centuries technology, there's no sense in mixing that with current tech.
msg231802 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-11-28 09:52
> Victor: That is a change that has been implemented in the downstream port to fix wxPython, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192365 this ended up in this patch as my primary objective was to fix it for the FreeBSD port.

It looks unrelated to LibreSSL, please split your patch in two parts and open a new issue for the wxPython fix.
msg231803 - (view) Author: Bernard Spil (spil) * Date: 2014-11-28 09:56
Remove https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192365 patch from this patch-set
msg231804 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-11-28 11:04
> I thikn RAND_egd() should probably raise NotImplementedError if the function isn't exposed by the ssl library.

I would prefer to follow the model of the os module: don't declare a function if it is not supported by the OS.
msg231807 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-11-28 12:04
> I would prefer to follow the model of the os module: don't declare a function if it is not supported by the OS.

I don't have any strong feelings, so let's do it like that. RAND_egd() isn't useful anyway.
msg231808 - (view) Author: Roundup Robot (python-dev) Date: 2014-11-28 12:31
New changeset 6f23bc5d480e by Victor Stinner in branch 'default':
Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The
https://hg.python.org/cpython/rev/6f23bc5d480e
msg231809 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-11-28 12:36
Ok, here is a first commit to try to support LibreSSL in Python 3.5.

Can someone please test to compile Python 3.5 with LibreSSL and run the test suite (at least test_ssl) to check that everything is fine? If you confirm that the change is correct, I will backport it to Python 2.7 and 3.4. Please mention your version of LibreSSL, OS and OS version in your feedback. LibreSSL has different releases: 2.0 to 2.1.1. Which one was embeded in OpenBSD 5.6?
http://www.libressl.org/

Bernard Spil's patches don't apply on Python 3.5, I guess that they were written for Python 2.7. I also fixed test_ssl.
msg231812 - (view) Author: Bernard Spil (spil) * Date: 2014-11-28 13:29
FAILED (failures=2, errors=2, skipped=5)
That is OK, as these 2 tests should fail with LibreSSL since SSLv2 and SSLv3 support has been removed from LibreSSL.

ERROR: test_protocol_sslv23 (__main__.ThreadedTests)
ERROR: test_protocol_sslv3 (__main__.ThreadedTests)
msg231814 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-11-28 13:33
> That is OK, as these 2 tests should fail with LibreSSL since SSLv2 and SSLv3 support has been removed from LibreSSL.

See the issue #22935.

I prefer to wait until this issue is fixed in Python 3.5, and that test_ssl pass on your PC, before backporting this change into Python 2.7 & 3.4.
msg231839 - (view) Author: Bernard Spil (spil) * Date: 2014-11-28 23:26
Merged the patch from haypo back into the FreeBSD port for 2.7 at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192511
In the process I discovered during test_ssl that I had to patch Lib/socket.py as well to make RAND_egd conditional
msg233535 - (view) Author: Roundup Robot (python-dev) Date: 2015-01-06 13:01
New changeset eddcb6671a48 by Victor Stinner in branch '2.7':
Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The
https://hg.python.org/cpython/rev/eddcb6671a48

New changeset 7f82f50fdad0 by Victor Stinner in branch '3.4':
Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The
https://hg.python.org/cpython/rev/7f82f50fdad0
msg233537 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-01-06 13:01
Ok, Python 2.7, 3.4 and 3.5 can now be *compiled* with LibreSSL.

There are still issues with LibreSSL: see the new issue #23177.
History
Date User Action Args
2015-01-06 13:01:58vstinnersetstatus: open -> closed
resolution: fixed
2015-01-06 13:01:52vstinnersetmessages: + msg233537
2015-01-06 13:01:08python-devsetmessages: + msg233535
2014-11-28 23:26:06spilsetmessages: + msg231839
2014-11-28 13:33:05vstinnersetmessages: + msg231814
2014-11-28 13:29:26spilsetfiles: + test_ssl.log

messages: + msg231812
2014-11-28 12:36:06vstinnersetmessages: + msg231809
2014-11-28 12:31:13python-devsetnosy: + python-dev
messages: + msg231808
2014-11-28 12:04:10pitrousetmessages: + msg231807
2014-11-28 11:04:43vstinnersetmessages: + msg231804
2014-11-28 09:56:57spilsetfiles: - patch-configure.ac
2014-11-28 09:56:48spilsetfiles: + patch-configure.ac

messages: + msg231803
2014-11-28 09:52:30vstinnersetmessages: + msg231802
2014-11-28 09:50:46spilsetmessages: + msg231801
2014-11-28 08:52:29pitrousetmessages: + msg231799
2014-11-28 08:49:02vstinnersetmessages: + msg231798
2014-11-28 08:47:46spilsetfiles: + patch-Modules__ssl.c
2014-11-28 08:47:34spilsetfiles: + patch-Lib_ssl.py
2014-11-28 08:47:09spilsetfiles: - patch-Modules__ssl.c
2014-11-28 08:46:49spilsetfiles: + patch-configure.ac

messages: + msg231797
2014-11-21 10:47:29spilsetfiles: + patch-Modules__ssl.c

messages: + msg231471
2014-11-21 01:08:18pitrousetmessages: + msg231462
2014-11-20 13:23:53vstinnersetmessages: + msg231428
2014-11-20 12:15:27spilsetnosy: + spil
messages: + msg231426
2014-11-20 11:56:51koobssetnosy: + koobs
2014-10-01 18:26:32polymorphmsetnosy: + polymorphm
2014-09-12 18:05:57pitrousetmessages: + msg226832
2014-09-12 13:50:54mgornysetnosy: + mgorny
messages: + msg226819
2014-09-04 10:44:23vstinnersetmessages: + msg226357
2014-09-04 10:36:59vstinnersetmessages: + msg226355
2014-05-23 11:37:33floxsetnosy: + flox
2014-05-23 08:20:29oberstetsetnosy: + oberstet
2014-04-27 00:28:51vstinnersettitle: LibreSSL/RAND_egd fix needed. -> Support LibreSSL (instead of OpenSSL): make RAND_egd optional
2014-04-26 15:13:05rpointelsetnosy: + rpointel
2014-04-26 14:48:35pitrousetversions: + Python 2.7
2014-04-26 14:48:21pitrousetnosy: + janssen, pitrou, vstinner, giampaolo.rodola, christian.heimes, dstufft

messages: + msg217199
versions: - Python 3.1, Python 2.7, Python 3.2, Python 3.3
2014-04-26 14:21:36Edd.Barrettcreate