Attached is a patch against the py3k branch which implements this.

I've checked that it builds against openssl-0.9.8o.tar.gz, openssl-1.0.0a.tar.gz, and against Fedora 12 and 13's heavily-patched openssl-1.0.0. The bulk of my testing has been against Fedora's openssl.

I've added selftests to try to verify the new API.  I try to detect if the OpenSSL enforces FIPS, via trying to run "openssl md5" as a subprocess, and seeing if I can trigger an error.

With FIPS enforcement off, all tests pass when built against 0.9.8o and 1.0.0a and F13's 1.0.0, other than those for FIPS enforcement itself, which skip.

With FIPS enforcement on, all tests pass when built against F13's openssl.  (I haven't yet figured out how to get the fips selftest to pass for the other builds, it's testing checksums against the wrong libcrypto for some reason; see caveat below):
$ ./python Lib/test/test_hashlib.py
$ OPENSSL_FORCE_FIPS_MODE=1 ./python Lib/test/test_hashlib.py

For all of the various contexts stored in _hashopenssl.c, we now store two: one with the override flag, one without.  This required some reworking of the various preprocessor magic in that file, so I've gathered everything related to an algorithm into a structure, and moved most of the logic into functions, rather than macros.  I'm assuming that these will get inlined under optimization, and that the bulk of the time that you're trying to optimize out are the EVP lookups and initializations, rather than function call overhead.

How's this looking?

Do I need to add a dummy "usedforsecurity" arg to all of the non-openssl message digest implementations within the tree?


Unfortunately, if fips mode is on, and the fips selftest fails for the openssl library, every hash use will fail, both with and without the flag:
  ValueError: error:2D07D06A:FIPS routines:EVP_DigestInit_ex:fips selftest failed
and this leads to a crippled hashlib module.  It's not clear to me if there's a good way to handle this.  (Having said that, a site that has the technical expertise to opt-in to FIPS mode is hopefully able to diagnose this, and fix their openssl library)