I agree that backporting X509_VERIFY_PARAM_set1_host is unreasonable, at least until the openssl ecosystem has moved forward a bit. But in earlier versions, would it be easy to detect that do_handshake() hasn't been called and raise an error?

The docs say you have to call do_handshake(), so if you don't that's already a bug and breaking that case should be OK, especially since it's never worked correctly.

I'm not very stressed about this myself because my code doesn't trigger the error -- only buggy code does :-). But it would be nice if the buggy code could fail closed.