Thanks for your comments Christian.
"You don't check ERR_GET_LIB() in some places."
Do you have a particular place in mind?

About DER.  As I understand, currently _ssl only supports PEM.  If that is the case, then supporting DER should, IMHO, be a separate patch.  It will probably involve adding an argument to the functions.

Both DER and PEM files are binary formats.  the PEM is base64 encoded ascii with some strict ascii headers and footers.  As such, the encoding is quite explicit.
I am not sure that automatic conversion from unicode to ascii should be undertaken on the C level, particularly if the intention later is to support DER, which is pure binary and where "string" has no place.

I'll have a look at your patch as well, haven't gotten round to do that yet.