> I don't know what exception type should be used here: 
> OSError with an errno EINVAL or some specialized type. 

No, I think the appropriate error is ValueError, at least if errno is EINVAL.

> because I don't know under what conditions it would be possible error

AFAICT, glibc never sets errno. 

msvcrt gives EILSEQ or ERANGE, but never EINVAL. EILSEQ is returned if LCMapString failed, and ERANE if the output buffer is too small. In either case, the result is INT_MAX.