I agree with Guido that it's better to design inspect.signature to not throw an error for annotations that don't eval() cleanly.

I use inspect.signature for getting information about callables (third-party and first-party) in my type checker: https://github.com/quora/pyanalyze/blob/master/pyanalyze/arg_spec.py#L436. 
 In that context, I'd much rather get string annotations that I can process myself later than get an exception if the annotations aren't valid at runtime. In the former case I can still get useful information out of the signature even if I don't know how to process some annotations. For example, I'll still know what parameters exist even if I don't know what their types are. In the latter case, I don't get any useful signature data.