Author andreash
Recipients AlexWaygood, andreash, gvanrossum, kj, kumaraditya303
Date 2022-01-13.23:48:14
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1642117695.26.0.396895524876.issue46371@roundup.psfhosted.org>
In-reply-to
Content
(De)Serialization of in-memory data structures is an important application. 
However there is a rather unpleasant issue with ForwardRefs.


    One cannot export type aliases when they contain ForwardRefs (and expect things to work).


Consider the example:

    Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]


When used in another module the containing ForwardRefs cannot be resolved, or in the worst case, resolve to 
wrong types if in the caller namespace contains a symbol with the same name as ForwardRef refers to.

This of course only applies to inspection-based tools which utilize the run-time information, not the static type checkers.


Type aliases sometimes have to be used sometimes - they cannot be avoided in all cases, 
especially with recursive types as in the example above.  



There are several options to improve the situation. These are all that came to my mind and I want to expose them to discussion.



1. Guard with NewType

    Json = NewType('Json', Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] )
   
   Advantage: This could allow automatic cross-module ForwardRef resolution provided issue 46369 [1] is acknowledged as a bug and fixed. 
   Disadvantage: Does not create a true type alias, but a sub-type. Type casts have to be used all the time, e.g.   `data = Json(bool)`.
      So can only applied to a subset of use-cases (but is IMO a clean solution when it fits). 


2. Use the `module` parameter of ForwardRef

    Json = Union[ List[ForwardRef('Json', module=__name__)], Dict[str, ForwardRef('Json', module=__name__)], int, float, bool, None ] )

   Advantage: Works in 3.10. 
   Disadvantage: Would require issue 46333 [2] to be fixed. ForwardRef is not meant to be instatiated by the user, 
       also `module` parameter is currently completely internal.


3. Modify ForwardRef so that it accepts a fully qualified name

    Json = Union[ List[__name__+'.Json'], Dict[str, __name__+'.Json'], int, float, bool, None ] )

   Advantage: This is only a tiny change (because ForwardRef has the `module` parameter). ForwardRef would stay internal. Less ugly than 2.
   Disadvantage: Still a bit ugly. Would also require issue 46333 [2] to be fixed. Relative module specification (as in relative imports) 
      would not work.


4. Manual evaluation

    Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
    resolve_type_alias(Json, globals(), locals() )


    def resolve_type_alias(type, globalns, localns):
        class dummy:
            test: type
        typing.get_type_hints(dummy, globalns, localns) # Note: this modifies ForwardRefs in-place

    Advantage: Works in many versions.
    Disadvantage: Requires user to explicily call function after the last referenced type is defined 
       (this may be physically separated from the type alias definition, which does not feel like a good solution especially since 
        this ForwardRef export problem is only technical, and not even close to beeing obvious to most people)


5. Make `get_type_hints()` to work with type-aliases (essentially integrate above `resolve_type_alias`). The return value 
   of get_type_hints() would be the same as argument, just with ForwardRefs in-place resolved. 
 
     Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
     get_type_hints(Json, globals(), locals())
 
   Advantage: same as 4) but hides useful (but ugly) code
   Disadvantage: same as 4)


 
6. Make all types in typing (such as List, Dict, Union, etc...) to capture their calling module and pass this information to ForwardRef, when 
   one is to be created. Then already during construction of ForwardRef the `module` will be correctly set.

     Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
    
   Advantage: This requires no user intervention. Things will "just work"
   Disadvantage: This is rather big change. It is incompatible with the caching used inside typing.py (the new __module__ parameter would 
       need to be taken into account in __hash__ and/or __eq__). And maybe has other issues I do not see at the moment.


7. Extend `TypeAlias` hint so it can be used in bracket  way similar to e.g. `Annotated`

     Json = TypeAlias[ Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]  ]
   

   I know, presently it is supposed to be used as  `Json: TypeAlias = Union[ .... ]`. But that is of no help at run-time, because
   the variable Json contains no run-time information. So even if TypeAlias would capture the calling module, 
   this information is not passed on to the variable `Json`. This is different for the bracket notation TypeAlias[ .. ]. 

   Advantage: Similar usage to Annotated. Would not require a big change such as in 4).
   Disadvantage: TypeAlias is not supposed to be used that way.


8. Introduce function to define a type alias, which sets the module parameter of all ForwardRefs

     Json = DefineTypeAlias( Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] )

   DefineTypeAlias would capture its calling module and recursively walk over the type tree to find and patch all ForwardRefs

   Advantage: Simpler change, but requires to recurively walk over all alias type-hints which do not capture their calling module.  
   Disadvantage: Together with TypeAlias, things look ugly and unnecessary complicated. (Json: TypeAlias = DefineTypeAlias( ... ) )


Personally, I think 1) is a good solution in cases where this can be applied (and [1] is considered a bug and fixed). 
For the other cases 6) would be the ideal solution, but this may be too much work. Alternatively I think 3), 8) or 5) (in that order) 
may be interesting and could be a potential enhancement to the `typing` module.



I would really appreciate some thoughts or comments on this! Thank you. 



- [1] https://bugs.python.org/issue46369
- [2] https://bugs.python.org/issue46333
History
Date User Action Args
2022-01-13 23:48:15andreashsetrecipients: + andreash, gvanrossum, kj, kumaraditya303, AlexWaygood
2022-01-13 23:48:15andreashsetmessageid: <1642117695.26.0.396895524876.issue46371@roundup.psfhosted.org>
2022-01-13 23:48:15andreashlinkissue46371 messages
2022-01-13 23:48:14andreashcreate