tok->enc and tok->encoding should always have the same value, except that tok->enc gets set earlier.

tok->enc is used when parsing from strings, to remember what codec to use. For file based parsing, the codec object created knows what encoding to use; for string-based parsing, tok->enc stores the encoding.

If the code is to be simplified, unifying the cases of string-based parsing and file-based parsing might be a worthwhile goal.