This problem is the parsing of f-strings. 

The expressions in an f-string are not "eval"ed in the sense of the eval() function. They are evaluated exactly the same as any other Python expression. However the parsing of f-strings does not provide correct line numbers.

This problem also manifests itself in the ast and tokenize modules.

>>> m = ast.parse("""f'''
... {
... FOO
... }
... '''
... """)

>>> m.body[0].value.values[1].value.id
'FOO'
>>> m.body[0].value.values[1].value.lineno
2

That 2 should be a 3, and yet
eval(compile(m, "test2", "exec"))
  File "<stdin>", line 1, in <module>
  File "test2", line 5, in <module>
NameError: name 'FOO' is not defined

gives line 5 for the error, so not only are the line numbers wrong they are inconsistent.

The problem is that the internals of the f-string are not tokenized and parsed using the normal mechanism, but in an ad-hoc fashion in Python-ast.c as demonstrated when tokenizing the source

$ python3.6 -m tokenize test2
0,0-0,0:            ENCODING       'utf-8'        
1,0-5,3:            STRING         "f'''\n{\nFOO\n}\n'''"
5,3-5,4:            NEWLINE        '\n'           
6,0-6,0:            ENDMARKER      ''

The f-string could should be tokenized as something like:
FSTRING_START f'''
STRING_PART \n 
LEFT_BRACE {
NEWLINE
IDENTIFIER FOO
NEWLINE
RIGHT_BRACE }
STRING_PART \n
FSTRING_END '''

Although this would complicate the tokenizer, it would mean that the internals of f-strings could be made explicit in the grammar, and that the compiler could generate correct offsets.