IDLE tries to be helpful, but it errors on something that isn't yet an error. Pasting in this code works fine:

>>> def a():
	def b():
		nonlocal q
		q+=1
	q=1
	b()
	return q

>>> a()
2

But typing it, line by line, results in an error:

>>> def a():
	def b():
		nonlocal q
		
SyntaxError: no binding for nonlocal 'q' found

This doesn't occur with interactive command-line Python. A small issue, more of curiosity value than anything else - I don't have a non-trivial use-case where this causes problems.

This started as a python-list thread. On that thread, Peter Otten verified the behavior on Linux, so it is not specific to the Windows pythonw executable. He also verified on 3.2 (which gets security fixes only) and 3.4. I just checked 2.7. On that thread, I verified a) the three lines submitted by themselves as a file cause the same error message, as they should ("must refer to pre-existing bindings in an enclosing scope ") and b) Python in interactive mode syntax-checks lines as entered, even for compound statements. I just verified that Idle behaves the same when started with -n (no subprocess):
C:\Programs\Python33>python -m idlelib.idle -n

Summary: Idle acts as if it syntax-checks the cumulative input instead of just the line entered. (But I have not looked as the code.)

The three lines constitute a test if they can be run from a test file.

A test will be a challenge since the three lines have to be run as if entered interactively. This may require new code to inject lines into the interactive processing stream, or a least a re-factoring of existing code.

The bug is not in Idle. Its interpreter is a subclass of code.InteractiveInterpreter (II) and that (and its subclass InteractiveConsole) have the bug.

C:\Programs\Python33>python -m code
Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> def a():
...  def b():
...   nonlocal c
  File "<string>", line None
SyntaxError: no binding for nonlocal 'c' found

II.runsource compiles cumulative source with codeop.CommandCompile, which wraps codeop._maybe_compile. That returns None (source incomplete), raises (source complete but invalid), or return code (source complete and valid) to be executed. It mis-classifies the code in question.

>>> import codeop as op
>>> src = '''def a():
  def b():
     nonlocal c
'''
>>> op.CommandCompiler()(src)
Traceback (most recent call last):
...
SyntaxError: no binding for nonlocal 'c' found

PyShell.ModifiedInterpreter.runsource wraps II.runsource.
       return InteractiveInterpreter.runsource(self, source, filename)

Someone needs to compare _maybe_compile to the equivalent C code used by the real interpreter.

> Someone needs to compare _maybe_compile to the equivalent C code used by the real interpreter.

Well, _maybe_compile() just calls the built-in function compile() internally, so I'm not sure what sort of comparison you want...

The comment at the top of codeop explains the problem (and why Terry is interested in what the C code is doing, since it's clearly different):

===============
Compile three times: as is, with \n, and with \n\n appended.  If it
compiles as is, it's complete.  If it compiles with one \n appended,
we expect more.  If it doesn't compile either way, we compare the
error we get when compiling with \n or \n\n appended.  If the errors
are the same, the code is broken.  But if the errors are different, we
expect more.  Not intuitive; not even guaranteed to hold in future
releases; but this matches the compiler's behavior from Python 1.4
through 2.2, at least.
===============

Unfortunately, the way the main interactive loop works isn't useful to codeop: the C level loop simply blocks on stdin, waiting until the parser spits out a complete AST. By contrast, codeop is handed complete strings, and has to decide whether they're a potentially incomplete or not.

"nonlocal c" is unique in that it's a name lookup that is *checked by the compiler*, so it triggers the *same* exception in all 3 cases that codeop._maybe_compile tries:

>>> src = "def a():\n   def b():\n    nonlocal c"
>>> compile(src, "", "single")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "", line 3
SyntaxError: no binding for nonlocal 'c' found
>>> compile(src + "\n", "", "single")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "", line 3
SyntaxError: no binding for nonlocal 'c' found
>>> compile(src + "\n\n", "", "single")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "", line 3
SyntaxError: no binding for nonlocal 'c' found

So it's a SyntaxError that *could be fixed* by later code, but that code never gets a chance to run, because codeop assumes that getting the same error in the last two cases means it will never pass.

As a potential fix (albeit an ugly hack), try changing this part of codeop._maybe_compile:

    if not code1 and repr(err1) == repr(err2):
        raise err1

To something like:

    if not code1 and repr(err1) == repr(err2):
        if isinstance(err1, SyntaxError) and "no binding for nonlocal" in str(err1) and not source.endswith("\n\n"):
            # Allow a nonlocal namebinding to be supplied *after* a
            # a function definition (the standard interpreter loop
            # handles this by blocking on stdin, but this module accepts
            # input as complete strings rather than as a stream)
            return None
        raise err1

A complete fix is going to require setting a flag that we have a pending non-local, and check that flag when the code input is complete to raise the SyntaxError at that point if the non-local hasn't been set.

codeop compiles the whole pending statement every time, so that part
shouldn't be needed (from the compiler's point of view, this input
looks complete - there's nothing dangling, any more than there is for
a normal syntax error like "name name"). As far as I can tell, the
only reason it works in the internal parser case is because the parser
doesn't consider the input complete until it sees the expected
dedents, which the tokenizer doesn't generate until it sees the
trailing newline.

That may actually be another alternative: instead of doing the "try
appending newlines and see if it works or generates different errors",
we may be able to switch to the tokenizer if the initial compilation
fails and check for hanging INDENT tokens (i.e. INDENTS without a
corresponding DEDENT). That would get us much closer to what the real
eval loop is doing.

Pablo and Lysandros: this issue is about a corner-case bug in either compile(,,'single') or its use by REPL and codeop._maybe_compile (comc).  What do either of you think?  Can the new parser handle it better?

Should we just add the hack suggested by Nick in msg200936 or is there a better solution?  Is interactive mode python calling compile() differently?  Or is it doing a check that could be incorporated into compile?  (Perhap after correction, see below.)

The issue above is about interactive entry of

def a():
  def b():
    nonlocal c

REPL does not raise SyntaxError, comc does.  Not raising is correct here because additional lines added to the nonlocal context may make the code valid.

Additional experiment: the same is true (comc raises, REPL not) for

def a():
  nonlocal c

Here, REPL not raising (until a blank is entered) could be considered a glitch because there is no pending nonlocal context to be completed.  Though raising later than necessary is better than raising too soon.  Nick's hack (and the REPL) could check that there are at least 2 pending indents.

When the new parser compiles a function, does it know or could it know whether it is nested?  It should in that the legal grammer (use of nonlocal) is different.

Note: Recently in #40807, Cheryl and I patched codeop._maybecompile to only emits warnings once in a given call.  I don't know if 3 calls (2 '\n' additions) to compile are really needed today.  The logic that handles the results is not clear to me either.  It could stand review by compile() experts.

> What do either of you think?  Can the new parser handle it better?

Pablo, correct me if I'm wrong, but I think that the parser has nothing to do with this. It's not the parser that produces this SyntaxError, it's some later compilation phase (I guess symtable creation phase?), which I know very very little about, so I can't really offer an opinion on what can be done.

In any case, this should actually indicate that this is not a parser issue:

(venv) ➜  cpython git:(master) ./python.exe
Python 3.10.0a0 (heads/master:7f569c9bc0, Jun 29 2020, 14:48:39)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = '''\
... def a():
...     def b():
...         nonlocal c
... '''
>>> a
'def a():\n    def b():\n        nonlocal c\n'
>>> compile(a, '?', 'single')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "?", line 3
SyntaxError: no binding for nonlocal 'c' found
>>> import ast
>>> ast.dump(ast.parse(a, mode='single'))
"Interactive(body=[FunctionDef(name='a', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[FunctionDef(name='b', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Nonlocal(names=['c'])], decorator_list=[])], decorator_list=[])])"

All in all, I think that Nick's hack is the way to go here, since it's comc's responsibility to correctly check if a statement is incomplete, right?


> When the new parser compiles a function, does it know or could it know whether it is nested?

It currently doesn't, but the tokenizer holds a stack of indent tokens, which can be accessed by the parser. This way we could indeed check if there are two or more pending indent tokens, before generating an AST for nonlocal. However, I don't think that this is generally a good idea. Parsing should not care about statement semantics, in that a nonlocal statement should be like any other statement in the parser's point of view.

> Pablo, correct me if I'm wrong, but I think that the parser has nothing to do with this. It's not the parser that produces this SyntaxError, it's some later compilation phase (I guess symtable creation phase?), which I know very very little about, so I can't really offer an opinion on what can be done.

Yeah, this is correct the parser has nothing to be done with this, what is involved in this is the compiler, as the generated AST looks correct.

>  When the new parser compiles a function, does it know or could it know whether it is nested?

Nop, the parser does not even know what a 'function' is while it parses, as that is only determined in the product of the parser itself: the AST. The piece that understands semantically what a function is and if is nested or not is the compiler.

> Parsing should not care about statement semantics, in that a nonlocal statement should be like any other statement in the parser's point of view.

Yup, I very much agree with this. The parser must not know about the semantics of the AST elements (forcing it to do so will be not only very complex but also will make maintenance much more difficult as the parser will be coupled with the semantics of the language is parsing as opposed to only having to know about the grammar).

> All in all, I think that Nick's hack is the way to go here, since it's comc's responsibility to correctly check if a statement is incomplete, right?

If I understand correctly, Nick is talking about modifying the different iterations for different errors in codeop no?

> If I understand correctly, Nick is talking about modifying the different iterations for different errors in codeop no?

I was talking about msg200936, where Nick proposed to just hardcode a check for nonlocal SyntaxErrors in codeop._maybe_compile, which should return None (which means incomplete) when the SyntaxError comes as a result of a nonlocal binding not existing.

> That may actually be another alternative: instead of doing the "try
> appending newlines and see if it works or generates different errors",
> we may be able to switch to the tokenizer if the initial compilation
> fails and check for hanging INDENT tokens (i.e. INDENTS without a
> corresponding DEDENT). That would get us much closer to what the real
> eval loop is doing.

From what I understand, "checking for two or more hanging INDENTS" and, "hardcoding a check for nonlocal SyntaxErrors in codeop._maybe_compile" are two different solutions, right?  If yes, do we have an answer to which one of them is more cleaner, and henceforth, the preferable solution?

I, personally, like the idea of checking INDENTS primarily because of it's reduced specificity, but I am in no position to comment on this (I already kinda did ':D), and you folks know better! For all we know, we should be optimizing for specificity.

Also, reading Nick's comments and the comc's code, gives me the feeling that a fix for this wouldn't require drastic changes.  I'm slowly starting my journey with CPython, and I'd like to contribute a patch if that is the case. Thanks!

> From what I understand, "checking for two or more hanging INDENTS" and, "hardcoding a check for nonlocal SyntaxErrors in codeop._maybe_compile" are two different solutions, right?  If yes, do we have an answer to which one of them is more cleaner, and henceforth, the preferable solution?

Note that these are two solution that take very different approaches. What Nick is suggesting with "checking for two or more hanging INDENTS" would drastically change how codeop._maybe_compile does its thing, while his other proposed solution, ""hardcoding a check for nonlocal SyntaxErrors in codeop._maybe_compile", would just fix this issue.


> I, personally, like the idea of checking INDENTS primarily because of it's reduced specificity, but I am in no position to comment on this (I already kinda did ':D), and you folks know better! For all we know, we should be optimizing for specificity.

You're right that this idea is more general. It would require significantly more effort from whoever tackles this though.

> Also, reading Nick's comments and the comc's code, gives me the feeling that a fix for this wouldn't require drastic changes.

That really depends on what solution is chosen out of the two.

> I'm slowly starting my journey with CPython, and I'd like to contribute a patch if that is the case. Thanks!

If there's consensus around one proposed approach, you could certainly take this up. I'd be glad to help out with workflow stuff or provide a first review. Let me know if you've got more questions.

>  Note that these are two solution that take very different approaches. What Nick is suggesting with "checking for two or more hanging INDENTS" would drastically change how codeop._maybe_compile does its thing, while his other proposed solution, ""hardcoding a check for nonlocal SyntaxErrors in codeop._maybe_compile", would just fix this issue.

Got it!

> If there's consensus around one proposed approach, you could certainly take this up. I'd be glad to help out with workflow stuff or provide a first review. Let me know if you've got more questions.

Thank you so much Lysandros!

_maybe_compile currently compiles the possibly complete statement up to 3 times -- with C-coded compile.  Without doing any timing tests, I wondered if 3 times is really necessary.  Nick suggested that using the tokenize module to determine the number of hanging indents, combined with some yet to be determined logic, might be an alternative.  But since tokenize is written in Python, the result might not be any faster.

Checking the error message aims at specifically fixing this issue.  But I am not sure if the check is sufficient or if more logic is needed.
  "def a():\n   nonlocal c\n" should ideally raise.
  "def a():\n   def b():\n    nonlocal c\n" must not.

At least the second should be used for a new test.

I would start by adding debug prints to see the result of each compile and then testing with those two lines.

I feel that I wasn't clear at all in my previous responses, so let me try to have another go at explaining how I see this:

We have two distinct problems, that are totally unrelated to one another:

1) nonlocal should raise when it's not in a nested function (question for the more experienced here: is this the *only* place where nonlocal is allowed?)

2) comc should return None, signifying an incomplete input, when presented with the bloack:

def a():
    def b():
        nonlocal c

because c, the variable, could be declared in a, the function, some time after the definition of b, the function, is done.

The way I see this, like I have expressed in my previous comments, is that whatever solution we come up with should not involve the parser, because it should not care about statement semantics. I'm not really sure what the best solutions are, but I'd propose the following for the problems I listed above:

1) We should alter code in symtable.c to check whether the namespace in which the nonlocal statement appears in is a function block and whether it is nested or not.

2) A check for the SyntaxError message to see if it comes from a nonlocal statement surely sounds like a hack, but I don't think that's a dealbreaker. I'd probably go with this rather than undergoing the effort of exposing the indentation stack in Python and then examining it in codeop.

Thoughts?

> 1) We should alter code in symtable.c to check whether the namespace in which the nonlocal statement appears in is a function block and whether it is nested or not.

We already raise in this case. Consider this 'test.py' file:

x = 34
def f():
    nonlocal x
    x = 24

f()

Executing python test.py:


  File "test.py", line 3
    nonlocal x
    ^
SyntaxError: no binding for nonlocal 'x' found

>1) We should alter code in symtable.c to check whether the namespace in which the nonlocal statement appears in is a function block and whether it is nested or not.

Also, the 'nesting' has very specific meanings in the symbol table: scoping. The validity of the keyword translates there if is able to find a variable in the enclosing intermediate scope (when its not the global one).

That's right, but the SyntaxError message is somewhat confusing. At least the fact that the SyntaxError message is some for both of

def a():
    def b():
        nonlocal x

and 

def a():
    nonlocal x

seems a bit misleading. No?

some -> same

> At least the fact that the SyntaxError message is some for both of

Well, is the same because the semantics are the same: a symbol is missing in the expected scopes. The only difference is that in the first case is not even possible to "fix it" because that scope cannot exist if the function is not 'nested'. But I do agree that it may be some value on specializing the error message to that case (but is not enough to have the nesting, you would need also to add a variable - the error is really 'you need a symbol in an intermediate scope' -> 'Oh, i don't even have an intermediate scope' -> 'then *first* you need to add an intermediate scope). Although this particular problem is indeed different from the main one (the one about codeop).

first case -> second case

I reread Nick's comment "the C level loop simply blocks on stdin, waiting until the parser spits out a complete AST."  I interpret that now as meaning that the REPL compiles user code only once per statement, which IDLE and code.InteractiveInterpreter compile a statement once per line, which codeop expands to 3 compiles in an attempt to determine whether more is needed.  This can all be considered to be a hack, so adding 1 more does not bother me now.

(Side note: if the first compile succeeds in producing a code object, it is unconditionally returned after the additional compiles, so they seem like a waste.)

I looked at a git blame listing for codeop.  It was extracted from code by Guido in 1998 so that Jython (JPython) could replace compile_command.  It was not revised after the addition of nonlocal broke it.

I traced the flow of calls from when a user hits <Enter> in the entry area to the call of compile, and of the returns back.  The attached idle-shell-compile.txt records what I found.  Of particular interest to me:

1. IDLE strips trailing ' 's, '\t's, and 1 '\n' from user input before sending it to be compiled. (This goes back to the original IDLE commit in 2000.) So there is a trailing '\n' only when the user has hit '<Enter>' twice to complete a compound statement.  It would solve this issue *for IDLE* to only print the nonlocal error message when there is a trailing '\n'.  I am willing to give up the special casing needed to get 'def a():\n  nonlocal c' (no double <Enter>, no trailing '\n') to raise immediately.

Nothing in code says that it expects source to be stripped as IDLE does.  The reason for the latter might just be to get rid of the 'smart indent' that will precede the first Enter, and possibly anything the user adds before the second.  

2. compile is passed the undocumented flag PyCF_DONT_IMPLY_DEDENT = 0x200.  Can anyone explain the full intent?  I found this difference.

>>> compile("def f():\n  nonlocal c", '', 'single')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "", line 2
SyntaxError: no binding for nonlocal 'c' found

# Same code, flag added, different message.
>>> compile("def f():\n  nonlocal c", '', 'single', 0x200)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "", line 2
    nonlocal c
             ^
SyntaxError: unexpected EOF while parsing

# Add '\n' and message is same as without flag.
>>> compile("def f():\n  nonlocal c\n", '', 'single', 0x200)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "", line 2
SyntaxError: no binding for nonlocal 'c' found

To use the message difference, the first compile-with-flag message would have to be saved for comparison, and the difference would only appear if source had no trailing '\n'.  We could make that be true, or we could wait until another user of codeop complains.