I recently ran 2to3 on some of my scripts and noticed a change in behavior.

I had a script that used the built-in execfile function. After the conversion, it was 
changed to manually open the file and read the contents into the exec function. Now I am 
getting a SyntaxError with the modified code.

The SyntaxError is caused when the new exec code is called on a script that ends with an 
empty indented line. I'm not sure if this is an error with the 2to3 script or the built-
in exec function.

I've attached a very minimal test case. The test.py file is a simple hello world script 
that ends with an empty indented line. The execfile_example.py file simply calls 
execfile on the test.py script. The execfile_example_converted.py file is the 2to3 
converted version of execfile_example.py.

I'm running Python 2.6.2 on Windows XP.

Could you attach the files separately or paste them into the bug? zip
files are hard to work with.

I just attached the files individually.

test.py is invalid Python 3 syntax.

I'm running this code under 2.6, so the print statement should not be the 
issue. I've attached a new version of test.py that simply performs a 
variable assignment and I still get the syntax error on both 2.6 and 3.1 
with the exec function. Also, the syntax error is reported on line 3, 
which is the empty indented line. Sorry for the confusion.

This is because you have DOS newlines which the Python compiler cannot
handle. In 2.x, open("test.py", "r") does not translate newlines. In
3.x, it does.

Ok, I converted test.py to use Unix style newlines and still get the 
syntax error on both 2.6 and 3.1. I'm confused as to why execfile works on 
the file but reading the contents and passing it to exec behaves 
differently under 2.6. Sorry if I'm just being dense.

2009/11/5 flashk <report@bugs.python.org>:
>
> flashk <flashk@gmail.com> added the comment:
>
> Ok, I converted test.py to use Unix style newlines and still get the
> syntax error on both 2.6 and 3.1. I'm confused as to why execfile works on
> the file but reading the contents and passing it to exec behaves
> differently under 2.6. Sorry if I'm just being dense.

open in 2.x does not translate newlines by default. It does so in 3.x.

Ok, but why am I still getting a syntax error in both 2.6 and 3.1 on the 
file, even after converting the newlines?

If I remove the trailing indentation then everything works properly on 2.6 
and 3.1, even with DOS newlines.

It just seems that exec does not properly handle code that ends with an 
empty indented line, but execfile handles it correctly.

2009/11/5 flashk <report@bugs.python.org>:
>
> flashk <flashk@gmail.com> added the comment:
>
> Ok, but why am I still getting a syntax error in both 2.6 and 3.1 on the
> file, even after converting the newlines?
>
> If I remove the trailing indentation then everything works properly on 2.6
> and 3.1, even with DOS newlines.
>
> It just seems that exec does not properly handle code that ends with an
> empty indented line, but execfile handles it correctly.

Well, it works for me with the empty newline. Can you isolate the exact problem?

On Thu, Nov 5, 2009 at 5:48 PM, Benjamin Peterson <report@bugs.python.org>wrote:
>
>
> Well, it works for me with the empty newline. Can you isolate the exact
> problem?

For me, the exact problem seems to be that exec raises a SyntaxError if the
code contains a trailing indentation. Here's a summary of everything I've
tried:

 * test.py runs successfully on 2.6 and 3.1, regardless of line ending style
(LF, CR+LF)

 * "execfile('test.py')" runs successfully on 2.6, regardless of line ending
style

 * "exec(compile(open('test.py').read(), 'test.py', 'exec'))" raises a
SyntaxError on line 3 of test.py under 2.6 and 3.1, regardless of line
ending style.

 * Removing the trailing indentation from test.py causes the above code to
succeed under 2.6 and 3.1, regardless of line ending style.

You mentioned that it worked for you with an empty newline. To be clear, the
last line of test.py contains a single tab, with no newline character.

I noticed that calling "exec('\t')" raises a SyntaxError, so maybe this is 
the root of the problem. I manually added a newline character to the end 
of the file contents and it fixes the issue for me:

exec(compile(open('test.py').read()+'\n', 'test.py', 'exec'))

So maybe the 2to3 script should automatically do this in order to be 
compatible with previous execfile behavior? Or is this a problem with 
exec?

The docs for the built-in function compile() say:

"Note: When compiling a string with multi-line statements, line endings 
must be represented by a single newline character ('\n'), and the input 
must be terminated by at least one newline character. If line endings 
are represented by '\r\n', use str.replace() to change them into '\n'."
http://www.python.org/doc/3.1/library/functions.html#compile

And the standard module py_compile ensures the source text ends with 
'\n' before calling the built-in compile function:

    if codestring and codestring[-1] != '\n':
        codestring = codestring + '\n'

I'd say the corresponding 2to3 fixer should do the same.

This is a patch for the execfile fixer, so it converts
        execfile("fn")
into this:
        exec(compile(open("fn").read()+'\n', "fn", 'exec'))

(Yes, it looks ugly. A better way would be to fix the compile() builtin 
so it does not require the last line of source to end with '\n')

This is my very first incursion into 2to3 so it may be terribly wrong...

I've fixed underlying compile() newline problem in the trunk and py3k.
However, I think that change is big enough that I don't want to to
backport it. I'm leaving this open to think about what to do with 2to3
in this situation.

Not sure if this merits a new bug report: The conversion currently promotes the open().read() anti-pattern, which is not guaranteed to release file handles as soon as possible in all VMs. Using a with block would fix that.

Closing since compile() will work properly now.