Problem:


lib2to3.fixer_util.touch_import('__future__', ...)  will insert the import statement below all other imports. This will then trigger the following error:

    SyntaxError: from __future__ imports must occur at the beginning of the file


How to reproduce the issue (using modernize, which uses lib2to3):

    $ git clone https://github.com/mitsuhiko/python-modernize.git
    $ cd python-modernize
    $ echo << EOF >> test.py
    # test.py
    "Test case for lib2to3.fixer_util.touch_import('__future__', ...)"
    import os
    print "hi"
    EOF

    $ python modernize.py test.py
    --- test.py (original)
    +++ test.py (refactored)
    @@ -1,4 +1,5 @@
     # test.py
     "Test case for lib2to3.fixer_util.touch_import('__future__', ...)"
     import os
    -print "hi"
    +from __future__ import print_function
    +print("hi")

    $ python3 test.py
      File "test.py", line 4
        from __future__ import print_function
        ^
      SyntaxError: from __future__ imports must occur at the beginning of the file


The following patch to lib2to3 seems to solve the issue:

    --- /usr/lib64/python2.7/lib2to3/fixer_util.py.orig 2012-03-09 21:53:10.841083479 -0800
    +++ /usr/lib64/python2.7/lib2to3/fixer_util.py  2012-03-09 21:58:18.678946683 -0800
    @@ -306,14 +306,15 @@
         # figure out where to insert the new import.  First try to find
         # the first import and then skip to the last one.
         insert_pos = offset = 0
    -    for idx, node in enumerate(root.children):
    -        if not is_import_stmt(node):
    -            continue
    -        for offset, node2 in enumerate(root.children[idx:]):
    -            if not is_import_stmt(node2):
    -                break
    -        insert_pos = idx + offset
    -        break
    +    if package != '__future__':
    +        for idx, node in enumerate(root.children):
    +            if not is_import_stmt(node):
    +                continue
    +            for offset, node2 in enumerate(root.children[idx:]):
    +                if not is_import_stmt(node2):
    +                    break
    +            insert_pos = idx + offset
    +            break

         # if there are no imports where we can insert, find the docstring.
         # if that also fails, we stick to the beginning of the file


After the patch, all is well:

    $ python modernize.py test.py
    --- test.py (original)
    +++ test.py (refactored)
    @@ -1,4 +1,5 @@
     # test.py
     "Test case for lib2to3.fixer_util.touch_import('__future__', ...)"
    +from __future__ import print_function
     import os
    -print "hi"
    +print("hi")

    $ python3 test.py
    hi

I think that's actually a bug in python-modernize, not in touch_import - or, rather, touch_import shouldn't be used to add future imports.

Instead, I think there should be a touch_future function which adds a future import, taking into account that future imports need to go before all other imports, but after a possible docstring.

Instead of changing touch_import, I propose to add a function similar to

https://github.com/loewis/python-modernize/commit/0db885e616807d0cc6859b4035d81fd260b06a67

Yep, that seems like the right solution. Thanks, Martin.