It would be nice with some official way to tell 2to3, "Leave
this code chunk alone.  This is 2.* code, that is 3.* code":

try:                      # Python 2.6
    from urlparse         import urlparse, urlunparse
except ImportError:       # Python 3
    from urllib.parse     import urlparse, urlunparse

Could 2to3 without -p notice more cases of print(single argument),
to avoid slapping another () around them?  For example:

print(2*3)
print(", ".join(dir))

I don't understand. If the code is already Python 3 code, why are you running 2to3 on it?

Martin v. Löwis writes:
> Martin v. Löwis <martin@v.loewis.de> added the comment:
> 
> I don't understand. If the code is already Python 3 code, why are you
> running 2to3 on it?

I should have clarified - it's still Python 2 code (maybe 2.7), moving
one step at a time towards something which will work on Python 3 as
well.

Or it's just that I'm handling one issue which 2to3 reports at a time,
like modifying str vs bytes vs unicode usage, then I come back and look
at another thing 2to3 has to say.

I still don't understand. If it's 2.x code, why do you want to say that it is 3.x code?

If you don't want to run a specific fixer, you can exclude it from the list of fixers.

Martin v. Löwis writes:
> I still don't understand. If it's 2.x code, why do you want to say that
> it is 3.x code?

It works on Python 2.  It runs on Python 3 - maybe correctly, or maybe
it's not that far along yet.  Maybe some files in a package work on
Python 3, and others have not yet been updated.

> If you don't want to run a specific fixer, you can exclude it from the
> list of fixers.

I know.  So this request is mostly for convenience, but then so is 2to3
in the first place.

print(single argument) would be nice to leave alone in any case though,
since there can be other reasons for the ().  E.g.:

    print(very long
          single argument)

Maybe we need to tackle this from a different angle: can you please specify the feature you are asking for exactly, with any syntax, API, or command line changes that you consider necessary?

Omitting redundant parentheses for print is a separate issue (feel free to open an issue); we should not mix it with this issue (which I still don't understand).

> Maybe we need to tackle this from a different angle: can you please
> specify the feature you are asking for exactly, with any syntax, API, or
> command line changes that you consider necessary?

First, nothing here is necessary, since it's just a request for a
convenience.  The syntax could be a generalization of the example I
gave:

  <whatever>:   # Python <version>
     <indented block> 

and

   <whatever>:
      # Python <version>
      <indented block>

tell 2to3 "The code in <block> it is expected to only run (or succeed)
in Python <version>.  Do not modify it, except it is an error if it
is a syntax error under Python 3 so the script won't even run".

'<whatever>:' can be some if,else,try,except, I don't know what else,
and is responsible for making different Python versions do the right
thing.

> Omitting redundant parentheses for print is a separate issue (feel
> free to open an issue); we should not mix it with this issue (which
> I still don't understand).

OK.  But I don't understand what you don't understand...

I have some Python 2 files in various stages of transition to be
runnable and correct in Python 3, while still working in Python 2.
It'd be convenient if the parts that do work under both Python 2 and 3
could be written so that 2to3 would be silent about them, without need
for different 2to3 command line options for different .py files.

Another syntax could be attached to if-else and try-except.  Given:

    if ...:
        <block 1>
    else:
        <block 2>

or
    try:
        <block 1>
    except ...:
        <block 2>

if 2to3 would translate <block 1> to <block 2> or vise versa, sans
whitespace/comment differences, then it does not do so.  Instead it
assumes the blocks are for running under Python 2 and 3 respectively.

How about this phrasing: “Make 2to3 fixers not touch code in a block starting with ’if sys.version >= '3'’“ (and hexversion, version_info, you get the idea)?

> How about this phrasing: “Make 2to3 fixers not touch code in a block
> starting with ’if sys.version >= '3'’“ (and hexversion, version_info,
> you get the idea)?

I don't think this can work. You may have to write code like

if sys.version_info >= (3,):
  try:
    some_code()
  except Exception, e:
    pass

(i.e. not use the "as" syntax), because it otherwise won't parse on
Python 2. Hence, one has to rely on 2to3 fixing it, even though
it will never be run on Python 2.

So any scheme of skipping code must be opt-in.

While I now understand what is being requested, I still fail to see
the rationale. In my applications of 2to3, I never look at the generated
code, so it doesn't bother me at all if print gets another pairs of
brackets, or if redundant (but dead) import statements are inserted.

In fact, I also avoid writing code explicitly so that it works
unmodified on Python 3 if I know that 2to3 will fix it.

There has been a long-standing request to suppress 2to3 in certain
expressions in cases where the 2to3 conversion would actually be
incorrect. However, this does not seem to be the motivation here.

[I got failed issue tracker submission, trying again...]

>> How about this phrasing: “Make 2to3 fixers not touch code in a
>> block starting with ’if sys.version >= '3'’“
>> (and hexversion, version_info, you get the idea)?

Right, almost... it doesn't need to be flexible, it only needs to be
documented for 2to3.  So it also doesn't need to handle the varions
version variables - if e.g. version_info is easiest for 2to3, it need
only handle that.  I've been posting with routine avoidance of testing
versions instead of features (learned from javascript frustration:-)
but that concern was misplaced here.

But it should be "if sys.version {either < or >=} '3':", and it should
not touch the else: block either.  Just trust that the programmer has
written correct version's code for each block, and parse one of the
blocks to pick up whatever info 2to3 needs to process the rest of the
python file.

So...

> I don't think this can work. You may have to write code like
> 
> if sys.version_info >= (3,):
>   try:
>     some_code()
>   except Exception, e:
>     pass
> 
> (i.e. not use the "as" syntax), because it otherwise won't parse on
> Python 2. Hence, one has to rely on 2to3 fixing it, even though
> it will never be run on Python 2.

I assume it should be "if sys.version_info < (3,):" since that looks
like Python 2 code, and that'll work with the above revised suggestion.

> So any scheme of skipping code must be opt-in.

Fair enough, if it's a 2to3 option which to obey whatever "skip some
code" hack is defined.  That's the same 2to3 command line for a lot of
files, instead of different commmand lines for different files.

Unless we're still talking past each other - if the example code will
never run on Python 2 as you say, there's no reason not to fix syntax
problems like the above.  It's fixing things like bytes vs str which
takes more thought.

> While I now understand what is being requested, I still fail to see
> the rationale. In my applications of 2to3, I never look at the generated
> code, so it doesn't bother me at all if print gets another pairs of
> brackets, or if redundant (but dead) import statements are inserted.

Wow.  We live in different mental worlds.  It would not have occurred
to me to take the 2to3 output as more than helpful suggestions.  Some
to be applied straight (like 'except' syntax), other to maybe apply
but also look closer at nearby code.

Indeed, one of my early 2to3 experiences led to a bug in python 3 code
which I'm now discussing in comp.lang.python.  Also my internal bug
detector zooms in on ((foo)) when I read Python code - I'm seeing code
where something was apparently left out, maybe an inner comma to make it
a tuple.

-- 
Hallvard

>> I don't think this can work. You may have to write code like
>>
>> if sys.version_info >= (3,):
>>   try:
>>     some_code()
>>   except Exception, e:
>>     pass
>>
>> (i.e. not use the "as" syntax), because it otherwise won't parse on
>> Python 2. Hence, one has to rely on 2to3 fixing it, even though
>> it will never be run on Python 2.
> 
> I assume it should be "if sys.version_info < (3,):" since that looks
> like Python 2 code, and that'll work with the above revised suggestion.

No, I meant this as stated. Suppose you would write (as you apparently
expected me to)

if sys.version_info >= (3,):
   try:
     some_code()
   except Exception as e:
     pass

then the entire module (and not just this if-block) will fail to import
in Python 2. Therefore, you must not use syntax that is exclusively
Python 3 in an if-python3 block if you want to continue to use the code
in Python 2 (and if you don't want to use the code in Python 2, you
don't need the if block).

Then you run the code (as I originally posted) through 2to3, and out
you get the block that will then get executed in Python 3.
Therefore, it is necessary to convert the code that is meant for
Python 3 with 2to3 still.

>> While I now understand what is being requested, I still fail to see
>> the rationale. In my applications of 2to3, I never look at the generated
>> code, so it doesn't bother me at all if print gets another pairs of
>> brackets, or if redundant (but dead) import statements are inserted.
> 
> Wow.  We live in different mental worlds.  It would not have occurred
> to me to take the 2to3 output as more than helpful suggestions.  Some
> to be applied straight (like 'except' syntax), other to maybe apply
> but also look closer at nearby code.

So please reconsider. Using that approach will allow you to have a
single source code for Python 2 and Python 3. You write it so that
it works fine on Python 2, and let 2to3 generate the Python 3 version,
which you then run unmodified. For that to work, it's important that
any modifications that 2to3 won't do will be done *before* invoking
2to3, and these modifications must therefore then work with Python 2
as well (albeit possibly in a block that is never executed on Python 2).

If, at some point, you are then ready to burn the bridges (i.e. give
up Python 2 support), you run 2to3 once, and start removing all
ugliness that you had collected during the transition phase.

Hi, I'm back...  I've been reading your last message a few times
and I'm not sure what I'm to reconsider.  We've had a way of
talking past each toerh before, as far as I can remember.

I think my request still now translates to:

- keep the current behavior by default,
- add some option which does not translate certain code blocks,
- these code blocks could be the syntax we've already discussed,
  or something else.

And I still don't understand the rationale for this request. Can you please post an example Python module that has this markup you are asking for, so I can show you how to achieve what you want without the markup?

Martin v. Löwis writes:
> And I still don't understand the rationale for this request. Can you
> please post an example Python module that has this markup you are asking
> for, so I can show you how to achieve what you want without the markup?

Eh.  I thought you said you did understand.  But here is one:
    http://folk.uio.no/hbf/ZipHTTPServer.py

I think 2to3 is designed to take 2.x code and turn it into 3.x code.  Codebases using tricks and hacks to support both 2.x and 3.x (like the example you linked to) cannot be handled by 2to3.  You have to make a choice between manual maintenance of parallel codebases, automatic 2to3 translation or cross-version compatibility.

I don't understand the example, either. If I run 2to3 on it, I get, as the only change

 if sys.version_info[0] < 3:             # Python 2
-    from urllib           import quote, unquote
-    from urlparse         import urlparse, urlunparse
-    from BaseHTTPServer   import HTTPServer
-    from SimpleHTTPServer import SimpleHTTPRequestHandler
+    from urllib.parse import quote, unquote
+    from urllib.parse         import urlparse, urlunparse
+    from http.server   import HTTPServer
+    from http.server import SimpleHTTPRequestHandler
     def str2data(s, is_nonascii=re.compile("[^\0-\x7F]").search):

As this code is in a Python 2.x block: why does this change cause problems to you? You are supposed to run the 2to3 result in Python 3, and this conversion result will run correctly in Python 3.

> Can you please post an example Python module that has this markup you are asking for, so I can show you how to achieve what you want without the markup?

Hi, Martin. Here's a real-world example:
http://www.sqlalchemy.org/trac/browser/lib/sqlalchemy/types.py?rev=6884:b181f1e53603

The syntax is:
# Py3K
# <python 3 code>
# Py2K
<python 2 code>
# end Py2K

For example, starting on line 152 we have,
        # Py3K
        #return unicode(self.compile())
        # Py2K
        return unicode(self.compile()).\
                        encode('ascii', 'backslashreplace')
        # end Py2K

When the code is converted with 2to3, the py3k code on line 153 will be uncommented and used to replace the py2k code on lines 155-156. Having the py3k version commented before conversion resolves the issue that "you must not use syntax that is exclusively Python 3 in an if-python3 block".

Here is their modified version of 2to3 to support this syntax:
http://www.sqlalchemy.org/trac/browser/sa2to3.py

Their explanation is:
"This tool monkeypatches a preprocessor onto lib2to3.refactor.RefactoringTool, so that conditional sections can replace non-fixable Python 2 code sections for the appropriate Python 3 version before 2to3 is run."

> For example, starting on line 152 we have,
>         # Py3K
>         #return unicode(self.compile())
>         # Py2K
>         return unicode(self.compile()).\
>                         encode('ascii', 'backslashreplace')
>         # end Py2K

Ok, I can propose two different spellings of this without any
macro processor:

A. conditionally encode for 2.x

   def __str__(self):
       res = unicode(self.compile())
       if sys.version_info < (3,):
           res = res.encode('ascii', 'backslashreplace')
       return res

B. (if A is deemed to incur too much per-call cost, due to
   the version test)

   if sys.version_info >= (3,):
       def __str__(self):
           return unicode(self.compile())
   else:
       def __str__(self):
           return unicode(self.compile()). \
                  encode('ascii', 'backslashreplace')

In addition, I fail to see how the markup hfuru proposed can be
used to express this case. IIUC, he asks for markup that can tell
2to3 to leave a certain piece of code alone, i.e. unmodified. I
fail to see how this would help in this sqlalchemy example.

Consider here:
http://www.sqlalchemy.org/trac/browser/lib/sqlalchemy/engine/base.py?rev=6884:b181f1e53603#L1329
the py3k code uses the "raise ... from" syntax which isn't legal in Python 2. Using either  "Approach A" or "Approach B" would prevent the program from working in py2k due to syntax errors.

I agree that the markup hrufu proposed didn't solve that problem either. SQLA's trick of having the py3k code commented out is the critical part that they add.

Additionally, the runtime cost prevents Approach A from being general solution for libraries like SQLA who have put a lot of work into reducing the number of function calls and checks in their critical paths.

Approach B isn't a general solution because it requires you to replace entire functions at a time. If you have a 25 line function where only 1 line needs to be changed for py3k, you either end up with 24 duplicated lines of code, or you have to factor that single line out into its own function, confusing the flow of the code and incurring runtime cost.

> Consider here:
> http://www.sqlalchemy.org/trac/browser/lib/sqlalchemy/engine/base.py?rev=6884:b181f1e53603#L1329
> the py3k code uses the "raise ... from" syntax which isn't legal in Python 2. 

In this case, I would write

   error = exc.DBAPIError.instance(statement,
                            parameters,
                            e,
                            connection_invalidated=is_disconnect)
   if sys.version_info < (3,):
       raise error, None, sys.exc_info()[2]
   else:
       error.__cause__ = e
       raise error

You don't *have* to use the from syntax to set the cause.

> Approach B isn't a general solution because it requires you to replace entire functions at a time. 

I don't claim that. However, I claim that there will be always an
appropriate solution using existing techniques, so that such a macro
processing wouldn't be necessary. IOW, I'm not aware of a case where
using such preprocessing would be appropriate and better than what
you can do without.

Martin v. Löwis writes:
> As this code is in a Python 2.x block: why does this change cause
> problems to you? You are supposed to run the 2to3 result in Python 3,
> and this conversion result will run correctly in Python 3.

As I've explained, and as the if-statement and the associated comment
expresses, that piece of code is intended to work with both Python 2
and 3.

2to3 turns it into code which would break on Python 2.  I've explained
why I'd run 2to3 on such code, and thus why it'd be convenient if I
could ask 2to3 to leave such code alone.  So I requested this feature.
I don't know exactly how I'm "supposed" to use 2to3.  If you are its
inventor, you do.  Otherwise it may also be supposed to suit other
workflows than yours.

> Ok, I can propose two different spellings of this without any
> macro processor: (...)

Both your examples fit my request perfectly.  Pieces of code which I
presume are correct for both Python 2 and 3, and 2to3 break them for
both Python versions.  With "my" feature, the code would keep working
with both.

However, it is true that there are other cases where I'd like to shut
up 2to3 but where my suggested solution would not be convenient to
use.  I am after all requesting a mere convenience feature, and aimed
for something I hoped would be a simple matter to implement.  I never
imagined that merely explaining it would grow into such a discussion.

Bobby, thanks for the sa2to3 reference.  Doesn't quite do what I
wanted, but looks useful and might also be a good starting point for
what I did ask for.  Hopefully I won't need to understand too much
2to3 code first...

Hallvard B Furuseth writes:
>Martin v. Löwis writes:
>> Ok, I can propose two different spellings of this without any
>> macro processor: (...)
> 
> Both your examples fit my request perfectly.  Pieces of code which I
> presume are correct for both Python 2 and 3, (...)

Sorry, ignore that.  I "saw" you doing what I would be doing, not what
you were doing...  Not example A.  And Example B looks (to me) like it
is intended to work unmodified for both Python 2 and 3, but doesn't.

Éric Araujo writes:
> I think 2to3 is designed to take 2.x code and turn it into 3.x code.
> Codebases using tricks and hacks to support both 2.x and 3.x (like the
> example you linked to) cannot be handled by 2to3.

That's fair enough.

> That's fair enough.

:) Do you want to close this feature request then?

Éric Araujo writes:
>> That's fair enough.
> 
> :) Do you want to close this feature request then?

Me?  No.  I just figured that after all this arguing, I should mention
that closing it as out of scope is not something I'll be difficult about.