classification
Title: backport python 3.0 language functionality to python 2.6 by adding 7 opcodes to ceval.c
Type: enhancement Stage:
Components: Demos and Tools, Extension Modules, Interpreter Core, Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: benjamin.peterson, collinwinter, gvanrossum, kaizhu, loewis, pitrou, rhettinger
Priority: low Keywords: patch

Created on 2008-06-29 20:07 by kaizhu, last changed 2008-10-24 21:58 by loewis. This issue is now closed.

Files
File name Uploaded Description Edit
ceval.2.6b1.c kaizhu, 2008-06-29 20:07 python 2.6b1 ceval.c with 7 backported opcdes
README.txt kaizhu, 2008-10-20 23:39 README from http://www-rcf.usc.edu/~kaizhu/work/py3to2/
ceval.20081020.diff kaizhu, 2008-10-21 01:57 20081020 patch to ceval.c
py3to2.py kaizhu, 2008-10-21 06:30 py3to2 script (used in conjunction w/ patched ceval.c)
example_py3k.py kaizhu, 2008-10-21 06:56 example "python 3.0" script (using numpy extension module) & runnable under a patched python2.6
img2txt mario.jpg screenshot.gif kaizhu, 2008-10-24 18:17 screenshot of a picture converted to colorized ascii on putty ssh terminal
img2txt 3dplot screenshot.gif kaizhu, 2008-10-24 18:20 screenshot of colorized 3d plots in putty ssh terminal
img2txt.py kaizhu, 2008-10-24 18:21 python 3.0 script (+curry extension) using 2.6 numpy, PIL, & scipy.weave extensions
mario.jpg kaizhu, 2008-10-24 18:28 u need this jpeg file for img2txt.test() to work
Messages (20)
msg68964 - (view) Author: kai zhu (kaizhu) Date: 2008-06-29 20:06
this patch touches only Python/ceval.c.

1. the only existing thing it modifies is PyEval_EvalFrameEx (adds 7
extra cases for the new 3.0 opcodes, doesn't mess w/ any of the existing
ones, or anything else as a matter of fact)

2. that, plus it defines (the new opcodes)
#define SET_ADD 17
#define STORE_LOCALS 69
#define LOAD_BUILD_CLASS 34 // 71 in py3k
#define MAKE_BYTES 35 // unused in py3k
#define POP_EXCEPT 36 // 89 in py3k
#define UNPACK_EX 94
#define BUILD_SET 109 // 104 in py3k
and some backported vars & helper functions

only 2 contiguous areas of ceval.c is patched (1. & 2. - areas are
delimited by the comments '// py3k')

this simple patch seems sufficient in theory to allow the interpreter to
natively execute most of 3.0's language syntax (nonlocal, print,
extended unpacking, ... have been tested to work)
*the one exception being pep3102 - keyword-only arguments

i wrote 2 small scripts which gives an interactive function 'compile_py3k'
similar to __builtin__.compile.  its a wrapper function which queries
the byte-compiling task to a python 3.0 server via pipe io.

example demonstrating pep3132 extended unpacking syntax:
a,b,*c = 1,2,3,4
(note the backported 3.0 opcode used in line2 of the disassembly)



Python 2.5.2 (r252:60911, Jun 27 2008, 21:19:51)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import py3to2
py3to2 server restarting with io: (4, 5)
py3to2 server: Python 3.0b1 (r30b1:64395, Jun 24 2008, 21:53:55)
py3to2 server: [GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
py3to2 server: Type "help", "copyright", "credits" or "license" for more
information.
py3to2 server:

>>> src = "a,b,*c = 1,2,3,4"
>>> codeobject = py3to2.compile_py3k(src,"","exec")
 1           0 LOAD_CONST               5 ((1, 2, 3, 4))
             3 UNPACK_EX_py3k           2
             6 STORE_NAME               0 (a)
             9 STORE_NAME               1 (b)
            12 STORE_NAME               2 (c)
            15 LOAD_CONST               4 (None)
            18 RETURN_VALUE

>>> exec(codeobject)
>>> print a,b,c
1, 2, [3, 4]
>>>

u can go c
http://pypi.python.org/pypi?name=py3to2&version=20080628&:action=display
for more info on the script

anyway, i think it would b a great boost for python 3.0 (which i think
is very cool) if developers can test/develop it under the 2.x
environment.  this patch plus some scripts to emulated 3k extensions 
(bytes, bytearray, ...) can go a long way in making 3.0 available for
the masses.

ps. i've also attempted the reverse (forward-port 2.x opcodes to 3.0),
& its a bit more complicated (namely the PRINTxxx opcodes).  if there's
significant interest in that, i could work on it a bit more, but my
current interest is in extending 3.0 functionality to the vast 2.x
software base.
msg68971 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-06-29 21:43
The first beta release for 2.6 has been made, so no new features can be
accepted here. Deferring this to 2.7.
msg68988 - (view) Author: kai zhu (kaizhu) Date: 2008-06-30 02:43
ok
msg69006 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-06-30 14:27
Some remarks:

- I don't think doing a bulk backport of ceval.c is the right approach.
Instead, each functionality should be considered and backported
independently, if desired.

- Guido already said that 3.0 should be mostly clean from
compatibility-related code, so forward-porting 2.x opcodes is out of the
question. There's a reason they removed in the first place :-)

- Some functionalities shouldn't be backported, e.g. introducing
SETUP_EXCEPT was motivated by the different semantics of exception
cleanup in py3k, backporting it would probably break some 2.x code.

- Compatibility between the two eval loops (2.x and py3k) is probably
the tip of the iceberg.
msg69023 - (view) Author: kai zhu (kaizhu) Date: 2008-06-30 22:39
ideally that may be true.

but its quite frustrating testing/developing new py3k software when many
modules/extensions we take for granted in 2.x isn't available.  in the
meantime, this patch serves as a very convenient stopgap for developers
(even w/ its bugs), for writing py3k migration code in 2.x (& access to
all its extensions)

for example, its a difficult task to port a big project like numpy to
py3k all @ once.  but this patch could allow u to piece-wise transform
it, one script @ a time, to py3k language syntax compliance.
msg69116 - (view) Author: Collin Winter (collinwinter) * (Python committer) Date: 2008-07-02 20:26
I don't know why this is assigned to me.
msg69461 - (view) Author: kai zhu (kaizhu) Date: 2008-07-09 11:27
update: these 3k language features have been tested to work in python
2.5.2 w/ the backported opcodes

   pep3104  Access to Names in Outer Scopes
   pep3105  Make print a function
   pep3111  Simple input built-in in Python 3000
   pep3113  Removal of Tuple Parameter Unpacking
   pep3114  Renaming iterator.next() to .__next__()
   pep3115  Metaclasses in Python 3000
   pep3127  Integer Literal Support and Syntax
   pep3129  Class Decorators
   pep3132  Extended Iterable Unpacking

had to backport __build_class__ in bltinmodule.c to get metaclasses
working.  u can go to http://www-rcf.usc.edu/~kaizhu/work/py3to2/ for
more info

install/usage summary:
  1. build python 2.5.2 w/ patched ceval.c & bltinmodule.c
  2. download scripts: py3to2.py & _py3to2.py
  3. ln -s python3.0 python3k
  4. run python2.5 & type 'import py3to2' # it will automatically run
the pep tests

the script provides 3 functions similar to those in __builtin__:
compile_py3k, exec_py3k, eval_py3k

right now, working on backporting pep 3102 & 3107 - annotations &
keyword-only arguments.  apparently appending the co_kwonlyargcount attr
to codeobject in 2.5.2 doesn't seem to affect the build
msg69611 - (view) Author: kai zhu (kaizhu) Date: 2008-07-13 16:07
import/reload now works.
accomplished by adding 5 lines in parse_source_module (import.c) to 1st
check for the hook __builtins__.parse_source_module_py3k.
the hook will automatically compile in py3k format if it finds the magic
comment: "# import as py3k\n"

* below is a working python 3.0 script integrated w/ numpy.

also added:
  pep3102  Keyword-Only Arguments
  pep3112  Bytes literals in Python 3000

download: http://www-rcf.usc.edu/~kaizhu/work/py3to2/current/

patched files:
  ceval.c (unchanged from last)
  bltinmodule.c (unchanged from last)
  import.c (added 5 continuous lines to parse_source_module)

there r 7 unimplemented pep's remaining: any suggested solutions?
  pep3107  Function Annotations (easy, just haven't gotten around yet)
  pep3109/3110  Exceptions in Python 3000 (hard?)
  pep3120  Using UTF-8 as the default source encoding
  pep3123  Making PyObject_HEAD conform to C (hard/who cares for 2.x?)
  pep3131  Supporting Non-ASCII Identifiers (looks emulable)
  pep3138  String representation in Python 3000

@ any rate, i think its feature complete enough to b useful in certain
areas (for me its scientific computing).

################################################################################
"""
numpy_py3k.py
this is a py3to2 demo showing a python3.0 script being run under python
2.5 &
utilizing numpy, a python2.5 extension module.

add the magic comment '# import as py3k\n' to import / reload a script
in py3k format

interactive usage:
>>> import py3to2
>>> import numpy_py3k
>>> reload(numpy_py3k)

commandline usage: python -c 'import py3to2; import numpy_py3k'
"""

# import as py3k
import numpy

print('pep3102  Keyword-Only Arguments')
# nth order polynomial fit of multiple y data
def polyfits(nth, x, *ys, rcond = None, full = False):
  return [numpy.polyfit(x, y, nth, rcond, full) for y in ys]

fits = polyfits(2, # 2nd order fit
                numpy.arange(16), # x data
                numpy.random.rand(16), numpy.random.rand(16), # multiple
y data
                rcond = numpy.MachAr().eps, # precision
                full = False, # return only coeffs
                )
print('fits', fits); print('#'*64)

print('pep3112  Bytes literals in Python 3000')
x = bytes( numpy.arange(256, dtype = numpy.int8).tostring() )
print('bytes', x); print('#'*64)

print('pep3114  Renaming iterator.next() to .__next__()')
x = (x for x in numpy.arange(16))
print('x.__next__()', x.__next__(), x.__next__(), x.__next__());
print('#'*64)

print('pep3132  Extended Iterable Unpacking')
a,b,*c = numpy.random.rand(4)
print('a = %s, b = %s, c = %s'%(a,b,c)); print('#'*64)
################################################################################
msg69612 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-07-13 17:01
This isn't dealing with the 2to3 conversion tool. Please do not add it.
msg69618 - (view) Author: kai zhu (kaizhu) Date: 2008-07-13 19:01
why not?  it allows developers to migrate 2.x scripts one-by-one to
working 3.0 conformant ones while maintaining backwards-compatibility w/
existing 2.x scripts & extension modules (eg. numpy, PIL, zope, ...)

py3to2 can transparently import & mix & match 2.x & 3.0 scripts (but
builtins/extensions must b 2.x - hence its a 2to3 migration tool).

@ the moment, every script compilable by py3to2 should b 3.0 language
conformant, or otherwise it would fail the syntax check & byte-compile
stage performed by the python 3.0 slave interpreter (see Mechanism for
details).
msg69619 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-07-13 19:05
On Sun, Jul 13, 2008 at 2:03 PM, kai zhu <report@bugs.python.org> wrote:
>
> kai zhu <davidbranniganz@gmail.com> added the comment:
>
> why not?  it allows developers to migrate 2.x scripts one-by-one to
> working 3.0 conformant ones while maintaining backwards-compatibility w/
> existing 2.x scripts & extension modules (eg. numpy, PIL, zope, ...)
>
> py3to2 can transparently import & mix & match 2.x & 3.0 scripts (but
> builtins/extensions must b 2.x - hence its a 2to3 migration tool).

Yes, I realize that, but there is another tool called 2to3 that
preforms syntax transformations on 2.x code. That's what the "2to3
conversion" tool component is for.
msg69624 - (view) Author: kai zhu (kaizhu) Date: 2008-07-13 20:58
then the 2 can complement each other well ;) i don't c any competition
between them as they have completely different objectives

2to3 -> convert 2x scripts to 3k
py3to2 <- use the newly created 3k scripts in existing 2x environ

u have to admit migrating to 3k is quite painful & slow
w/ its lack of 2x's vast extension base. given py3to2's potential to
alleviate this problem for migrating developers,
don't u think that possibly merits its mention in the 2to3 category?

btw, py3to2's patches minimally affects 2.x compatibility:
  ceval.c - backports missing 3.0 opcodes (8 in python 2.5.2)
  bltinmodule.c - backports missing 3.0 function __build_class__
  import.c - adds a 5 line hook to function parse_source_module
             allowing automatic checking for and compiling of 3k scripts
             during import/reload

i just ran 'make test' w/ a patched build of python 2.5.2 w/ following
results on redhat, x86_64:
285 tests OK.
37 tests skipped:
    test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
    test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dl test_gdbm test_gl test_imageop test_imgfile
    test_linuxaudiodev test_macfs test_macostools test_normalization
    test_ossaudiodev test_pep277 test_plistlib test_rgbimg
    test_scriptpackages test_socket_ssl test_socketserver
    test_startfile test_sunaudiodev test_timeout test_urllib2net
    test_urllibnet test_winreg test_winsound test_zipfile64
1 skip unexpected on linux2:
    test_gdbm

not bad considering it now supports 11 of the 18 (soon to b 12 & likely
more...) 3000 series pep's ^_^
msg69627 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-07-13 21:09
> don't u think that possibly merits its mention in the 2to3 category?

Please trust our judgment on how to use the bug tracker. This isn't a
competition for publicity; it's a tool to help organizing work among
committers and other contributors.
msg75016 - (view) Author: kai zhu (kaizhu) Date: 2008-10-20 23:39
ported to python-2.6 & is a bit more stable (written a few py3k programs
w/ it).  the patches have been simplified & consolidated to a single
file: ceval.c

u can also test out scripts generated by 2to3 by adding to them the
magic line:
"from __future__ import py3k_syntax"

an example "python-2.6" script using "python-3.0" language syntax (&
numpy if its installed) is provided @:
http://www-rcf.usc.edu/~kaizhu/work/py3to2/current/example_py3k.py

new features added by patch to python-2.6 (under py3k_syntax mode):
  pep3102  Keyword-Only Arguments
  pep3104  Access to Names in Outer Scopes
  pep3107  Function Annotations
  pep3111  Simple input built-in in Python 3000
  pep3113  Removal of Tuple Parameter Unpacking
  pep3114  Renaming iterator.next() to iterator.__next__()
  pep3115  Metaclasses in Python 3000
  pep3127  Integer Literal Support and Syntax
           - oct() functions differently in py3k_syntax mode
  pep3129  Class Decorators
  pep3132  Extended Iterable Unpacking
  pep3135  New Super
  misc     exec becomes a function

sans unicode & py3k's builtins module, the above (plus __future__)
implements pretty much all of py3k's language syntax

py3k_syntax mode:
  a special bitflag is embedded in codeobj->co_argcount, which triggers
PyEval_EvalFrameEx to enable the backported opcodes (which r disabled by
default) & use an alternate py3k __builtin__.

  codeobj->co_argcount now limited to range 0-0xffff with upper bits
reserved for py3k flags

  codeobj->co_kwonlyargcount (limited to range 0-0xff) is embedded in
upper bits of codeobj->co_argcount

most of the trouble have been gettin kwonlyarg, annotations, & classes
to work.

there is one unrelated opcode (LOAD_ATTR_py3k) added to ceval.c for my
personal use (curries attributes as a universal function), but can b
safely disabled by commenting its case out of PyEvalFrameEx in ceval.c &
deleting it from dict opnew_py2x in py3to2.py
msg75018 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-10-21 05:03
Recommend rejecting this.  I believe that it wasn't backported for a
reason.  Guido, care to pronounce?
msg75020 - (view) Author: kai zhu (kaizhu) Date: 2008-10-21 06:56
hi, i'm happy i got a response on this :)

anyway, can u elaborate on the "reason" y something like this was never
done?

yes, i kno the patch is rough right now (& will never get accepted in
its current state), but its now proven to work, & gives migrating
developers *something* to test their py3k_syntax code while *still*
maintaining compatibility w/ python2.x extension modules.

isn't that the spirit of python 2.7?  in the meantime, i'm in no hurry
to have anything committed.
msg75025 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-10-21 14:23
This doesn't sound like it will ever be reliable.

Kai Zhu, a better strategy would be for you to maintain your own
experimental port of Python 2.6+3.0 based upon these patches.
msg75044 - (view) Author: kai zhu (kaizhu) Date: 2008-10-21 20:59
k
msg75174 - (view) Author: kai zhu (kaizhu) Date: 2008-10-24 18:17
post fyi:

here's moderately complex python-3.0 script (plus curry extension) i
wrote using numpy, PIL, & scipy.weave 2.6 extensions.

it takes a jpeg, gif... image file & outputs it in colorized ascii art.
the actual purpose is for colorized 3-d scientific plottin in text
terminal (screenshots of image conversion & 3d plots in putty terminal
included)

usage:
python -c "import py3to2; import img2txt; img2txt.img2txt.test()"
python -c "import py3to2; import img2txt; img2txt.tplot3d.test()"
msg75185 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-10-24 21:58
> Added file: http://bugs.python.org/file11875/mario.jpg

Please don't use the bug tracker to upload unrelated code
(such as Numpy code - Numpy is not part of Python core).
I'm tempted to mark all these images as spam.
History
Date User Action Args
2008-10-24 21:58:42loewissetmessages: + msg75185
2008-10-24 18:28:24kaizhusetfiles: + mario.jpg
2008-10-24 18:21:33kaizhusetfiles: + img2txt.py
2008-10-24 18:20:10kaizhusetfiles: + img2txt 3dplot screenshot.gif
2008-10-24 18:17:33kaizhusetfiles: + img2txt mario.jpg screenshot.gif
messages: + msg75174
2008-10-21 20:59:25kaizhusetmessages: + msg75044
2008-10-21 14:23:41gvanrossumsetstatus: open -> closed
resolution: rejected
messages: + msg75025
2008-10-21 06:56:20kaizhusetfiles: + example_py3k.py
messages: + msg75020
2008-10-21 06:30:22kaizhusetfiles: + py3to2.py
2008-10-21 05:03:57rhettingersetpriority: low
assignee: gvanrossum
messages: + msg75018
nosy: + rhettinger, gvanrossum
2008-10-21 01:57:47kaizhusetfiles: + ceval.20081020.diff
keywords: + patch
2008-10-20 23:39:31kaizhusetfiles: + README.txt
messages: + msg75016
components: + Extension Modules, Library (Lib), - None
2008-07-13 21:09:50loewissetmessages: + msg69627
2008-07-13 20:58:25kaizhusetmessages: + msg69624
2008-07-13 19:05:23benjamin.petersonsetmessages: + msg69619
2008-07-13 19:01:58kaizhusetnosy: + benjamin.peterson
messages: + msg69618
components: + None
2008-07-13 17:01:26benjamin.petersonsetnosy: - benjamin.peterson
2008-07-13 17:01:07benjamin.petersonsetassignee: collinwinter -> (no value)
messages: + msg69612
components: - Tests, 2to3 (2.x to 3.x conversion tool)
nosy: + benjamin.peterson
2008-07-13 16:07:51kaizhusetassignee: collinwinter
messages: + msg69611
components: + Demos and Tools, Interpreter Core, Tests, 2to3 (2.x to 3.x conversion tool), - None
2008-07-09 11:27:52kaizhusetmessages: + msg69461
2008-07-02 20:26:39collinwintersetassignee: collinwinter -> (no value)
messages: + msg69116
components: + None, - 2to3 (2.x to 3.x conversion tool)
2008-06-30 22:39:32kaizhusetmessages: + msg69023
2008-06-30 14:27:47pitrousetnosy: + pitrou
messages: + msg69006
2008-06-30 02:43:13kaizhusetmessages: + msg68988
2008-06-29 21:43:04loewissetnosy: + loewis
messages: + msg68971
versions: + Python 2.7, - Python 2.6
2008-06-29 20:07:06kaizhucreate