Title: PEP 269 Implementation
Type: Stage:
Components: Interpreter Core Versions: Python 2.3
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, jriehl, loewis
Priority: low Keywords: patch

Created on 2002-08-23 17:03 by jriehl, last changed 2004-08-19 19:02 by jriehl. This issue is now closed.

File name Uploaded Description Edit
pep269.tar.gz jriehl, 2002-08-23 17:03 Gzip'ed tar file with patch diffs, extension modules (new to core), and examples/tests.
pep269.tar.gz jriehl, 2003-04-01 01:52 Update to previous u/l.
Messages (15)
msg41006 - (view) Author: Jon Riehl (jriehl) Date: 2002-08-23 17:03
The following are files for the implementation of PEP
269.  The primary changes to the core involve moving
required pgen modules from the pgen only module list to
the Python library module list in 
Some of the modules required better memory allocation
and management and the corresponding deallocators are
in the Python extension module (maybe these should be
moved into the pgen modules in Parser).

Initially included are two implementations.  The first
is a basic implementation that follows the PEP API more
or less.  The second (currently unfinished)
implementation provides a more object oriented
interface to the extension module.

Please note there are some commented out modifications
to as I was unable to build the extension
module(s) automagically.  For some reason the linker I
was using (on a BSD box) complained that it couldn't
find the _Py_pgen symbol, even though I verified its
existence in the new Python library.  Maybe it is
checking against an older Python library on the system?

Things to be done (as of initial submission) include
resolving on a single interface, documenting the one
true interface, and finishing any unimplemented
routines (such as are found in the OO implementation).

In the final integration, a pgenmodule.c file should be
added to the Modules directory in the main distribution.
msg41007 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-09-03 19:44
Logged In: YES 

I guess I'm going to hve to look at this to pronounce...
msg41008 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-09-09 12:09
Logged In: YES 

I looked a bit, and it's very raw - the author admits that.
Jon, are you still working on this? Do you have a more
polished version?

Maybe we can separate out the memory allocation patches and
commit these already? They don't break anything.
msg41009 - (view) Author: Jon Riehl (jriehl) Date: 2002-09-12 18:04
Logged In: YES 

Guido, as per my private message, I'll attempt to submit
another patch by the end of the month, pending resumption of
"work" on the 23rd.  Commitment of the memory allocation
patch is fine, and any future patches would be against the
updated pgen code (I don't have commit permissions, so
someone else will have to do this possibly.)
msg41010 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-09-13 04:35
Logged In: YES 

Cool. Maybe I'll get to it, maybe not. :-(
msg41011 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-11-14 17:32
Logged In: YES 

Lowering priority until Jon has his next version ready.
msg41012 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2003-02-05 16:42
Logged In: YES 

Closed. Jon, if you're still interested in this, upload
something new and reopen the patch (or I can do that).
msg41013 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2003-04-14 20:44
Logged In: YES 

OK, reading the README now...
msg41014 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2003-04-17 15:12
Logged In: YES 

I've checked in the changes to the parser and Makefile etc.
This means you should be able to distribute pgenmodule.c as
a clean extension to Python 2.3.

I've decided not to incorporate pgenmodule.c at this time.
There are three problems with it as I see it:

- The C coding style is really different, and makes it hard
to read: 2-space indents instead of tabs, spaces before left
parentheses in argument lists, space between * and name in
things like "PyObject * name", placement of curly braces,
reversal of arguments to comparisons (e.g. "NULL != x"), and
long strings of underscores in comments.

- The code isn't robust enough: I got a segfault (can't
remember after what sequence of events, sorry), and it
caused a SystemError (error return without exception set)
when the grammar file couldn't be read.

- The pgen API seems a bit strange -- some things that I
would expect to be methods of AST objects are functions in
the pgen module (even when a companion function *is* an AST

- The unit test doesn't fit in the Python test framework.

- I cooked up a little expression grammar and tried to parse
a sample string, but all I got was exceptions. I guess I am
missing something, but there's no tutorial that explains me
how to do things (I was copying the code from
which works :-( ).

Nevertheless, thanks for the work so far -- I'm sure at some
point pgen will be usable from Python!
msg41015 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-10-31 13:54
Logged In: YES 

Jon, are you still interested in taking this further?
msg41016 - (view) Author: Jon Riehl (jriehl) Date: 2003-11-01 23:20
Logged In: YES 

I am to some extent.  I've done several wacky things, namely
putting the pgen extension module into my Basil project
(basically I think PEP 269 is dead), as well as re-written
pgen in "pure" Python (no C extension module calls)   The
pure Python pgen looks like it will be going into Pypy.  I
don't know if I am interested in the speed enough to warrant
serious analysis of the pgen C extension and the
segmentation fault Guido reported.  Let me know if you had
anything specific in mind.
msg41017 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-11-02 18:35
Logged In: YES 

I was wondering what the possible next steps could be. With
your current analysis, it appears that
1. declare PEP 269 withdrawn
2. reject this patch
would be a possibility. I would want to keep this open only
if some volunteer would indicate interest in completing this
msg41018 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2004-08-18 11:57
Logged In: YES 

I have now marked the PEP as deferred, until some volunteer
picks it up again. I'm rejecting this patch (but left a
pointer in the PEP for reference).
msg41019 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2004-08-19 03:57
Logged In: YES 

FWIW, I *also* rewrote pgen in Python. Is the Pypy version
real? If so, I ma not have to publish my code (which is
owned by Elemental Security but if there's demand I can
attempt to open-source it).
msg41020 - (view) Author: Jon Riehl (jriehl) Date: 2004-08-19 19:02
Logged In: YES 

It may not be super usable (I believe it is as usable as
pgen is, which is just usable enough to automate Python
parser generation), but I consider it finished.  The SVN URL
is now:
Date User Action Args
2002-08-23 17:03:19jriehlcreate