On Fri, Feb 4, 2011 at 11:35 AM, Georg Brandl <report@bugs.python.org> wrote:

>.. But seeing the history of this case, I don't want to play around here before 3.2 final.

Here is my understanding of the history of this case:  tmp1.patch was
applied in r65539 and reverted in r65543 with the log entry saying:

------------------------------------------------------------------------
r65543 | andrew.kuchling | 2008-08-04 22:05:23 -0400 (Mon, 04 Aug 2008) | 1 line

#3367: revert rev. 65539: this change causes test_parser to fail
------------------------------------------------------------------------

Revision 65539 has never been applied to py3k, but it would be
equivalent to the following diff:

Index: Parser/tokenizer.c
===================================================================
--- Parser/tokenizer.c	(revision 88320)
+++ Parser/tokenizer.c	(working copy)
@@ -1289,7 +1289,7 @@
     register int c;
     int blankline, nonascii;

-    *p_start = *p_end = NULL;
+    tok->line_start = *p_start = *p_end = NULL;
   nextline:
     tok->start = NULL;
     blankline = 0;

Applying the above diff now makes test_parser crash on a debug and
fail on a regular build.  The problem with initializing
tok->line_start to NULL is that doing so trades one undefined behavior
for another: pointer comparison such as a >= tok->line_start is only
defined if both pointers point to the same buffer.  Ordering between
NULL and non-NULL pointers is undefined.  My patch does not have this
issue because it initializes tok->line_start to tok->buf.