classification
Title: tokenize bug
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: tim.peters Nosy List: BreamoreBoy, aguiar, rhettinger, tim.peters
Priority: normal Keywords: patch

Created on 2005-06-30 17:35 by aguiar, last changed 2010-08-21 17:06 by BreamoreBoy. This issue is now closed.

Messages (3)
msg25692 - (view) Author: Eduardo Aguiar (aguiar) Date: 2005-06-30 17:35
hi,

I have found a bug in 'tokenize' module: it is merging
a COMMENT and a NL token for lines that start with a
comment.

I have made a fell changes and it seems to be working
fine. Follows a patch:

 *** /usr/lib/python2.4/tokenize.py      2005-01-02
03:34:20.000000000 -0200
--- tokenize.py 2005-06-30 14:31:19.000000000 -0300
***************
*** 216,223 ****
                  pos = pos + 1
              if pos == max: break
  
!             if line[pos] in '#\r\n':           # skip
comments or blank lines
!                 yield ((NL, COMMENT)[line[pos] ==
'#'], line[pos:],
                             (lnum, pos), (lnum,
len(line)), line)
                  continue
  
--- 216,235 ----
                  pos = pos + 1
              if pos == max: break
  
!             if line[pos] == '#':           # skip
comments
!                 end = len(line) - 1
!                 while end > pos and line[end] in '\r\n':
!                    end = end - 1
!                 end = end + 1
! 
!                 yield (COMMENT, line[pos:end],
!                            (lnum, pos), (lnum, end),
line)
!                 yield (NL, line[end:],
!                            (lnum, end), (lnum,
len(line)), line)
!                 continue
! 
!             if line[pos] in '\r\n':           # skip
blank lines
!                 yield (NL, line[pos:],
                             (lnum, pos), (lnum,
len(line)), line)
                  continue

Best regards,
Eduardo Aguiar (eaguiar@programmer.net)
msg25693 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2005-07-15 09:00
Logged In: YES 
user_id=80475

Tim, this affects a line you checked-in, 1.23 on 6/18/2001.
msg114517 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-21 17:06
Fixed in r51526.
History
Date User Action Args
2010-08-21 17:06:34BreamoreBoysetstatus: open -> closed

nosy: + BreamoreBoy
messages: + msg114517

dependencies: - tokenize: mishandles line joining
resolution: fixed
2009-02-16 02:26:11ajaksu2setkeywords: + patch
versions: + Python 2.6, - Python 2.4
dependencies: + tokenize: mishandles line joining
type: behavior
stage: test needed
2005-06-30 17:35:21aguiarcreate