➜
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2005-06-30 17:35 by aguiar, last changed 2022-04-11 14:56 by admin. This issue is now closed.
| Messages (3) | |||
|---|---|---|---|
| msg25692 - (view) | Author: Eduardo Aguiar (aguiar) | Date: 2005-06-30 17:35 | |
hi,
I have found a bug in 'tokenize' module: it is merging
a COMMENT and a NL token for lines that start with a
comment.
I have made a fell changes and it seems to be working
fine. Follows a patch:
*** /usr/lib/python2.4/tokenize.py 2005-01-02
03:34:20.000000000 -0200
--- tokenize.py 2005-06-30 14:31:19.000000000 -0300
***************
*** 216,223 ****
pos = pos + 1
if pos == max: break
! if line[pos] in '#\r\n': # skip
comments or blank lines
! yield ((NL, COMMENT)[line[pos] ==
'#'], line[pos:],
(lnum, pos), (lnum,
len(line)), line)
continue
--- 216,235 ----
pos = pos + 1
if pos == max: break
! if line[pos] == '#': # skip
comments
! end = len(line) - 1
! while end > pos and line[end] in '\r\n':
! end = end - 1
! end = end + 1
!
! yield (COMMENT, line[pos:end],
! (lnum, pos), (lnum, end),
line)
! yield (NL, line[end:],
! (lnum, end), (lnum,
len(line)), line)
! continue
!
! if line[pos] in '\r\n': # skip
blank lines
! yield (NL, line[pos:],
(lnum, pos), (lnum,
len(line)), line)
continue
Best regards,
Eduardo Aguiar (eaguiar@programmer.net)
|
|||
| msg25693 - (view) | Author: Raymond Hettinger (rhettinger) *  |
Date: 2005-07-15 09:00 | |
Logged In: YES user_id=80475 Tim, this affects a line you checked-in, 1.23 on 6/18/2001. |
|||
| msg114517 - (view) | Author: Mark Lawrence (BreamoreBoy) * | Date: 2010-08-21 17:06 | |
Fixed in r51526. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:56:12 | admin | set | github: 42147 |
| 2010-08-21 17:06:34 | BreamoreBoy | set | status: open -> closed nosy: + BreamoreBoy messages: + msg114517 dependencies: - tokenize: mishandles line joining resolution: fixed |
| 2009-02-16 02:26:11 | ajaksu2 | set | keywords:
+ patch versions: + Python 2.6, - Python 2.4 dependencies: + tokenize: mishandles line joining type: behavior stage: test needed |
| 2005-06-30 17:35:21 | aguiar | create | |