classification
Title: curses.ascii.isblank() function is broken. It confuses backspace (BS 0x08) with tab (0x09)
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: akira, kevinpt, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2010-09-04 00:01 by kevinpt, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
curses_ascii_isblank_issue9770.patch akira, 2014-06-19 17:35 fix isblank, add tests for character classification functions from curses.ascii module review
curses-ascii-negative.patch serhiy.storchaka, 2016-12-17 10:51 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (6)
msg115544 - (view) Author: Kevin Thibedeau (kevinpt) Date: 2010-09-04 00:01
The isblank() function defined in curses.ascii is incorrect and doesn't match the output from C's isblank() from ctype.h

Incorrect definition:
  def isblank(c): return _ctoi(c) in (8,32)

Should be:
  def isblank(c): return _ctoi(c) in (9,32)

This most likely affects all versions of Python, not just 2.7.
msg220852 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-17 18:27
The problem and fix are simple but who is best placed to take a look at this?
msg221008 - (view) Author: Akira Li (akira) * Date: 2014-06-19 17:35
I've fixed isblank to accept tab instead of backspace and added tests 
for character classification functions from curses.ascii module that
have corresponding analogs in ctype.h. They've uncovered issues in 
isblank, iscntrl, and ispunct functions.

Open questions:

- is it a security bug (backspace is treated as tab in isblank())?
  If it is then 3.1, 3.2, 3.3 branches should also be updated
  [devguide]. If not then only 2.7, 3.4, and default branches should
  be changed.

  [devguide]: http://hg.python.org/devguide/file/9794412fa62d/devcycle.rst#l105

- iscntrl() mistakenly returns false for 0x7f but c11 defines it as
  a control character. Should iscntrl behavior (and its docs) be
  changed to confirm? Should another issue be opened?

- ispunct() mistakenly returns true for control characters such as
  '\n'. The documentation says (paraphrasing) 'any printable except
  space and alnum'. string.printable includes '\n' but 'printing
  character' in C11 does not include the newline. Moreover
  curses.ascii.isprint follows C behavior and excludes control
  characters. Should another issue be opened to return false from
  ispunct() for control characters such as '\n'?

- ispunct() mistakenly returns true for non-ascii characters such as
  0xff

- negative integer values: C functions are defined for EOF macros
  (some negative value) and the behavior is undefined for any other
  negative integer value. What should be curses.ascii.is* predicates
  behavior? Should Python guarantee that False is returned?

- curses.ascii.isspace/.isblank doesn't raise TypeError for bytes,
  None on Python 3

- should constants from string module be used? What is more
  fundamental: string.digits or curses.ascii.isdigit?

- no tests for: isascii, isctrl, ismeta (they are not defined in
  ctype.h). It is unclear what the behaviour should be e.g., isascii
  mistakenly returns True for negative ints, ismeta returns True for
  any non-ascii character including Unicode letters. It is not clear
  how isctrl is different from iscntrl.
msg223833 - (view) Author: Akira Li (akira) * Date: 2014-07-24 13:20
I've made the title more explicit: "curses.isblank function doesn't match
ctype.h" -> "curses.ascii.isblank() function is broken. It confuses
backspace (BS 0x08) with tab (0x09)"

If a core developer could review the open questions from the 
previous message msg221008 then I could prepare a proper patch for the 
issue.
msg283484 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-17 10:51
Most issues was fixed in issue27079. Except handling negative integers. Following patch fixes the latter issue.
msg284161 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-12-28 08:07
New changeset cba619a7bf6a by Serhiy Storchaka in branch '3.5':
Issue #9770: curses.ascii predicates now work correctly with negative integers.
https://hg.python.org/cpython/rev/cba619a7bf6a

New changeset 84ca252ac346 by Serhiy Storchaka in branch '2.7':
Issue #9770: curses.ascii predicates now work correctly with negative integers.
https://hg.python.org/cpython/rev/84ca252ac346

New changeset eb81f2d2a42b by Serhiy Storchaka in branch '3.6':
Issue #9770: curses.ascii predicates now work correctly with negative integers.
https://hg.python.org/cpython/rev/eb81f2d2a42b

New changeset 1c0b72996e60 by Serhiy Storchaka in branch 'default':
Issue #9770: curses.ascii predicates now work correctly with negative integers.
https://hg.python.org/cpython/rev/1c0b72996e60
History
Date User Action Args
2017-03-31 16:36:08dstufftsetpull_requests: + pull_request840
2016-12-28 08:08:42serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2016-12-28 08:07:50python-devsetnosy: + python-dev
messages: + msg284161
2016-12-17 20:14:05BreamoreBoysetnosy: - BreamoreBoy
2016-12-17 10:51:07serhiy.storchakasetfiles: + curses-ascii-negative.patch
versions: + Python 3.6, Python 3.7, - Python 3.4
nosy: + serhiy.storchaka

messages: + msg283484

stage: patch review
2014-12-31 16:21:42akuchlingsetnosy: - akuchling
2014-07-24 13:20:09akirasetmessages: + msg223833
title: curses.isblank function doesn't match ctype.h -> curses.ascii.isblank() function is broken. It confuses backspace (BS 0x08) with tab (0x09)
2014-06-19 17:35:12akirasetfiles: + curses_ascii_isblank_issue9770.patch

nosy: + akira
messages: + msg221008

keywords: + patch
2014-06-17 18:27:08BreamoreBoysetnosy: + BreamoreBoy

messages: + msg220852
versions: + Python 3.4, Python 3.5, - Python 3.1, Python 3.2
2010-09-04 03:55:44ned.deilysetversions: + Python 3.1, Python 3.2
2010-09-04 03:55:15ned.deilysetnosy: + akuchling
2010-09-04 00:01:48kevinptcreate