Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve some sqlite3 errors #89022

Closed
serhiy-storchaka opened this issue Aug 7, 2021 · 11 comments · Fixed by #27695 or #91572
Closed

Improve some sqlite3 errors #89022

serhiy-storchaka opened this issue Aug 7, 2021 · 11 comments · Fixed by #27695 or #91572
Labels
3.11 only security fixes extension-modules C modules in the Modules dir topic-sqlite3

Comments

@serhiy-storchaka
Copy link
Member

BPO 44859
Nosy @berkerpeksag, @serhiy-storchaka, @JelleZijlstra, @erlend-aasland
PRs
  • bpo-44859: Improve error handling in sqlite3 and change some errors #27654
  • bpo-44859: Raise more accurate exceptions in sqlite3 #27695
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2021-08-07.14:14:16.688>
    labels = ['extension-modules', '3.11']
    title = 'Improve some sqlite3 errors'
    updated_at = <Date 2022-03-17.05:58:28.684>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2022-03-17.05:58:28.684>
    actor = 'JelleZijlstra'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Extension Modules']
    creation = <Date 2021-08-07.14:14:16.688>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 44859
    keywords = ['patch']
    message_count = 8.0
    messages = ['399183', '399185', '399187', '399211', '399241', '399289', '399441', '415388']
    nosy_count = 4.0
    nosy_names = ['berker.peksag', 'serhiy.storchaka', 'JelleZijlstra', 'erlendaasland']
    pr_nums = ['27654', '27695']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue44859'
    versions = ['Python 3.11']

    @serhiy-storchaka
    Copy link
    Member Author

    • MemoryError is now raised instead of sqlite3.Warning when memory is not enough for encoding statement to UTF-8 in Connection.__call__() and Cursor.execute().
    • UnicodEncodeError is now raised instead of sqlite3.Warning when statement contains surrogate characters in Connection.__call__() and Cursor.execute().
    • TypeError is now raised instead of ValueError for non-string script in Cursor.execute().
    • ValueError is now raised for script containing NUL instead of truncating it in Cursor.execute().
    • Correctly handled exceptions raised when getting boolean value of the result of the progress handler to bool.

    Also added many tests which cover different exceptional cases.

    @serhiy-storchaka serhiy-storchaka added 3.11 only security fixes extension-modules C modules in the Modules dir labels Aug 7, 2021
    @serhiy-storchaka
    Copy link
    Member Author

    There is also a problem with the sqlite3.InterfaceError raised when bind parameters with message "Error binding parameter XXX - probably unsupported type." It is raised not only for unsupported types, but:

    • For too large integers (outside of the range of signed 64-bit integers).
    • For too large strings and blobs with size <= INT_MAX but larger than the SQLite limit (1_000_000_000 by default). But OverflowError is raised for size >= INT_MAX.
    • When memory error or any other error occurred inside SQLite.

    @serhiy-storchaka
    Copy link
    Member Author

    Entries 3 and 4 are about Cursor.executescript(), not Cursor.execute().

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset 0eec627 by Serhiy Storchaka in branch 'main':
    bpo-44859: Improve error handling in sqlite3 and and raise more accurate exceptions. (GH-27654)
    0eec627

    @erlend-aasland
    Copy link
    Contributor

    • sqlite3.ProgrammingError is raised for SQLITE_MISUSE. IMO, sqlite3.InterfaceError, is more appropriate.
    • In case of too large integer values, OverflowError is raised on bind, but sqlite3.DataError is raised on UDF return. Using sqlite3.DataError in both cases would be more aligned with PEP-249.
    • Too long strings/blobs raise OverflowError. Consider raising sqlite3.DataError instead.
    • For non-contiguous (blob) buffers, we overwrite the BufferError with ValueError. Instead of overwriting BufferError, we should consider chaining with sqlite3.DataError.
    • Consider raising sqlite3.Warning when nan is implicitly converted to NULL by SQLite
    • In pysqlite_statement_create(), consider raising sqlite3.ProgrammingError iso. sqlite3.Warning when more than one statement is executed
    • In pysqlite_statement_create(), consider raising sqlite3.DataError iso. ValueError for queries with null characters

    @erlend-aasland
    Copy link
    Contributor

    Also, for SQLite 3.7.17 and above, we should raise sqlite3.Warning for error codes SQLITE_NOTICE and SQLITE_WARNING.

    @erlend-aasland
    Copy link
    Contributor

    Also, for SQLite 3.7.17 and above, we should raise sqlite3.Warning for error codes SQLITE_NOTICE and SQLITE_WARNING.

    Er, forget that. Neither are returned by any SQLite C interface, at the moment. They can (currently) only be used as the first argument to sqlite3_log().

    @JelleZijlstra
    Copy link
    Member

    New changeset 4674fd4 by Erlend Egeberg Aasland in branch 'main':
    bpo-44859: Raise more accurate exceptions in sqlite3 (GH-27695)
    4674fd4

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    erlend-aasland pushed a commit to erlend-aasland/cpython that referenced this issue Apr 15, 2022
    SQLITE_MISUSE implies misuse of the SQLite C API, which, if it happens,
    is _not_ a user error; it is an sqlite3 extension module error.
    JelleZijlstra pushed a commit that referenced this issue May 4, 2022
    …I misuse (#91572)
    
    * Map SQLITE_MISUSE to sqlite3.InterfaceError
    
    SQLITE_MISUSE implies misuse of the SQLite C API, which, if it happens,
    is _not_ a user error; it is an sqlite3 extension module error.
    
    * Raise better errors when binding parameters fail.
    
    Instead of always raising InterfaceError, guessing what went wrong,
    raise accurate exceptions with more accurate error messages.
    @erlend-aasland
    Copy link
    Contributor

    erlend-aasland commented May 18, 2022

    The following PRs have improved this issue:

    Suggesting to close this. If other inconsistencies are discovered, we can open a new issue.

    @erlend-aasland erlend-aasland added the pending The issue will be closed if no feedback is provided label May 18, 2022
    @erlend-aasland
    Copy link
    Contributor

    Thanks for bringing this up and pushing it, Serhiy.

    @erlend-aasland erlend-aasland removed the pending The issue will be closed if no feedback is provided label May 19, 2022
    @erlend-aasland
    Copy link
    Contributor

    erlend-aasland commented Jun 1, 2022

    There is one inconsistency remaining:

    sqlite3.Cursor.execute raise sqlite3.Warning instead of sqlite3.ProgrammingError if multiple SQL statements are supplied. IMO, we should fix this.

    Actually, it is not raised in the execute method implementation; it is raised during statement creation in statement.c.

    Forget it; I already fixed this ☕ ☕ ☕

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 only security fixes extension-modules C modules in the Modules dir topic-sqlite3
    Projects
    None yet
    3 participants