Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[sqlite3] lazy creation of result rows #89139

Closed
erlend-aasland opened this issue Aug 22, 2021 · 3 comments
Closed

[sqlite3] lazy creation of result rows #89139

erlend-aasland opened this issue Aug 22, 2021 · 3 comments
Labels
extension-modules C modules in the Modules dir topic-sqlite3 type-feature A feature request or enhancement

Comments

@erlend-aasland
Copy link
Contributor

BPO 44976
Nosy @berkerpeksag, @serhiy-storchaka, @pablogsal, @erlend-aasland
PRs
  • bpo-44976: Lazy creation of sqlite3 result rows #27884
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-08-25.10:29:18.621>
created_at = <Date 2021-08-22.09:06:30.968>
labels = ['extension-modules', 'type-feature']
title = '[sqlite3] lazy creation of result rows'
updated_at = <Date 2021-08-25.10:29:18.620>
user = 'https://github.com/erlend-aasland'

    bugs.python.org fields:

    activity = <Date 2021-08-25.10:29:18.620>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-25.10:29:18.621>
closer = 'pablogsal'
components = ['Extension Modules']
creation = <Date 2021-08-22.09:06:30.968>
creator = 'erlendaasland'
dependencies = []
files = []
hgrepos = []
issue_num = 44976
keywords = ['patch']
message_count = 3.0
messages = ['400062', '400065', '400252']
nosy_count = 4.0
nosy_names = ['berker.peksag', 'serhiy.storchaka', 'pablogsal', 'erlendaasland']
pr_nums = ['27884']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44976'
versions = []
    @erlend-aasland
    Copy link
    Contributor Author

    Currently, we build the first result row in the _pysqlite_query_execute() loop if sqlite3_step() returned SQLITE_ROW. When the user asks for a row (for example, using sqlite3.Cursor.fetchone()), this pre-built row is returned, and the next row is prepared.

    Suggesting to lazily build result rows instead.

    Cons:

    • no result tuples are built unless sqlite3.Cursor.fetch*() is called
    • no need to keep the next result row (tuple) in pysqlite_Cursor; rows are built on demand
    • pysqlite_cursor_iternext() is vastly simplified (50% less lines of code)
    • the main loop in _pysqlite_query_execute() is further simplified

    Cons:

    • code churn

    git diff main --shortstat: 2 files changed, 29 insertions(+), 58 deletions(-)

    @erlend-aasland erlend-aasland added extension-modules C modules in the Modules dir type-feature A feature request or enhancement labels Aug 22, 2021
    @erlend-aasland
    Copy link
    Contributor Author

    Historical note:
    The current behaviour was introduced in pysqlite v2.0.2 (May 21 2005), in order to be able to finalise statements and thus close cursors as soon as possible. However, that effect was cancelled just after a couple of months, by the addition of the LRU cache and the ability to reuse prepared statements.

    @pablogsal
    Copy link
    Member

    New changeset 3df0fc8 by Erlend Egeberg Aasland in branch 'main':
    bpo-44976: Lazy creation of sqlite3 result rows (GH-27884)
    3df0fc8

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    extension-modules C modules in the Modules dir topic-sqlite3 type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @pablogsal @erlend-aasland