New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[sqlite3] Remove special rollback handling #88258
Comments
Quoting from the SQLite 3.7.11 changelog[1]: Quoting from the SQLite 3.8.7.2 changelog[2]: Quoting from the SQLite docs[3]: I've done some tests with SQLite versions 3.35.4 and 3.7.15 where I've removed the call to pysqlite_do_all_statements() (introduced by [4]) in pysqlite_connection_rollback_impl(), and I've also removed the pysqlite_Cursor.reset member and all of the related code. The test suite passes fine (except for, as expected, the two tests that check for InterfaceError in case of fetch across rollback). Do we really need to special case rollbacks anymore? I've tried to come up with tests that prove this approach wrong, but I haven't found any yet. [1] https://sqlite.org/changes.html#version_3_7_11 |
Quoting pysqlite commit 5a009ed message (ghaering/pysqlite@5a009ed): The commit is from 2005-12-09. SQLite 3.7.11 wasn't released until 2012, so in 2005 any pending statements would block a rollback. I'm guessing commit 5a009ed addressed that issue. |
The effect of PR 26026 is that InterfaceError is no longer raised for fetch across rollback; instead it is up to SQLite how to handle this:
A NEWS entry should mention the change in behaviour, but I can't see how it would break existing projects; the current code disallows fetch across rollback (InterfaceError), so any problematic code would have been found, handled and fixed during debugging. |
I've crafted a number of rollback tests, but it occurred to me that they are simply just testing SQLite behaviour; not sqlite3 behaviour. I had to adjust the tests according to which version of SQLite was used (for example 3.8.7.2 introduced new behaviour). Such tests are bound to break as SQLite evolves. I'm not sure we want such tests in our test suite; it can make the CI fail for completely unrelated PRs. Suggesting to leave detailed rollback testing to SQLite and just keep a couple of basic tests in our suite. |
Berker, does this look ok to you? |
I think this change is no problem. There is only one situation that a problem may occur. Write code with SQLite 3.8.7.2+ (2014-11-18), and run it on 3.7.15 (2012-12-12) ~ 3.8.7.1-, but this situation may be difficult to happen, we can note this situation in doc. More securely, if run on SQLite 3.8.7.1-, and encounter SQLITE_ABORT_ROLLBACK error code, a prompt can be given to explain the reason. Also note that the current main branch is buggy. If don't adopt this change or revert this change later, don't forget to fix the bug of msg407185 ( |
Thanks, and thank you for looking reviewing this change.
How realistic is this scenario? If you compile with, for example 3.14.0 or newer, you'd link with sqlite3_trace_v2, not sqlite3_trace, so the loader would prevent you from running with anything pre 3.14. AFAIK, we've never had such problems.
You already get both an error message, an (extended) error code. That should be sufficient.
It is a change of behaviour of the internal machinery. Does the change lead to wrong results (duplicate rows, wrong rows returned, no rows returned)? Corrupted/garbage data? Non-deterministic behaviour? Does any of the API's provided by sqlite3 not behave according to the documentation anymore? NB: I plan to get rid of the in_use flag long before the beta sets in. |
I mean, after this change, different versions of SQLite will behave differently. And give a message for SQLITE_ABORT_ROLLBACK to explain this problem.
It just leaks resource, apart from this, there seems to be no problem. |
Yes, this is explained in the news item:
I'm not sure this is a good idea; I believe letting the underlying SQLite library explain the error is better than trying to deduce stuff based on what was known at compile time and what we know at runtime.
Can you provide a reproducer? We've run this change through the ref. leak bots, and they are all green, so if there's a ref. leak, the test suite needs improvements. |
Imagine a person write a code with Python 3.11 and SQLite 3.8.7.2+, and then deploying it to Python 3.11 and SQLite 3.8.7.1-, error may occur. However, this situation is difficult to happen.
The statement in cache will be never reused. If you don't mind, it's not a big problem. |
Yes, I also think this is a far fetched case. I'll see if there's an easy way to handle this. If there isn't, I'm not sure the added complexity will be worth it.
That is not necessarily correct; if the "problematic cursor" executes a new statement, the statement in the cache will be reset, thus it can be reused. Also, if the "problematic cursor" is closed (or deleted), the statement can also be reused. But, if the "problematic cursor" just gets stale and is _never_ used again, it may be a performance issue if it is a hot statement. But, as soon as I can land my changes related to the in_use flag, this will not be an issue anyway. |
If the special rollback handling is removed, the behavior of Connection.rollback() and 'ON CONFLICT ROLLBACK' clause will be consistent. |
Reopening; there's some clean-up left. After #70214, the |
reset
fromsqlite3.Cursor
#30377Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: