classification
Title: execute("begin immediate") throwing OperationalError
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Big Stone, Cédric Bellegarde, Ma Lin, berker.peksag, callidomus, fschulze, ghaering, jaraco, ned.deily, palaviv, r.david.murray, serhiy.storchaka, socketpair, xiang.zhang
Priority: high Keywords: patch

Created on 2016-10-24 11:05 by fschulze, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
28518.patch palaviv, 2016-11-25 09:45 review
sqlite-explicit-begin.patch serhiy.storchaka, 2016-11-25 19:43 review
sqlite-explicit-begin2.patch serhiy.storchaka, 2016-12-16 12:55 review
sqlite-ddl-dml.patch palaviv, 2016-12-29 14:49 review
sqlite-ddl-dml-2.patch palaviv, 2017-01-20 09:10 review
sqlite-ddl-dml-3.patch palaviv, 2017-01-23 22:26 review
Pull Requests
URL Status Linked Edit
PR 245 merged berker.peksag, 2017-02-22 22:49
PR 318 merged berker.peksag, 2017-02-26 15:30
PR 703 larry, 2017-03-17 21:00
PR 552 closed dstufft, 2017-03-31 16:36
Messages (35)
msg279301 - (view) Author: Florian Schulze (fschulze) Date: 2016-10-24 11:05
Using:

conn = sqlite3.connect(':memory:', isolation_level='IMMEDIATE')
conn.execute('begin immediate')

Throws:

sqlite3.OperationalError: cannot start a transaction within a transaction

This didn't happen in previous versions and the conn.in_transaction attribute is False right before the call to execute, so this situation doesn't seem to be detectable upfront for backward compatibility.
msg279305 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-10-24 13:26
Looks like commit 284676cf2ac8 in #10740 introduces this. Haven't read through the thread yet.
msg279808 - (view) Author: Aviv Palivoda (palaviv) * Date: 2016-10-31 17:07
I looked into this error and I think the problem is the sqlite3_stmt_readonly check in _pysqlite_query_execute (cursor.c line 517):

    if (self->connection->begin_statement && !sqlite3_stmt_readonly(self->statement->st) && !self->statement->is_ddl) {
        if (sqlite3_get_autocommit(self->connection->db)) {
            result = _pysqlite_connection_begin(self->connection);
            if (!result) {
                goto error;
            }
            Py_DECREF(result);
        }
    }

As you can see we check if the current statement is not readonly and if so we start a transaction. The problem is that sqlite3_stmt_readonly return false for the "begin immediate" statement. When this happens we open a transaction with _pysqlite_connection_begin and then executing the "begin immediate".

I think this is a error in sqlite as the documentation says:
"ransaction control statements such as BEGIN, COMMIT, ROLLBACK, SAVEPOINT, and RELEASE cause sqlite3_stmt_readonly() to return true,"
(https://www.sqlite.org/c3ref/stmt_readonly.html)
msg281626 - (view) Author: Florian Schulze (fschulze) Date: 2016-11-24 10:01
How do we proceed from here? The Python 3.6.0rc1 is looming. If this is a bug in sqlite, then we need a workaround. We can't wait for a fix in sqlite, because it's not bundled with Python and without the fix Python 3.6 breaks with older sqlite versions. I'd like to help, but don't know how.
msg281633 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-11-24 14:38
I'm going to mark this as a release blocker because it is a regression.  Ned may or may not agree.  What is needed is for someone to propose a patch.
msg281638 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-24 15:07
Sorry, I'm not experienced with sqlite3, but in what circumstances conn.execute('begin immediate') makes sense? Could you please provide a larger example where it is needed?
msg281639 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-11-24 15:17
Also, if there is indeed a suspected error in the underlying library, has the issue been reported to the SQLite project?
msg281691 - (view) Author: Aviv Palivoda (palaviv) * Date: 2016-11-25 09:45
I searched in the sqlite tickets from something related to this issue and I couldn't find anything. I wanted to check the sqlite-users mailing list to see if anyone had already reported it but there is a problem in gmane archives.

I tried to find a good way to solve this problem but I couldn't. I am adding a patch that should solve this problem but it is for sure not a good way :(.
msg281700 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-11-25 11:20
I haven't had a time to investigate this yet, but I don't think there is an issue with SQLite itself.

Note that pysqlite behaves the same way since version 2.8.0 and I couldn't find a similar report in their issue tracker.

Like Serhiy, I'm also a little puzzled with the reproducer. I think if you want to start an explicit transaction, the preferred way would be to set isolation_level to None. Otherwise, the sqlite3 module will start one implicitly for you (or perhaps I still don't understand PEP 249's transaction model 100% correctly :))

(If we decide that this is not a regression, we should add a test case to cover this use case.)
msg281704 - (view) Author: Florian Schulze (fschulze) Date: 2016-11-25 12:17
Ok, I reread https://docs.python.org/3/library/sqlite3.html#controlling-transactions several times now. I think it's confusing.

I want explicit transaction handling, but ``isolation_level = None`` turns on "autocommit mode". Unfortunately I don't have suggestions on how to improve the documentation.

For us, ``isolation_level = None`` and the explicit ``begin immediate`` seems to be the correct way and it works with all Python versions, including 3.6.0.

So, I would say this is "not a bug" in Python, but my understanding of the sqlite3 module is flawed. The documentation could be better, but I don't know how to make it better.
msg281709 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-11-25 12:39
There is a patch in issue 8145 to improve that part of the documentation. It would be great if you could take a look at it :)
msg281724 - (view) Author: Aviv Palivoda (palaviv) * Date: 2016-11-25 14:35
There are already a few test's that start a transaction explicitly:

In dbapi.SqliteOnConflictTests CheckOnConflictRollbackWithExplicitTransaction, CheckOnConflictAbortRaisesWithExplicitTransactions which set ``isolation_level = None``.

On the other hand in transaction.TransactionalDDL.CheckTransactionalDDL the test do not set the isolation_level to None but start a transaction explicitly. The test pass because sqlite3_stmt_readonly return true for "begin" and "begin deferred".

I think that if we say that in order to start a transaction implicitly the isolation_level needs to be None CheckTransactionalDDL should be changed.
msg281735 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-25 19:43
Here is a patch that allows executing "begin immediate" explicitly with any isolation_level. I don't know what behavior is more preferable.
msg281738 - (view) Author: Big Stone (Big Stone) Date: 2016-11-25 20:19
Comment kindly provided by D. Richard Hipp himself:
"""""
I don't have a login for the python bug tracker so I cannot comment
there.  But I think I see the problem.  This is as Aviv Polivoda
remarks at https://bugs.python.org/issue28518#msg279808

I think this is a error in sqlite as the documentation says:
"ransaction control statements such as BEGIN, COMMIT, ROLLBACK,
SAVEPOINT, and RELEASE cause sqlite3_stmt_readonly() to return true,"

Except it is not a bug in SQLite, but rather an ambiguity in the
documentation.  In his quote, Aviv omitted the second clause from the
documentation:  "since the statements themselves do not actually
modify the database but rather they control the timing of when other
statements modify the database."  (Full text here:
https://www.sqlite.org/c3ref/stmt_readonly.html)

For a plain BEGIN statement, there are no changes to the database
file, so sqlite3_stmt_readonly() does indeed return TRUE.  But for a
BEGIN IMMEDIATE statement, there are database changes, because the
extra IMMEDIATE keyword causes the statement to go ahead and start
transaction "immediately".

So technically, the documentation is correct, though I can certainly
understand that it is misleading and ambiguous as currently worded.
Clarification will be checked in shortly and will appear in the 3.16.0
release.

Note also that behavior of sqlite3_stmt_readonly() has not changed
since that interface was first added for the 3.7.5 release on
2011-02-01.  So this is not an SQLite regression.  Rather, I presume
that Python has recently started using the sqlite3_stmt_readonly()
interface in a new way.



--
D. Richard Hipp
drh@sqlite.org
"""""
msg281739 - (view) Author: Big Stone (Big Stone) Date: 2016-11-25 20:28
rewording of sqlite documentation:
http://www.sqlite.org/src/info/a4205a83e4ed977a
msg282463 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-05 20:34
Thanks everyone for investigating this, in particular Big Stone for the reference to the SQLite project and especially Richard Hipp for once again responding quickly to issues we've brought up.  While it would be good to get this resolved soon one way or another (doc and/or code changes), it doesn't sound like it needs to be a release blocker for 3.6.0 so I'm lowering its priority.
msg283345 - (view) Author: Aviv Palivoda (palaviv) * Date: 2016-12-15 21:31
I think that adding Serhiy patch in addition to the documentation improvement in issue #8145 would be the best way to proceed. This will insure no regression problems.
We can add a warning when someone try to use BEGIN, ROLLBACK, SAVEPOINT, and RELEASE without changing the isolation_level to None.
msg283398 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-16 12:55
Here is a little simplified version. No need to handle a single "begin".
msg284268 - (view) Author: Aviv Palivoda (palaviv) * Date: 2016-12-29 14:49
Issue #29003 seems to be related to this one. I think that they can be solved the same way as done in Serhiy patch but I would like to suggest a different approach.
I suggest changing the check for DDL statement with a check for DML statement. We actually should start a transaction only for DML statements (https://docs.python.org/3/library/sqlite3.html#controlling-transactions) so all other statements (DDL, DCL and TCL) should not start a transaction.
The attached patch solve both this issue and issue #29003.
msg284373 - (view) Author: Ma Lin (Ma Lin) * Date: 2016-12-31 07:18
I have no idea about how to fix it, but I would like to suggest that add back this test which was removed in commit 284676cf2ac8 :

file: /Lib/sqlite3/test/transactions.py
def CheckVacuum(self):
    self.cur.execute("create table test(i)")
    self.cur.execute("insert into test(i) values (5)")
    self.cur.execute("vacuum")

Thank you for your efforts! regards.
msg284392 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-12-31 14:29
I only did a quick review, but Aviv's latest patch looks reasonable to me. Thanks!

And yes, we can add CheckVacuum back if it's not already tested.
msg284397 - (view) Author: Aviv Palivoda (palaviv) * Date: 2016-12-31 16:08
Adding the vacuum test can't actually happen because the change in commit 284676cf2ac8 changed the API and we don't commit before non DML statements. I opened a issue that will clarify the docs (#29121).
msg284538 - (view) Author: Ma Lin (Ma Lin) * Date: 2017-01-03 04:42
I'm trying to catch up with you.
Does it mean, if I want execute VACUUM, I have to commit manually? Like this:

if conn.in_transaction:
    conn.commit()
conn.execute("vacuum")
msg284585 - (view) Author: Aviv Palivoda (palaviv) * Date: 2017-01-03 18:02
Yes. If a transaction is open you will need to explicitly commit before doing vacuum.
I am not sure if that was intentional or not. From quick look in the sqlite source code there are few things that cannot happen from a transaction:
1. VACUUM
2. ATTACH
3. DETACH
4. BEGIN
5. Few PRAGMAs (temp_store, synchronous)

All of the above worked with implicit commit before commit 284676cf2ac8 but now has to call commit explicitly.
msg285618 - (view) Author: Ma Lin (Ma Lin) * Date: 2017-01-17 08:02
some comments on Aviv Palivoda's patch.
msg285893 - (view) Author: Aviv Palivoda (palaviv) * Date: 2017-01-20 09:10
Uploading a new patch with fixes from Ma Lin comments.
Two points:
1. Should we add the VACUUM with a explicit commit? Maybe there should be an implicit commit before VACUUM?
2. Should a SELECT start a transaction? I think it should according to PEP 249. There is a open issue on the case (#9924). Should we just change this on this patch?
msg286084 - (view) Author: Ma Lin (Ma Lin) * Date: 2017-01-23 13:28
If the answer is (no, no) , the behavior strictly follows the doc changes in commit 284676cf2ac8.
Anyway, I'm not a deep user of SQLite, I can't give further advices. :(
msg286087 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-01-23 14:10
> 1. Should we add the VACUUM with a explicit commit? Maybe there should
> be an implicit commit before VACUUM?

VACUUM is often an expensive operation so I think people should need to explicitly handle it anyway.

> 2. Should a SELECT start a transaction? I think it should according to
> PEP 249. There is a open issue on the case (#9924). Should we just
> change this on this patch?

Let's discuss that in issue 9924 :) Last time I look at it there was some backward compatibility concern so we might able to fix it only in 3.7.
msg286122 - (view) Author: Aviv Palivoda (palaviv) * Date: 2017-01-23 22:26
Removed opening a transaction on select. I will argue for that in issue 9924 after this is resolved.
msg286300 - (view) Author: Cédric Bellegarde (Cédric Bellegarde) Date: 2017-01-26 09:32
Adding a commit doesn't fix the issue here:
sql.commit()
sql.execute('VACUUM')

And on run:
cannot VACUUM from within a transaction
msg286308 - (view) Author: Ma Lin (Ma Lin) * Date: 2017-01-26 12:24
You can set isolation_level = None in sqlite3.connect() parameters, then sqlite3 module will not begin a transaction implicitly.

Another way is disabling auto-begin-transaction temporarily:
sql.isolation_level = None
sql.execute('VACUUM')
sql.isolation_level = ''  # <- note that this is the default value of isolation_level
msg286525 - (view) Author: Ma Lin (Ma Lin) * Date: 2017-01-31 12:38
Let me give a summary.

Before 3.6.0 (before 284676cf2ac8):
implicit begin: st in (INSERT, UPDATE, DELETE, REPLACE)
implicit commit: st not in (SELECT, INSERT, UPDATE, DELETE, REPLACE)

In 3.6.0 (after 284676cf2ac8):
implicit begin: (not sqlite3_stmt_readonly(st))  and  (not in (CREATE, DROP, REINDEX))
implicit commit: no

With Palivoda's sqlite-ddl-dml-3.patch:
implicit begin: st in (INSERT, UPDATE, DELETE, REPLACE)
implicit commit: no
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Some possible subsequent steps base on this issue:

1, remove sqlite3_stmt_readonly(), see issue29355.
Base on sqlite-ddl-dml-3.patch, only change a single line of code:
+ if (self->is_dml) {
- if (!sqlite3_stmt_readonly(self->statement->st)) {
Then restore the behavior before 3.6.0, this is even better than using sqlite3_stmt_readonly().

2, enhance backward compatibility.
Please read msg284585, msg286217, msg286293 in a row. 
If we implicitly commit before executing some statements (and print a warning), we can improve compatibility largely, the cost is almost free.

3, docments enhancement.
see issue8145, issue29121.
msg288599 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-02-26 16:09
Thank you everyone! This should be fixed now.
msg290380 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-03-24 23:40
New changeset 76995cab69d5ef83d31d8a5754cbad11be4038cb by Berker Peksag in branch '3.6':
bpo-28518: Start a transaction implicitly before a DML statement (#245) (#318)
https://github.com/python/cpython/commit/76995cab69d5ef83d31d8a5754cbad11be4038cb
msg290391 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-03-24 23:42
New changeset 4a926caf8e5fd8af771b2c34bfb6e91c732331fe by Berker Peksag in branch 'master':
bpo-28518: Start a transaction implicitly before a DML statement (#245)
https://github.com/python/cpython/commit/4a926caf8e5fd8af771b2c34bfb6e91c732331fe
History
Date User Action Args
2017-03-31 16:36:23dstufftsetpull_requests: + pull_request976
2017-03-24 23:42:13berker.peksagsetmessages: + msg290391
2017-03-24 23:40:36berker.peksagsetmessages: + msg290380
2017-03-17 21:00:34larrysetpull_requests: + pull_request608
2017-02-26 16:09:49berker.peksagsetstatus: open -> closed
resolution: fixed
messages: + msg288599

stage: patch review -> resolved
2017-02-26 15:30:12berker.peksagsetpull_requests: + pull_request279
2017-02-22 22:53:32berker.peksaglinkissue29355 dependencies
2017-02-22 22:49:47berker.peksagsetpull_requests: + pull_request210
2017-01-31 12:38:15Ma Linsetmessages: + msg286525
2017-01-26 12:24:41Ma Linsetmessages: + msg286308
2017-01-26 09:32:34Cédric Bellegardesetnosy: + Cédric Bellegarde
messages: + msg286300
2017-01-23 22:26:51palavivsetfiles: + sqlite-ddl-dml-3.patch

messages: + msg286122
2017-01-23 14:10:53berker.peksagsetmessages: + msg286087
2017-01-23 13:28:18Ma Linsetmessages: + msg286084
2017-01-20 09:10:53palavivsetfiles: + sqlite-ddl-dml-2.patch

messages: + msg285893
2017-01-17 08:02:46Ma Linsetmessages: + msg285618
2017-01-10 18:33:37berker.peksaglinkissue29228 superseder
2017-01-03 18:02:02palavivsetmessages: + msg284585
2017-01-03 04:42:41Ma Linsetmessages: + msg284538
2016-12-31 16:08:18palavivsetmessages: + msg284397
2016-12-31 14:29:36berker.peksagsetmessages: + msg284392
2016-12-31 13:17:52callidomussetnosy: + callidomus
2016-12-31 07:18:48Ma Linsetnosy: + Ma Lin
messages: + msg284373
2016-12-29 15:07:17berker.peksaglinkissue29003 superseder
2016-12-29 14:49:22palavivsetfiles: + sqlite-ddl-dml.patch

messages: + msg284268
2016-12-16 12:55:10serhiy.storchakasetfiles: + sqlite-explicit-begin2.patch

messages: + msg283398
2016-12-15 21:31:17palavivsetmessages: + msg283345
2016-12-05 20:34:36ned.deilysetpriority: release blocker -> high

messages: + msg282463
stage: patch review
2016-11-25 20:28:01Big Stonesetmessages: + msg281739
2016-11-25 20:19:57Big Stonesetnosy: + Big Stone
messages: + msg281738
2016-11-25 19:43:54serhiy.storchakasetfiles: + sqlite-explicit-begin.patch

messages: + msg281735
2016-11-25 14:35:54palavivsetmessages: + msg281724
2016-11-25 12:39:33berker.peksagsetmessages: + msg281709
2016-11-25 12:17:54fschulzesetmessages: + msg281704
2016-11-25 11:20:29berker.peksagsetmessages: + msg281700
2016-11-25 09:45:29palavivsetfiles: + 28518.patch
keywords: + patch
messages: + msg281691
2016-11-24 15:17:48ned.deilysetmessages: + msg281639
2016-11-24 15:07:02serhiy.storchakasetmessages: + msg281638
2016-11-24 14:38:48r.david.murraysetpriority: normal -> release blocker
versions: + Python 3.7
nosy: + ned.deily, r.david.murray

messages: + msg281633
2016-11-24 10:01:36fschulzesetmessages: + msg281626
2016-10-31 17:07:36palavivsetnosy: + palaviv
messages: + msg279808
2016-10-24 13:26:59xiang.zhangsetmessages: + msg279305
2016-10-24 12:52:44jaracosetnosy: + jaraco
2016-10-24 11:44:51socketpairsetnosy: + socketpair
2016-10-24 11:25:44serhiy.storchakasetnosy: + ghaering, berker.peksag, serhiy.storchaka, xiang.zhang
2016-10-24 11:05:18fschulzecreate