This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: [sqlite3] context manager leaves db locked if commit fails in __exit__
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, erlendaasland, ghaering, lciti, lukasz.langa, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2016-06-16 16:45 by lciti, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
fix_pysqlite_connection_exit.patch lciti, 2016-06-16 16:45 Patch. review
Pull Requests
URL Status Linked Edit
PR 26202 merged erlendaasland, 2021-05-17 22:38
PR 27942 merged erlendaasland, 2021-08-25 11:36
PR 27943 merged erlendaasland, 2021-08-25 11:43
PR 27944 merged erlendaasland, 2021-08-25 12:10
Messages (10)
msg268678 - (view) Author: Luca Citi (lciti) * Date: 2016-06-16 16:45
I have reported this bug to the pysqlite module for python2 ( https://github.com/ghaering/pysqlite/issues/103 ) but I also report it here because it applies to python3 too.

The pysqlite3 context manager does not perform a rollback when a transaction fails because the database is locked by some other process performing non-DML statements (e.g. during the sqlite3 command line .dump method).

To reproduce the problem, open a terminal and run the following:

```bash
sqlite3 /tmp/test.db 'drop table person; create table person (id integer primary key, firstname varchar)'
echo -e 'begin transaction;\nselect * from person;\n.system sleep 1000\nrollback;' | sqlite3 /tmp/test.db
```

Leave this shell running and run the python3 interpreter from a different shell, then type:

```python
import sqlite3
con = sqlite3.connect('/tmp/test.db')
with con:                                                        
    con.execute("insert into person(firstname) values (?)", ("Jan",))
    pass
```

You should receive the following:

```
      1 with con:
      2         con.execute("insert into person(firstname) values (?)", ("Jan",))
----> 3         pass
      4 

OperationalError: database is locked
```

Without exiting python, switch back to the first shell and kill the `'echo ... | sqlite3'` process. Then run:

```bash
sqlite3 /tmp/test.db .dump
```

you should get:

```
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
/**** ERROR: (5) database is locked *****/
ROLLBACK; -- due to errors
```

This means that the python process never executed a `rollback` and is still holding the lock. To release the lock one can exit python (clearly, this is not the intended behaviour of the context manager).

I believe the reason for this problem is that the exception happened in the implicit `commit` that is run on exiting the context manager, rather than inside it. In fact the exception is in the `pass` line rather than in the `execute` line. This exception did not trigger a `rollback` because the it happened after `pysqlite_connection_exit` checks for exceptions.

The expected behaviour (pysqlite3 rolling back and releasing the lock) is recovered if the initial blocking process is a Data Modification Language (DML) statement, e.g.:

```bash
echo -e 'begin transaction; insert into person(firstname) values ("James");\n.system sleep 1000\nrollback;' | sqlite3 /tmp/test.db
```

because this raises an exception at the `execute` time rather than at `commit` time.

To fix this problem, I think the `pysqlite_connection_exit` function in src/connection.c should handle the case when the commit itself raises an exception, and invoke a rollback. Please see patch attached.
msg268680 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-06-16 17:13
You may also want to check that the method_name is "commit". A test case would nice to have.

Note that the connection context manager will still fail cases like nested context managers. See issue 16958.
msg393839 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-05-17 22:40
> In fact the exception is in the `pass` line rather than in the `execute` line.

I can reproduce this without the `pass` line.

I've taken the liberty to create a PR based on your patch, Luca. Berker's comments have been addressed in the PR.
msg393912 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-05-18 23:15
> I believe the reason for this problem is that the exception happened in the
> implicit `commit` that is run on exiting the context manager, rather than
> inside it. In fact the exception is in the `pass` line rather than in the
> `execute` line. This exception did not trigger a `rollback` because the it
> happened after `pysqlite_connection_exit` checks for exceptions.

FYI, here's the SQLite API interaction from the context manager, chronologically (using the test from the PR). (I only show the relevant arguments passed to the API, for readability.)

sqlite3_prepare_v2("insert into t values('test')", insert_stmt) => SQLITE_OK
sqlite3_get_autocommit()
# Note, the insert statement is now prepared, but not executed yet.

# Transaction control now begins
sqlite3_prepare_v2("BEGIN ", begin_stmt) => SQLITE_OK
sqlite3_step(begin_stmt) => SQLITE_DONE
sqlite3_finalize(begin_stmt)

# Here, the insert statement is executed
sqlite3_bind_blob_parameter_count(insert_stmt)
sqlite3_step(insert_stmt) => SQLITE_DONE
sqlite3_changes()
sqlite3_last_insert_rowid()
sqlite3_reset(insert_stmt) => SQLITE_OK
sqlite3_get_autocommit()

# Enter __exit__: no exception has been raised, so it tries to commit
sqlite3_prepare_v2("commit", commit_stmt) => SQLITE_OK
sqlite3_step(commit_stmt) => SQLITE_BUSY (database is locked)
sqlite3_finalize(commit_stmt)

# After the fix, rollback is now executed
sqlite3_prepare_v2("rollback", rollback_stmt)
sqlite3_step(rollback_stmt) => SQLITE_DONE
sqlite3_finalize(rollback_Stmt)


As you can see, it does not fail (and raise an exception) until commit is issued inside __exit__.
msg400253 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-25 10:59
New changeset 7ecd3425d45a37efbc745788dfe21df0286c785a by Erlend Egeberg Aasland in branch 'main':
bpo-27334: roll back transaction if sqlite3 context manager fails to commit (GH-26202)
https://github.com/python/cpython/commit/7ecd3425d45a37efbc745788dfe21df0286c785a
msg400266 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-25 13:58
New changeset a3c11cebf174e0c822eda8c545f7548269ce7a25 by Erlend Egeberg Aasland in branch 'main':
bpo-27334: Fix reference leak introduced by GH-26202 (GH-27942)
https://github.com/python/cpython/commit/a3c11cebf174e0c822eda8c545f7548269ce7a25
msg400287 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-25 19:02
New changeset 52702e8ba0d777ac92ec36038213976545c4759e by Erlend Egeberg Aasland in branch '3.9':
[3.9] bpo-27334: roll back transaction if sqlite3 context manager fails to commit (GH-26202) (GH-27944)
https://github.com/python/cpython/commit/52702e8ba0d777ac92ec36038213976545c4759e
msg400491 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-28 18:26
New changeset 2a80893e5c023a73ccd32cc319f4f0404f548c00 by Erlend Egeberg Aasland in branch '3.10':
[3.10] bpo-27334: roll back transaction if sqlite3 context manager fails to commit (GH-26202) (GH-27943)
https://github.com/python/cpython/commit/2a80893e5c023a73ccd32cc319f4f0404f548c00
msg400493 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-08-28 18:41
Thanks Luca, for the report, reproducer, and initial patch, Berker for helpful suggestion, and Łukasz, Pablo, & Victor for reviewing and merging.
msg400790 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-31 22:06
Nice!
History
Date User Action Args
2022-04-11 14:58:32adminsetgithub: 71521
2021-08-31 22:06:54vstinnersetnosy: + vstinner
messages: + msg400790
2021-08-28 18:41:36erlendaaslandsetstatus: open -> closed
resolution: fixed
messages: + msg400493

stage: patch review -> resolved
2021-08-28 18:26:08pablogsalsetmessages: + msg400491
2021-08-25 19:02:33pablogsalsetmessages: + msg400287
2021-08-25 13:58:01lukasz.langasetnosy: + lukasz.langa
messages: + msg400266
2021-08-25 12:10:34erlendaaslandsetpull_requests: + pull_request26390
2021-08-25 11:43:59erlendaaslandsetpull_requests: + pull_request26389
2021-08-25 11:36:12erlendaaslandsetpull_requests: + pull_request26388
2021-08-25 10:59:46pablogsalsetnosy: + pablogsal
messages: + msg400253
2021-05-18 23:15:37erlendaaslandsetmessages: + msg393912
2021-05-17 22:40:47erlendaaslandsettitle: pysqlite3 context manager not performing rollback when a database is locked elsewhere for non-DML statements -> [sqlite3] context manager leaves db locked if commit fails in __exit__
messages: + msg393839
versions: + Python 3.9, Python 3.10, Python 3.11, - Python 3.5, Python 3.6
2021-05-17 22:38:40erlendaaslandsetnosy: + erlendaasland
pull_requests: + pull_request24819
2016-06-16 17:13:01berker.peksagsetversions: + Python 3.5
nosy: + ghaering, berker.peksag

messages: + msg268680

type: behavior
stage: patch review
2016-06-16 16:45:50lciticreate