classification
Title: SQLITE_MISUSE race condition in sqlite3 is misleadingly raised as a binding error
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Jack Robison
Priority: normal Keywords:

Created on 2019-10-08 15:15 by Jack Robison, last changed 2019-10-08 16:08 by Jack Robison.

Messages (1)
msg354218 - (view) Author: Jack Robison (Jack Robison) Date: 2019-10-08 15:15
There is a race condition in the sqlite3 module that is misleadingly raised as sqlite3.InterfaceError('Error binding parameter 0 - probably unsupported type.')

There are two issues here, one is the incorrectly raise error (https://bugs.python.org/issue16379), the other is the underlying SQLITE_MISUSE caused by the race condition. I believe this is a race between sqlite3 vs python garbage collection due to the releasing of the GIL. If the GIL releasing is removed from sqlite3, I cannot reproduce the race.

Here is a script to reproduce the error, as the number of runners or queries_per_executemany is increased the error happens more frequently. The minimal test case only needs two executemanys where each has two queries to run.





import asyncio
import sqlite3
from concurrent.futures.thread import ThreadPoolExecutor


async def misuse_sqlite(runners=2, queries_per_executemany=2):
    loop = asyncio.get_event_loop()

    expected_error = 'Error binding parameter 0 - probably unsupported type.'
    exceptions = []
    executor = ThreadPoolExecutor(1)
    db = executor.submit(sqlite3.connect, ':memory:', isolation_level=None).result()
    executor.submit(db.executescript, "create table test (id text primary key);")

    def query_in_executor():
        return asyncio.wrap_future(executor.submit(executemany))

    def executemany():
        try:
            return db.executemany("select * from test where id=?", ((str(i),) for i in range(queries_per_executemany)))
        except Exception as err:
            exceptions.append(err)

    for _ in range(runners):
        loop.call_soon(query_in_executor)
    await asyncio.sleep(0.01)

    assert all(str(err) == expected_error and isinstance(err, sqlite3.InterfaceError) for err in exceptions)

    executor.submit(db.close).result()
    executor.shutdown()

    return len(exceptions) > 0


attempts = 0

while True:
    attempts += 1
    if asyncio.run(misuse_sqlite()):
        print('error hit on attempt', attempts)
        break
History
Date User Action Args
2019-10-08 16:08:32Jack Robisonsettype: behavior
2019-10-08 15:15:47Jack Robisoncreate