diff -r 3151f6f9df85 Doc/library/sqlite3.rst --- a/Doc/library/sqlite3.rst Thu Jun 26 01:41:06 2014 -0400 +++ b/Doc/library/sqlite3.rst Tue Aug 25 09:04:09 2015 +0200 @@ -166,7 +166,7 @@ first blank for the column name: the column name would simply be "x". -.. function:: connect(database[, timeout, detect_types, isolation_level, check_same_thread, factory, cached_statements, uri]) +.. function:: connect(database[, timeout, detect_types, isolation_level, check_same_thread, factory, cached_statements, uri, transaction_mode]) Opens a connection to the SQLite database file *database*. You can use ``":memory:"`` to open a database connection to a database that resides in RAM @@ -180,6 +180,15 @@ For the *isolation_level* parameter, please see the :attr:`Connection.isolation_level` property of :class:`Connection` objects. + This parameter is deprecated. Use *transaction_mode* instead. + + By default, the :mod:`sqlite3` module automatically starts transactions as + prescribed by :pep:`249`. The *transaction_mode* parameter provides + control of this behavior. It may be set to ``'traditional'`` (default), + ``'traditional deferred'``", ``'traditional immediate'``", ``'traditional + exclusive'``", ``'standard'``, ``'standard deferred'``", ``'standard + immediate'``", ``'standard exclusive'``" or ``'autocommit'``. See section + :ref:`sqlite3-controlling-transactions` for a more detailed explanation. SQLite natively supports only the types TEXT, INTEGER, REAL, BLOB and NULL. If you want to use other types you must add support for them yourself. The @@ -214,6 +223,8 @@ .. versionchanged:: 3.4 Added the *uri* parameter. + .. versionchanged:: 3.5 + Added the *transaction_mode* parameter. .. function:: register_converter(typename, callable) @@ -263,11 +274,44 @@ A SQLite database connection has the following attributes and methods: + .. attribute:: transaction_mode + + Get or set the transaction mode, one of ``'traditional'`` (default), + ``'traditional deferred'``", ``'traditional immediate'``", + ``'traditional exclusive'``", ``'standard'``, ``'standard deferred'``", + ``'standard immediate'``", ``'standard exclusive'``" or + ``'autocommit'``. This attribute cannot be changed while a transaction + is active. You must :meth:`commit()` or :meth:`rollback()` first. See + section :ref:`sqlite3-controlling-transactions` for a more detailed + explanation. + + .. versionadded:: 3.5 + + .. attribute:: autocommit + + Short-hand property to get or set :attr:`transaction_mode` to common + values. :const:`True` if :attr:`transaction_mode` is ``'autocommit'``, + :const:`False` otherwise. Setting it to :const:`True` sets + :attr:`transaction_mode` to ``'autocommit'``, setting it to + :const:`False` sets :attr:`transaction_mode` to ``'standard'``. (This + differs from the default value for :attr:`transaction_mode` which is + ``'traditional'`` for backwards-compatibility reasons.) + + .. versionadded:: 3.5 + .. attribute:: isolation_level - Get or set the current isolation level. :const:`None` for autocommit mode or - one of "DEFERRED", "IMMEDIATE" or "EXCLUSIVE". See section - :ref:`sqlite3-controlling-transactions` for a more detailed explanation. + Get or set the current default isolation level. :const:`None` for + autocommit mode or one of "DEFERRED", "IMMEDIATE" or "EXCLUSIVE". See + section :ref:`sqlite3-controlling-transactions` for a more detailed + explanation. + + .. warning:: + This parameter is unrelated with the concept of isolation levels. + SQLite runs at the SERIALIZABLE isolation level (unless you've + enabled cache sharing and ``PRAGMA read_uncommitted;``). + + This attribute is deprecated. Use *transaction_mode* instead. .. attribute:: in_transaction @@ -857,11 +901,28 @@ Controlling Transactions ------------------------ -By default, the :mod:`sqlite3` module opens transactions implicitly before a -Data Modification Language (DML) statement (i.e. +By default, the underlying SQLite 3 library autocommits, but the Python +:mod:`sqlite3`` module does not, in order to comply with :pep:`249`. + +The :mod:`sqlite3`` module offers three main transaction management modes, +controlled by *transaction_mode*: ``'traditional'`` (default), ``'standard'`` +and ``'autocommit'``. + +In addition, the traditional and standard mode can specify a particular +transaction behavior: none (default), ``'deferred'``, ``'immediate'`` or +``'exclusive'``. + +.. versionchanged:: 3.5 + Added *transaction_mode* and introduced the standard mode. + +Traditional mode +^^^^^^^^^^^^^^^^ + +In this mode, the :mod:`sqlite3` module opens transactions implicitly before a +Data Modification Language (DML) statement (i.e. ``INSERT``/``UPDATE``/``DELETE``/``REPLACE``), and commits transactions -implicitly before a non-DML, non-query statement (i. e. -anything other than ``SELECT`` or the aforementioned). +implicitly before a non-DML, non-query statement (i. e. anything other than +``SELECT`` or the aforementioned). So if you are within a transaction and issue a command like ``CREATE TABLE ...``, ``VACUUM``, ``PRAGMA``, the :mod:`sqlite3` module will commit implicitly @@ -871,16 +932,62 @@ is active or not). The current transaction state is exposed through the :attr:`Connection.in_transaction` attribute of the connection object. -You can control which kind of ``BEGIN`` statements sqlite3 implicitly executes -(or none at all) via the *isolation_level* parameter to the :func:`connect` -call, or via the :attr:`isolation_level` property of connections. +The :mod:`sqlite3` module also commits implicitly before ``SAVEPOINT`` +statements, making it impossible to use savepoints in traditional mode. -If you want **autocommit mode**, then set :attr:`isolation_level` to None. +You can control which kind of ``BEGIN`` statements :mod:`sqlite3` implicitly +executes by specifying a transaction behavior (see below). -Otherwise leave it at its default, which will result in a plain "BEGIN" -statement, or set it to one of SQLite's supported isolation levels: "DEFERRED", -"IMMEDIATE" or "EXCLUSIVE". +Since it doesn't start transactions for ``SELECT`` queries, this mode provides +a good compromise between concurrency and transactional guarantees. However, +doing so breaks the serializability guarantees of SQLite. If you care about +serializable transaction isolation, you should use the standard mode instead. +Standard mode +^^^^^^^^^^^^^ + +In this mode, the :mod:`sqlite3` module opens a transaction implicitly before +any statement, unless a transaction is already active. It never commits +transactions implicitly. There are also two reasons for doing that. The first +is that many non-DML, non-query statements such as ``SAVEPOINT`` make perfect +sense inside a transaction. The second is that implicit commits can introduce +subtle and hard to detect transactional integrity bugs. + +This mode implements the transaction semantics described in :pep:`249`. It's +less suitable to concurrent workloads than the traditional mode. If you hit +locking issues when many resources + + +You can control which kind of ``BEGIN`` statements :mod:`sqlite3` implicitly +executes by specifying a transaction behavior (see below). + + +Autocommit mode +^^^^^^^^^^^^^^^ + +``'autocommit'`` mode means that statements that modify the database take +effect immediately, without needing to call ``commit()``. + +This mode enables SQLite 3 library, without any interference of the :mod:`sqlite3` +module. You can implement your own transaction managament features as needed. + +The proper way to start a transaction is to switch to the ``'standard'`` or +``'traditional'`` mode. Then, before the next statement runs, a ``BEGIN`` +statement will be issued automatically, starting a transaction. This also +happens after each ``commit()`` or ``rollback()``. + +To return to autocommit, first run ``commit()`` or a ``rollback()`` to end the +current transaction, then switch back to the ``'autocommit'`` mode. + +Transaction behaviors +^^^^^^^^^^^^^^^^^^^^^ + +Transaction behavior controls whether :mod:`sqlite3` emits a ``BEGIN``, +``BEGIN DEFERRED``, ``BEGIN IMMEDIATE`` or ``BEGIN EXCLUSIVE`` when it opens a +transaction implicitly in traditional or standard mode. See SQLite's +documentation of `BEGIN TRANSACTION`_ for details. + +.. _BEGIN TRANSACTION: http://www.sqlite.org/lang_transaction.html Using :mod:`sqlite3` efficiently @@ -916,13 +1023,23 @@ Using the connection as a context manager ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Connection objects can be used as context managers -that automatically commit or rollback transactions. In the event of an -exception, the transaction is rolled back; otherwise, the transaction is -committed: +Connection objects can be used as context managers that automatically commit +or rollback transactions. In the event of an exception, the transaction is +rolled back; otherwise, the transaction is committed: .. literalinclude:: ../includes/sqlite3/ctx_manager.py +Such context managers don't have any effect in autocommit mode. They're +designed for the traditional and standard modes. + +.. warning:: + + Don't nest these context managers! + + If you do, when you exit successfully the inner context, all changes up to + this point are committed to the database. If an exception is raised later + in the outer context, the rollback will only go back to the end of the + inner block, not to the beginning of the outer block like you would expect. Common issues ------------- diff -r 3151f6f9df85 Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c Thu Jun 26 01:41:06 2014 -0400 +++ b/Modules/_sqlite/connection.c Tue Aug 25 09:04:09 2015 +0200 @@ -44,6 +44,7 @@ _Py_IDENTIFIER(cursor); static int pysqlite_connection_set_isolation_level(pysqlite_Connection* self, PyObject* isolation_level); +static int pysqlite_connection_set_transaction_mode(pysqlite_Connection* self, PyObject* transaction_mode); static void _pysqlite_drop_unused_cursor_references(pysqlite_Connection* self); @@ -64,7 +65,7 @@ static char *kwlist[] = { "database", "timeout", "detect_types", "isolation_level", "check_same_thread", "factory", "cached_statements", "uri", - NULL + "transaction_mode", NULL }; char* database; @@ -74,13 +75,15 @@ int check_same_thread = 1; int cached_statements = 100; int uri = 0; + PyObject* transaction_mode = NULL; double timeout = 5.0; int rc; - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s|diOiOip", kwlist, + if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s|diOiOipO", kwlist, &database, &timeout, &detect_types, &isolation_level, &check_same_thread, - &factory, &cached_statements, &uri)) + &factory, &cached_statements, &uri, + &transaction_mode)) { return -1; } @@ -119,20 +122,39 @@ return -1; } - if (!isolation_level) { - isolation_level = PyUnicode_FromString(""); - if (!isolation_level) { + if (isolation_level) { + /* Prevent one option from silently overriding the other */ + if (transaction_mode) { + PyErr_SetString(pysqlite_ProgrammingError, "isolation_level is incompatible with transaction_mode"); return -1; } - } else { + Py_INCREF(isolation_level); + self->isolation_level = NULL; + self->transaction_mode = NULL; + if (pysqlite_connection_set_isolation_level(self, isolation_level) < 0) { + Py_DECREF(isolation_level); + return -1; + } + Py_DECREF(isolation_level); + + } else { // !isolation_level + if (transaction_mode) { + Py_INCREF(transaction_mode); + } else { + transaction_mode = PyUnicode_FromString("traditional"); + if (!transaction_mode) { + return -1; + } + } + self->isolation_level = NULL; + self->transaction_mode = NULL; + if (pysqlite_connection_set_transaction_mode(self, transaction_mode) < 0) { + Py_DECREF(transaction_mode); + return -1; + } + Py_DECREF(transaction_mode); } - self->isolation_level = NULL; - if (pysqlite_connection_set_isolation_level(self, isolation_level) < 0) { - Py_DECREF(isolation_level); - return -1; - } - Py_DECREF(isolation_level); self->statement_cache = (pysqlite_Cache*)PyObject_CallFunction((PyObject*)&pysqlite_CacheType, "Oi", self, cached_statements); if (PyErr_Occurred()) { @@ -1156,6 +1178,12 @@ return 1; } +static PyObject* pysqlite_connection_get_autocommit(pysqlite_Connection* self, void* unused) +{ + // TODO(aaugustin) + return NULL; +} + static PyObject* pysqlite_connection_get_isolation_level(pysqlite_Connection* self, void* unused) { Py_INCREF(self->isolation_level); @@ -1171,23 +1199,88 @@ } } -static int pysqlite_connection_set_isolation_level(pysqlite_Connection* self, PyObject* isolation_level) +static PyObject* pysqlite_connection_get_transaction_mode(pysqlite_Connection* self, void* unused) { - PyObject* res; - PyObject* begin_statement; - static PyObject* begin_word; + Py_INCREF(self->transaction_mode); + return self->transaction_mode; +} - Py_XDECREF(self->isolation_level); +static int pysqlite_connection_set_autocommit(pysqlite_Connection* self, PyObject* autocommit) +{ + // TODO(aaugustin) + return 0; +} +static int _pysqlite_unset_begin_statement(pysqlite_Connection* self) +{ if (self->begin_statement) { PyMem_Free(self->begin_statement); self->begin_statement = NULL; } - if (isolation_level == Py_None) { + return 0; +} + +static int _pysqlite_set_begin_statement(pysqlite_Connection* self, PyObject* transaction_behavior) +{ + static PyObject* begin_word; + PyObject* begin_statement; + const char *statement; + Py_ssize_t size; + + if (_pysqlite_unset_begin_statement(self) < 0) { + return -1; + } + + if (!begin_word) { + begin_word = PyUnicode_FromString("BEGIN "); + if (!begin_word) { + return -1; + } + } + + begin_statement = PyUnicode_Concat(begin_word, transaction_behavior); + if (!begin_statement) { + return -1; + } + + statement = PyUnicode_AsUTF8AndSize(begin_statement, &size); + if (!statement) { + Py_DECREF(begin_statement); + return -1; + } + + self->begin_statement = PyMem_Malloc(size + 2); + if (!self->begin_statement) { + Py_DECREF(begin_statement); + return -1; + } + + strcpy(self->begin_statement, statement); + Py_DECREF(begin_statement); + + return 0; +} + +static int pysqlite_connection_set_isolation_level(pysqlite_Connection* self, PyObject* transaction_behavior) +{ + PyObject* res; + char* transaction_behavior_utf8; + + if (transaction_behavior == Py_None) { + Py_XDECREF(self->isolation_level); Py_INCREF(Py_None); self->isolation_level = Py_None; + Py_XDECREF(self->transaction_mode); + self->transaction_mode = PyUnicode_FromString("autocommit"); + if (!self->transaction_mode) { + return -1; + } + self->transaction_mode_code = 'a'; + + _pysqlite_unset_begin_statement(self); + res = pysqlite_connection_commit(self, NULL); if (!res) { return -1; @@ -1195,38 +1288,181 @@ Py_DECREF(res); self->inTransaction = 0; + } else { - const char *statement; - Py_ssize_t size; + Py_XDECREF(self->isolation_level); + Py_INCREF(transaction_behavior); + self->isolation_level = transaction_behavior; - Py_INCREF(isolation_level); - self->isolation_level = isolation_level; + transaction_behavior_utf8 = PyUnicode_AsUTF8(transaction_behavior); + if (!transaction_behavior_utf8) { + return -1; + } + Py_XDECREF(self->transaction_mode); + if (0 == PyOS_stricmp(transaction_behavior_utf8, "")) { + self->transaction_mode = PyUnicode_FromString("traditional"); + } else if (0 == PyOS_stricmp(transaction_behavior_utf8, "DEFERRED")) { + self->transaction_mode = PyUnicode_FromString("traditional deferred"); + } else if (0 == PyOS_stricmp(transaction_behavior_utf8, "IMMEDIATE")) { + self->transaction_mode = PyUnicode_FromString("traditional immediate"); + } else if (0 == PyOS_stricmp(transaction_behavior_utf8, "EXCLUSIVE")) { + self->transaction_mode = PyUnicode_FromString("traditional exclusive"); + } else { + /* This is a reasonable fallback for undefined values */ + self->transaction_mode = PyUnicode_FromString("traditional"); + } + if (!self->transaction_mode) { + return -1; + } + self->transaction_mode_code = 't'; - if (!begin_word) { - begin_word = PyUnicode_FromString("BEGIN "); - if (!begin_word) return -1; + if (_pysqlite_set_begin_statement(self, transaction_behavior) < 0) { + return -1; } - begin_statement = PyUnicode_Concat(begin_word, isolation_level); - if (!begin_statement) { + } + + return 0; +} + +static int pysqlite_connection_set_transaction_mode(pysqlite_Connection* self, PyObject* transaction_mode) +{ + PyObject* transaction_mode_split; + Py_ssize_t transaction_mode_parts; + char* mode_utf8; + char* behavior_utf8; + char* behavior_utf8_upper; + static PyObject* space; + PyObject* behavior; + PyObject* mode_and_behavior; + + if (self->inTransaction) { + PyErr_SetString(pysqlite_ProgrammingError, + "a transaction is in progress, commit or rollback before changing transaction_mode"); + return -1; + } + + /* Split transaction_mode into a mode (traditional, standard, autocommit) + * and possibly a behavior (, deferred, immediate, exclusive) + */ + + transaction_mode_split = PyUnicode_Split(transaction_mode, NULL, 1); + if (!transaction_mode) { + return -1; + } + transaction_mode_parts = PyList_Size(transaction_mode_split); + if (transaction_mode_parts > 0) { + mode_utf8 = PyUnicode_AsUTF8(PyList_GetItem(transaction_mode_split, 0)); + if (!mode_utf8) { + Py_DECREF(transaction_mode_split); + return -1; + } + } else { + mode_utf8 = NULL; + } + if (transaction_mode_parts > 1) { + behavior_utf8 = PyUnicode_AsUTF8(PyList_GetItem(transaction_mode_split, 1)); + if (!behavior_utf8) { + Py_DECREF(transaction_mode_split); + return -1; + } + } else { + behavior_utf8 = NULL; + } + Py_DECREF(transaction_mode_split); + + /* Validate transaction_mode case-insensitively and normalize it */ + + if (0 == PyOS_stricmp(mode_utf8, "autocommit")) { + mode_utf8 = "autocommit"; + } else if (0 == PyOS_stricmp(mode_utf8, "standard")) { + mode_utf8 = "standard"; + } else if (0 == PyOS_stricmp(mode_utf8, "traditional")) { + mode_utf8 = "traditional"; + } else { + PyErr_SetString(pysqlite_ProgrammingError, "invalid transaction mode"); + return -1; + } + + if (behavior_utf8 == NULL) { + /* Behavior is always optional */ + behavior_utf8_upper = NULL; + } else if (mode_utf8[0] == 'a') { + /* Behavior isn't supported in autocommit mode */ + PyErr_SetString(pysqlite_ProgrammingError, "invalid transaction mode"); + return -1; + } else if (0 == PyOS_stricmp(behavior_utf8, "deferred")) { + behavior_utf8 = "deferred"; + behavior_utf8_upper = "DEFERRED"; + } else if (0 == PyOS_stricmp(behavior_utf8, "immediate")) { + behavior_utf8 = "immediate"; + behavior_utf8_upper = "IMMEDIATE"; + } else if (0 == PyOS_stricmp(behavior_utf8, "exclusive")) { + behavior_utf8 = "exclusive"; + behavior_utf8_upper = "EXCLUSIVE"; + } else { + PyErr_SetString(pysqlite_ProgrammingError, "invalid transaction mode"); + return -1; + } + + Py_XDECREF(self->transaction_mode); + self->transaction_mode = PyUnicode_FromString(mode_utf8); + if (!self->transaction_mode) { + return -1; + } + if (behavior_utf8) { + if (!space) { + space = PyUnicode_FromString(" "); + if (!space) { + return -1; + } + } + behavior = PyUnicode_FromString(behavior_utf8); + if (!behavior) { + return -1; + } + mode_and_behavior = PyList_New(2); + if (!mode_and_behavior) { + Py_DECREF(behavior); + return -1; + } + if (!PyList_SetItem(mode_and_behavior, 0, self->transaction_mode) || + !PyList_SetItem(mode_and_behavior, 1, behavior)) { + Py_DECREF(behavior); + return -1; + } + self->transaction_mode = PyUnicode_Join(space, mode_and_behavior); + if (!self->transaction_mode) { + Py_DECREF(mode_and_behavior); + return -1; + } + Py_DECREF(mode_and_behavior); + } + + self->transaction_mode_code = mode_utf8[0]; + + if (self->transaction_mode_code == 'a') { + Py_XDECREF(self->isolation_level); + Py_INCREF(Py_None); + self->isolation_level = Py_None; + + _pysqlite_unset_begin_statement(self); + + } else { + Py_XDECREF(self->isolation_level); + if (behavior_utf8_upper) { + self->isolation_level = PyUnicode_FromString(behavior_utf8_upper); + } else { + self->isolation_level = PyUnicode_FromString(""); + } + + if (_pysqlite_set_begin_statement(self, self->isolation_level) < 0) { return -1; } - statement = _PyUnicode_AsStringAndSize(begin_statement, &size); - if (!statement) { - Py_DECREF(begin_statement); - return -1; - } - self->begin_statement = PyMem_Malloc(size + 2); - if (!self->begin_statement) { - Py_DECREF(begin_statement); - return -1; - } - - strcpy(self->begin_statement, statement); - Py_DECREF(begin_statement); } return 0; + } PyObject* pysqlite_connection_call(pysqlite_Connection* self, PyObject* args, PyObject* kwargs) @@ -1624,8 +1860,10 @@ PyDoc_STR("SQLite database connection object."); static PyGetSetDef connection_getset[] = { + {"autocommit", (getter)pysqlite_connection_get_autocommit, (setter)pysqlite_connection_set_autocommit}, {"isolation_level", (getter)pysqlite_connection_get_isolation_level, (setter)pysqlite_connection_set_isolation_level}, {"total_changes", (getter)pysqlite_connection_get_total_changes, (setter)0}, + {"transaction_mode", (getter)pysqlite_connection_get_transaction_mode, (setter)pysqlite_connection_set_transaction_mode}, {NULL} }; diff -r 3151f6f9df85 Modules/_sqlite/connection.h --- a/Modules/_sqlite/connection.h Thu Jun 26 01:41:06 2014 -0400 +++ b/Modules/_sqlite/connection.h Tue Aug 25 09:04:09 2015 +0200 @@ -52,9 +52,18 @@ * first get called with count=0? */ double timeout_started; - /* None for autocommit, otherwise a PyString with the isolation level */ + /* None for autocommit, otherwise a PyString with the "isolation level" + * (a deprecated concept unrelated to SQL transaction isolation levels) */ PyObject* isolation_level; + /* combination of a transaction mode (traditional, standard, autocommit) + * and optionally a transaction behavior (deferred, immediate, exclusive); + * behaviors only apply to the legacy and standard modes. */ + PyObject* transaction_mode; + + /* 't', 's', 'a' for traditional, standard, autocommit respectively */ + char transaction_mode_code; + /* NULL for autocommit, otherwise a string with the BEGIN statement; will be * freed in connection destructor */ char* begin_statement; diff -r 3151f6f9df85 Modules/_sqlite/module.c --- a/Modules/_sqlite/module.c Thu Jun 26 01:41:06 2014 -0400 +++ b/Modules/_sqlite/module.c Tue Aug 25 09:04:09 2015 +0200 @@ -53,7 +53,7 @@ static char *kwlist[] = { "database", "timeout", "detect_types", "isolation_level", "check_same_thread", "factory", "cached_statements", "uri", - NULL + "transaction_mode", NULL }; char* database; int detect_types = 0; @@ -62,14 +62,16 @@ int check_same_thread = 1; int cached_statements; int uri = 0; + PyObject* transaction_mode = NULL; double timeout = 5.0; PyObject* result; - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s|diOiOip", kwlist, + if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s|diOiOipO", kwlist, &database, &timeout, &detect_types, &isolation_level, &check_same_thread, - &factory, &cached_statements, &uri)) + &factory, &cached_statements, &uri, + &transaction_mode)) { return NULL; }