classification
Title: sqlite.iterdump does not work for (most) databases with autoincrement
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, erlendaasland, itssme
Priority: normal Keywords: patch

Created on 2018-09-28 06:51 by itssme, last changed 2021-07-15 21:18 by erlendaasland.

Pull Requests
URL Status Linked Edit
PR 9621 open itssme, 2018-09-28 13:04
Messages (3)
msg326610 - (view) Author: Joel Klimont (itssme) * Date: 2018-09-28 06:51
There is a bug in sqlite3/dump.py when wanting to dump databases that use autoincrement in one or more tables.

The problem is that the iterdump command assumes that the table "sqlite_sequence" is present in the new database in which the old one is dumped into.

From the sqlite3 documentation:
"SQLite keeps track of the largest ROWID using an internal table named "sqlite_sequence". The sqlite_sequence table is created and initialized automatically whenever a normal table that contains an AUTOINCREMENT column is created. The content of the sqlite_sequence table can be modified using ordinary UPDATE, INSERT, and DELETE statements."
Source: https://sqlite.org/autoinc.html#the_autoincrement_keyword

Example:
BEGIN TRANSACTION;
CREATE TABLE "posts" (
    id int primary key
);
INSERT INTO "posts" VALUES(0);
CREATE TABLE "tags" (
    id integer primary key autoincrement,
    tag varchar(256) unique,
    post int references posts
);
INSERT INTO "tags" VALUES(NULL, "test", 0);
COMMIT;

The following code should work but because of the assumption that "sqlite_sequence" exists it will fail:

>> import sqlite3
>> cx = sqlite3.connect("test.db")
>> for i in cx.iterdump():
>>    print i
>> cx2 = sqlite3.connect(":memory:")
>> query = "".join(line for line in cx.iterdump())
>> cx2.executescript(query)

Exception:
Traceback (most recent call last):
  File "/home/test.py", line 10, in <module>
    cx2.executescript(query)
sqlite3.OperationalError: no such table: sqlite_sequence

Here is the ouput of cx.iterdrump()
BEGIN TRANSACTION;
CREATE TABLE "posts" (
    id int primary key
);
INSERT INTO "posts" VALUES(0);
DELETE FROM "sqlite_sequence";
INSERT INTO "sqlite_sequence" VALUES('tags',1);
CREATE TABLE "tags" (
    id integer primary key autoincrement,
    tag varchar(256) unique,
    post int references posts
);
INSERT INTO "tags" VALUES(1,'test',0);
COMMIT;

As you can see the problem is that "DELETE FROM "sqlite_sequence";" and "INSERT INTO "sqlite_sequence" VALUES('tags',1);" are put into the dump before that table even exists. They should be put at the end of the transaction. (Like the sqlite3 command ".dump" does.)
Note that some databases that use autoincrement will work as it could be that a table with autoincrement is created before the sqlite_sequence commands are put into the dump.

File: https://github.com/python/cpython/blob/master/Lib/sqlite3/dump.py

I have already forked the repository, written tests etc. and if you want I will create a pull request.
msg326611 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-09-28 06:56
> I have already forked the repository, written tests etc. and if you
> want I will create a pull request.

Please do!

Note that 3.4 and 3.5 are in security-fix-only mode and let's decide whether fixing 2.7 is worth the trouble when you submit your PR. Thank you!
msg326657 - (view) Author: Joel Klimont (itssme) * Date: 2018-09-28 20:26
I made the pull request: https://github.com/python/cpython/pull/9621
History
Date User Action Args
2021-07-15 21:18:56erlendaaslandsetnosy: + erlendaasland
2018-09-28 20:26:37itssmesetmessages: + msg326657
2018-09-28 13:04:27itssmesetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request9019
2018-09-28 06:56:33berker.peksagsetversions: - Python 2.7, Python 3.4, Python 3.5
nosy: + berker.peksag

messages: + msg326611

stage: needs patch
2018-09-28 06:51:55itssmecreate