classification
Title: sqlite only accept buffer() for BLOB objects (input/output)
Type: behavior Stage: needs patch
Components: Extension Modules Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: ghaering Nosy List: benjamin.peterson, flox, ghaering, lemburg, merwok, pablomouzo, pitrou
Priority: normal Keywords: patch

Created on 2010-01-17 13:45 by flox, last changed 2011-10-18 18:54 by flox. This issue is now closed.

Files
File name Uploaded Description Edit
issue7723-py3k-docs.diff pablomouzo, 2010-01-23 03:27 removing old buffer references from sqlite3 docs review
Messages (13)
msg97943 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-01-17 13:45
Since buffer() is deprecated in Python 2.7, it should not be used for BLOB input/output.


SAMPLES = (
  ('unicode', u''),
  ('bytes', ''),
  ('buffer', buffer('')),
# ('bytearray', bytearray('')),     # unsupported
# ('memoryview', memoryview(''))    # unsupported
)

import sqlite3
conn = sqlite3.connect(':memory:')
c = conn.cursor()
c.execute('create table test(s varchar, b blob)')
c.executemany('insert into test(s, b) values (?, ?)', SAMPLES)
c.execute('select s, b from test')
print('\n'.join(str(l) for l in c.fetchall()))

# Output:
# (u'unicode', u'')
# (u'bytes', u'')
# (u'buffer', <read-write buffer ptr 0x1d8ccd0, size 0 at 0x1d8cc80>)
#
# Errors and warnings:
# __main__:4: DeprecationWarning: buffer() not supported in 3.x
# sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type.


Here is the Python 3 input/output:

SAMPLES = (  # Python3
  ('unicode', ''),
  ('bytes', b''),
  ('bytearray', bytearray(b'')),
  ('memoryview', memoryview(b''))
)

# ('unicode',     '')
# ('bytes',      b'')
# ('bytearray',  b'')
# ('memoryview', b'')
msg97948 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-01-17 14:48
Sidenotes:
 - documentation for Python 3 is outdated
 - it may be a release blocker, since there's no alternative:
   currently it uses exclusively buffer() for BLOB object
msg97958 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-01-17 17:51
buffer() is only deprecated in Python 3.x, not in Python 2.7, so the current implementation is perfectly valid for Python 2.7.

Note that buffer() has long been the preferred type for passing binary objects to and from a database.
msg97963 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-01-17 19:02
This is not exact. buffer() doesn't exist at all in 3.x, and it issues a warning in 2.x when used with the -3 flag:

$ ./python -3
Python 2.7a2+ (trunk:77580M, Jan 17 2010, 16:51:51) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> buffer('abc')
__main__:1: DeprecationWarning: buffer() not supported in 3.x
<read-only buffer for 0x7f913ac79800, size -1, offset 0 at 0x7f913aba8570>
msg97985 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-01-17 23:22
Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> This is not exact. buffer() doesn't exist at all in 3.x, and it issues a warning in 2.x when used with the -3 flag:
> 
> $ ./python -3
> Python 2.7a2+ (trunk:77580M, Jan 17 2010, 16:51:51) 
> [GCC 4.4.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> buffer('abc')
> __main__:1: DeprecationWarning: buffer() not supported in 3.x
> <read-only buffer for 0x7f913ac79800, size -1, offset 0 at 0x7f913aba8570>

Sorry, that's what I meant. It's not deprecated in the 2.x branch of Python.
msg98050 - (view) Author: Gerhard Häring (ghaering) * (Python committer) Date: 2010-01-19 10:07
As far as I can see the docs for Python 3.x need to be adjusted. In particular http://docs.python.org/3.1/library/sqlite3.html#sqlite-and-python-types

buffer should then there probably be replaced by memoryview. As far as I can see, the Python 3 definition of the old "buffer" interface is "anything that I can apply memoryview to". At least that was my interpretation when doing the Python 3 port of the sqlite3 module.

Correct?

When returning BLOBs, the sqlite3 module now returns bytes objects. I think this could be changed to memoryview without causing any problems.
msg98051 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-01-19 13:09
Hello Gerhard,

> As far as I can see, the Python 3 definition of the old "buffer"
> interface is "anything that I can apply memoryview to".

Roughly, yes.

> When returning BLOBs, the sqlite3 module now returns bytes objects. I
> think this could be changed to memoryview without causing any problems.

No, returning bytes objects is the right thing to do under py3k. This "bug" report is only about 2.x, where "bytes" is the same as "str" and therefore doesn't allow to distinguish text from binary data.
msg98175 - (view) Author: Pablo Mouzo (pablomouzo) Date: 2010-01-23 03:27
I'm attaching a correction to sqlite3 documentation, removing all 'buffer' references and setting bytes as BLOB equivalent.
msg98768 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-02-02 23:02
Pablo's documentation patch for py3k has been committed.
msg100658 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-03-08 17:46
So we keep buffer() as the standard way to create BLOBs for 2.x?

It is the only use of "py3k deprecated" buffer() which cannot be replaced in 2.x.

Set to "release blocker" until a decision is made.
msg102800 - (view) Author: Gerhard Häring (ghaering) * (Python committer) Date: 2010-04-10 22:46
I see that the status of this issue keeps changing.

Now does anything in the sqlite3 module or the docs need to be changed?

Or what's left to close this?
msg105305 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-05-08 16:07
Documentation can always be updated later.
msg115538 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-09-03 23:15
I propose to close this, since 2.7 is released and the situation is better in Python 3.
History
Date User Action Args
2011-10-18 18:54:03floxsetstatus: pending -> closed
2010-09-03 23:15:51floxsetstatus: open -> pending
resolution: wont fix
messages: + msg115538
2010-07-31 16:35:28merwoksetnosy: + merwok
2010-07-31 12:06:36floxunlinkissue7092 dependencies
2010-05-08 16:07:52benjamin.petersonsetpriority: release blocker -> normal
nosy: + benjamin.peterson
messages: + msg105305

2010-04-10 22:46:39ghaeringsetmessages: + msg102800
2010-04-10 19:29:55benjamin.petersonsetpriority: deferred blocker -> release blocker
2010-04-03 15:26:35benjamin.petersonsetpriority: release blocker -> deferred blocker
2010-03-08 17:46:36floxsetpriority: critical -> release blocker

messages: + msg100658
2010-02-02 23:02:16pitrousetmessages: + msg98768
2010-01-23 03:27:32pablomouzosetfiles: + issue7723-py3k-docs.diff

nosy: + pablomouzo
messages: + msg98175

keywords: + patch
2010-01-19 13:09:52pitrousetmessages: + msg98051
2010-01-19 10:07:53ghaeringsetmessages: + msg98050
2010-01-17 23:22:51lemburgsetmessages: + msg97985
title: sqlite only accept buffer() for BLOB objects (input/output) -> sqlite only accept buffer() for BLOB objects (input/output)
2010-01-17 19:02:48pitrousetnosy: + pitrou
messages: + msg97963
2010-01-17 17:51:55lemburgsetnosy: + lemburg
messages: + msg97958
2010-01-17 14:48:35floxsetmessages: + msg97948
2010-01-17 14:23:15floxsetpriority: high -> critical
2010-01-17 14:05:46pitrousetassignee: ghaering

nosy: + ghaering
stage: needs patch
2010-01-17 13:49:06floxlinkissue7092 dependencies
2010-01-17 13:45:51floxcreate