It used to be possible to use hashlib with arrays; it no longer seems
possible.

exarkun@charm:~$ python -c '
import sys, hashlib, array
print sys.version_info
print hashlib.sha1(array.array("b", [1, 2, 3])).hexdigest()
'
(2, 5, 2, 'final', 0)
7037807198c22a7d2b0807371d763779a84fdfcf
exarkun@charm:~$ ~/Projects/python/trunk/python -c '
import sys, hashlib, array
print sys.version_info
print hashlib.sha1(array.array("b", [1, 2, 3])).hexdigest()
'
sys.version_info(major=2, minor=7, micro=0, releaselevel='alpha', serial=0)
Traceback (most recent call last):
  File "<string>", line 4, in <module>
TypeError: object supporting the buffer API required
exarkun@charm:~$

That's probably because hashlib switched to using the new buffer API,
and array.array() only supports the old one in 2.x.

Perhaps hashlib should have some fallback code for such legacy types...

exarkun: would you like to propose a patch?

I would certainly like to, but unfortunately at present I am unable to.

Perhaps Gregory has some idea about how this can most easily be
resolved, since I think he did the work on #3745 which introduced this
change in behavior.

to confirm, this is only a problem in 2.7 trunk right?

if so, i won't rush a fix.

but yes fallback code for legacy types or adding buffer api support to 
array would work.

It no longer works with objects created with buffer() either:

>>> hashlib.sha1(buffer("x")).hexdigest()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object supporting the buffer API required

> It no longer works with objects created with buffer() either:

That's because the buffer objects don't implement the buffer protocol
(in particular, they don't define the Py_TPFLAGS_HAVE_NEWBUFFER flag,
and the bf_getbuffer/bf_releasebuffer operations that go with that flag.

Can this ticket be marked as a release blocker so it's not forgotten
about for 2.7?

Can't hurt :)

I tried to implement the new buffer API, but as soon as I add 
bf_getbuffer/bf_releasebuffer to PyBufferProcs writing an array to a file 
breaks:
  f.write(a)
  TypeError: must be contiguous buffer, not array.array

I searched through the file functions, but couldn't find the point where 
this happens. Has anybody a suggestion? Does file.write() use the old 
buffers? Doesn't it use reprfunc?

> I tried to implement the new buffer API, but as soon as I add 
> bf_getbuffer/bf_releasebuffer to PyBufferProcs writing an array to a file 
> breaks:

You should take a look at the array module in py3k, it supports the new
buffer API. With a bit of luck, the two relevant functions shouldn't be
too difficult to backport to the trunk.

By the way, to signal that it supports the new buffer API, you must add
Py_TPFLAGS_HAVE_NEWBUFFER to the type flags.

I added the two functions for the new buffer API. Having an exported 
memory view needs some handling elsewhere, so the array does not change. I 
also added tests for checking that behaviour.

Mainly I copypasted code from py3k which involved redirecting to 
array_resize() instead of doing it manually and doing the checking there.

The patch breaks five unit tests from other modules, I'll look into it.

I stumbled upon the following function:

static Py_ssize_t
convertbuffer(PyObject *arg, void **p, char **errmsg)
in Python/getargs.c

The first thing the function does is checking if the object implements 
the old buffer api, but also fails if pb->bf_releasebuffer != NULL. So I 
guess it's also making sure the new buffer api is not implemented.

What's the thought behind this? Removing that condition fixes three of 
the failing tests but breaks none.

> The patch breaks five unit tests from other modules, I'll look into it.

What are those tests?

> > The patch breaks five unit tests from other modules, I'll look into
> > it.
> 
> What are those tests?

test_codecs, test_ioctl, test_multiprocessing, test_socket and 
test_struct.

You asked "what" not "which" :)

test test_codecs failed -- Traceback (most recent call last):
  File "/Users/jan/src/python-svn/Lib/test/test_codecs.py", line 531, in 
test_array
    codecs.readbuffer_encode(array.array("c", "spam")),
TypeError: must be string or read-only buffer, not array.array


test test_ioctl failed -- Traceback (most recent call last):
  File "/Users/jan/src/python-svn/Lib/test/test_ioctl.py", line 34, in 
test_ioctl_mutate
    r = fcntl.ioctl(tty, termios.TIOCGPGRP, buf, 1)
TypeError: ioctl requires a file or file descriptor, an integer and 
optionally an integer or buffer argument


test test_multiprocessing failed -- Traceback (most recent call last):
  File "/Users/jan/src/python-svn/Lib/test/test_multiprocessing.py", 
line 1269, in test_connection
    self.assertEqual(conn.send_bytes(arr), None)
TypeError: must be string or read-only buffer, not array.array


test test_socket failed -- errors occurred; run in verbose mode for 
details
(have to recheck)


test test_struct failed -- Traceback (most recent call last):
  File "/Users/jan/src/python-svn/Lib/test/test_struct.py", line 468, in 
test_unpack_with_buffer
    value, = struct.unpack('>I', data)
error: unpack requires a string argument of length 4

> The first thing the function does is checking if the object implements 
> the old buffer api, but also fails if pb->bf_releasebuffer != NULL. So I 
> guess it's also making sure the new buffer api is not implemented.
> 
> What's the thought behind this?

The idea is that if a type implements the bf_releasebuffer function, it
shouldn't be used in a situation where the caller won't try to release
the buffer.

It looks a bit unwarranted though, because a type implementing the old
buffer API should be able to function without the releasing anyway.
Otherwise it wouldn't implement that API at all.

Martin, ISTR you did that addition, would you be opposed to removing the
NULL check on bf_releasebuffer when trying to get a buffer through the
old buffer API?

After thinking about it, we should remover the bf_releasebuffer checks
when using the old typecodes '#' (the doc should also clearly state that
these are unsafe and '*' is preferred).

If a type chooses to implement the /old/ buffer API and if a method
requires the /old/ buffer API instead of the new one (that is, by using
a '#' typecode), we have to accept these choices.

Here's a patch for trunk, combining the new buffer API for `array` and
the checks removal in getargs.c.

Moving to deferred blocker to get this out of the way for 2.6.3

I was looking at the remaining differences between Modules/arraymodule.c 
in python 2.7 and the 3k branch, in particular I was testing why 
including the changes to the function array_ass_slice into the patch 
breaks the unit test of the array module.

The manual resizing of the memory basically was replaced by a call to 
array_resize with the advantage, that checking for exported memory views 
is done there (which is not necessary because it's already done in the 
function) and the code gets clearer. I think when PyMem_RESIZE is called 
the pointer to the memory might change. So in 3k this now happens in 
array_resize, so the array->ob_item pointer changes but not it's local 
copy in array_ass_slice. Isn't that potentially causing trouble in 
python 3k?

I fixed the array_ass_slice for python 2.7 in the attached patch.

The problem should apply to python 3k as well: firstly the above which 
might stay unnoticed and secondly the function moves to much memory if the 
array size is increased: (Py_SIZE(a)-d-ihigh) items should be moved, 
because Py_SIZE(a) was already modified by array_resize, but the function 
moves (Py_SIZE(a)-ihigh) items.

The attached patch for python 2.7 passes all unit tests for me.

> I fixed the array_ass_slice for python 2.7 in the attached patch.

I would prefer if I opened a separate issue for this bug. That way it
will be easier to port the patch separately to py3k.

> I would prefer if I opened a separate issue for this bug.

if /you/ opened, of course...

I think the error is really in _hashlib, not in the array object. It
should not require 3.x style buffers, but continue to support 2.x
readbuffers. Attached is a patch that takes this route to fixing the bug.

As for the checks for bf_releasebuffer: I still think they are
necessary. If an object implements bf_releasebuffer, that means that the
object may change the buffer underneath, unless proper locking and
unlocking takes place. Indeed, the array's getreadbuf operation is not
thread-safe. It might be possible to remove them if it is clarified that
anybody calling getreadbuffer must not release the GIL while they hold
on to the buffer.

> As for the checks for bf_releasebuffer: I still think they are
> necessary. If an object implements bf_releasebuffer, that means that the
> object may change the buffer underneath, unless proper locking and
> unlocking takes place.

I know, but the problem is that by switching some argument definitions
to "s*" and friends we have broken compatibility for the (admittedly
uncommon) use case of giving an array object to those functions. Since
we probably don't want to backout those changes perhaps adding support
for the new buffer API to the array object is the best course of action.

Will this be fixed for Python 2.7 final?

I believe this was fixed in r77252, which was fixing http://bugs.python.org/issue3745

Even if it's fixed a patch with tests should be submitted before closing this issue.

Done in r77313.