classification
Title: ndbm can't iterate through values on OS X
Type: behavior Stage:
Components: Library (Lib), macOS Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Forest Gregg, emilyemorehouse, ned.deily, ronaldoussoren, terry.reedy
Priority: normal Keywords:

Created on 2017-05-17 16:03 by Forest Gregg, last changed 2020-11-27 11:32 by ronaldoussoren.

Files
File name Uploaded Description Edit
ndbm.zip Forest Gregg, 2017-05-17 16:03 The ndbm files created by shelve.
ndbm-repro.c ronaldoussoren, 2020-11-27 11:32
Messages (11)
msg293858 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-05-17 16:03
On Mac OS 10.12.4, a large shelve, backed by ndbm, can be created. But when I attempt to iterate through the values of the shelve it raises this exception:


  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/dedupe/api.py", line 281, in _blockData
    for block in viewvalues(blocks):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_collections_abc.py", line 693, in __iter__
    for key in self._mapping:
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shelve.py", line 95, in __iter__
    for k in self.dict.keys():
SystemError: Negative size passed to PyBytes_FromStringAndSize

I've confirmed that this works with Python 3.6.1. All the Python versions were installed from homebrew.

I cannot reproduce on linux on windows machines. I've attached a zip file of the ndbm files that python created.
msg293934 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-05-18 18:16
The ndbm db's two files (in the attachment) have the following size

tmp___otctx 0 bytes
tmp___otctx.db 12857344 bytes
msg293974 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-05-19 21:02
Do you mean that on OSX, you code works with 3.6.1 and fails with 3.5.3?
msg293975 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-05-19 21:04
Very sorry for the ambiguity. 

The bug appears for both python 3.5 and python 3.6.1 on OS X. I have not tried other versions of python on OS X.
msg294040 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-05-20 17:32
Can you provide a complete example of how to reproduce the behavior you are seeing?  Ideally, it would include producing the shelve object.  Also, please provide the output of this command:

otool -L $(python3.6 -c 'import _dbm;print(_dbm.__file__)')
msg295189 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-06-05 13:57
I have been trying to make a small reproducible example, but haven't been able to isolate it. 

Running this script [1] csv_example on mac os x under either py27 or py3 does seem *often* cause this problem [2], [3]:

[1] https://github.com/dedupeio/dedupe-examples/blob/master/csv_example/csv_example.py
[2] https://github.com/dedupeio/dedupe/issues/571
[3] https://github.com/dedupeio/dedupe-examples/issues/54
msg295215 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-06-05 22:12
It would be helpful if you could provide the output of this command:

otool -L $(python3.6 -c 'import _dbm;print(_dbm.__file__)')

if necessary, substituting whatever command name(s) you are using to invoke python when failing.
msg295282 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-06-06 17:42
From one user who had problems under both 3.5 and 3.6

otool -L $(python3.5 -c 'import _dbm;print(_dbm.__file__)')

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload/_dbm.cpython-35m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 125.2.0)

otool -L $(python3.6 -c 'import _dbm;print(_dbm.__file__)')

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload/_
dbm.cpython-36m-darwin.so:

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
125.2.0)
msg295512 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-06-09 10:48
A different user:

otool -L $(python3.6 -c 'import _dbm;print(_dbm.file)')
/usr/local/var/pyenv/versions/3.6.1/lib/python3.6/lib-dynload/_dbm.cpython-36m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
msg380963 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-14 08:45
See also #33074
msg381944 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-27 11:32
I've attached a C program that demonstrates the problem, and have filed an issue with Apple about this: FB8919203

It is not clear to me what we can do about this, other than not building the ndbm binding on macOS.  The manpages for the ndbm APIs do not mention limitations on item sizes, and the dbopen manpage mention that there a no limitations on item sizes).
History
Date User Action Args
2020-11-27 11:32:30ronaldoussorensetfiles: + ndbm-repro.c

messages: + msg381944
2020-11-14 08:45:26ronaldoussorensetmessages: + msg380963
2020-11-14 08:44:36ronaldoussorenlinkissue33074 superseder
2017-06-09 10:48:26Forest Greggsetmessages: + msg295512
2017-06-06 17:42:23Forest Greggsetmessages: + msg295282
2017-06-05 22:12:45ned.deilysetmessages: + msg295215
2017-06-05 13:57:53Forest Greggsetmessages: + msg295189
2017-05-21 20:50:03emilyemorehousesetnosy: + emilyemorehouse
2017-05-20 17:32:31ned.deilysetmessages: + msg294040
2017-05-19 21:04:30Forest Greggsetmessages: + msg293975
2017-05-19 21:02:05terry.reedysetnosy: + terry.reedy, ned.deily, ronaldoussoren
messages: + msg293974
components: + macOS
2017-05-18 18:16:42Forest Greggsetmessages: + msg293934
2017-05-17 16:04:36Forest Greggsettype: behavior
2017-05-17 16:03:55Forest Greggcreate