classification
Title: ndbm can't iterate through values on OS X
Type: behavior Stage:
Components: Library (Lib), macOS Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Forest Gregg, emilyemorehouse, ned.deily, ronaldoussoren, terry.reedy
Priority: normal Keywords:

Created on 2017-05-17 16:03 by Forest Gregg, last changed 2017-06-09 10:48 by Forest Gregg.

Files
File name Uploaded Description Edit
ndbm.zip Forest Gregg, 2017-05-17 16:03 The ndbm files created by shelve.
Messages (9)
msg293858 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-05-17 16:03
On Mac OS 10.12.4, a large shelve, backed by ndbm, can be created. But when I attempt to iterate through the values of the shelve it raises this exception:


  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/dedupe/api.py", line 281, in _blockData
    for block in viewvalues(blocks):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_collections_abc.py", line 693, in __iter__
    for key in self._mapping:
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shelve.py", line 95, in __iter__
    for k in self.dict.keys():
SystemError: Negative size passed to PyBytes_FromStringAndSize

I've confirmed that this works with Python 3.6.1. All the Python versions were installed from homebrew.

I cannot reproduce on linux on windows machines. I've attached a zip file of the ndbm files that python created.
msg293934 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-05-18 18:16
The ndbm db's two files (in the attachment) have the following size

tmp___otctx 0 bytes
tmp___otctx.db 12857344 bytes
msg293974 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-05-19 21:02
Do you mean that on OSX, you code works with 3.6.1 and fails with 3.5.3?
msg293975 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-05-19 21:04
Very sorry for the ambiguity. 

The bug appears for both python 3.5 and python 3.6.1 on OS X. I have not tried other versions of python on OS X.
msg294040 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-05-20 17:32
Can you provide a complete example of how to reproduce the behavior you are seeing?  Ideally, it would include producing the shelve object.  Also, please provide the output of this command:

otool -L $(python3.6 -c 'import _dbm;print(_dbm.__file__)')
msg295189 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-06-05 13:57
I have been trying to make a small reproducible example, but haven't been able to isolate it. 

Running this script [1] csv_example on mac os x under either py27 or py3 does seem *often* cause this problem [2], [3]:

[1] https://github.com/dedupeio/dedupe-examples/blob/master/csv_example/csv_example.py
[2] https://github.com/dedupeio/dedupe/issues/571
[3] https://github.com/dedupeio/dedupe-examples/issues/54
msg295215 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-06-05 22:12
It would be helpful if you could provide the output of this command:

otool -L $(python3.6 -c 'import _dbm;print(_dbm.__file__)')

if necessary, substituting whatever command name(s) you are using to invoke python when failing.
msg295282 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-06-06 17:42
From one user who had problems under both 3.5 and 3.6

otool -L $(python3.5 -c 'import _dbm;print(_dbm.__file__)')

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload/_dbm.cpython-35m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 125.2.0)

otool -L $(python3.6 -c 'import _dbm;print(_dbm.__file__)')

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload/_
dbm.cpython-36m-darwin.so:

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
125.2.0)
msg295512 - (view) Author: Forest Gregg (Forest Gregg) Date: 2017-06-09 10:48
A different user:

otool -L $(python3.6 -c 'import _dbm;print(_dbm.file)')
/usr/local/var/pyenv/versions/3.6.1/lib/python3.6/lib-dynload/_dbm.cpython-36m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
History
Date User Action Args
2017-06-09 10:48:26Forest Greggsetmessages: + msg295512
2017-06-06 17:42:23Forest Greggsetmessages: + msg295282
2017-06-05 22:12:45ned.deilysetmessages: + msg295215
2017-06-05 13:57:53Forest Greggsetmessages: + msg295189
2017-05-21 20:50:03emilyemorehousesetnosy: + emilyemorehouse
2017-05-20 17:32:31ned.deilysetmessages: + msg294040
2017-05-19 21:04:30Forest Greggsetmessages: + msg293975
2017-05-19 21:02:05terry.reedysetnosy: + terry.reedy, ned.deily, ronaldoussoren
messages: + msg293974
components: + macOS
2017-05-18 18:16:42Forest Greggsetmessages: + msg293934
2017-05-17 16:04:36Forest Greggsettype: behavior
2017-05-17 16:03:55Forest Greggcreate