ndbm can't iterate through values on OS X #74573

ForestGregg · 2017-05-17T16:03:55Z

BPO	30388
Nosy	@terryjreedy, @ronaldoussoren, @ned-deily, @emilyemorehouse
Files	ndbm.zip: The ndbm files created by shelve. ndbm-repro.c

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-05-17.16:03:55.042>
labels = ['OS-mac', 'type-bug', 'library']
title = "ndbm can't iterate through values on OS X"
updated_at = <Date 2020-11-27.11:32:30.539>
user = 'https://bugs.python.org/ForestGregg'

bugs.python.org fields:

activity = <Date 2020-11-27.11:32:30.539>
actor = 'ronaldoussoren'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'macOS']
creation = <Date 2017-05-17.16:03:55.042>
creator = 'Forest Gregg'
dependencies = []
files = ['46872', '49627']
hgrepos = []
issue_num = 30388
keywords = []
message_count = 11.0
messages = ['293858', '293934', '293974', '293975', '294040', '295189', '295215', '295282', '295512', '380963', '381944']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'ronaldoussoren', 'ned.deily', 'emilyemorehouse', 'Forest Gregg']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30388'
versions = ['Python 3.5']

Linked PRs

ForestGregg · 2017-05-17T16:03:55Z

On Mac OS 10.12.4, a large shelve, backed by ndbm, can be created. But when I attempt to iterate through the values of the shelve it raises this exception:

File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/dedupe/api.py", line 281, in _blockData
for block in viewvalues(blocks):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/collections_abc.py", line 693, in __iter_
for key in self._mapping:
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shelve.py", line 95, in __iter__
for k in self.dict.keys():
SystemError: Negative size passed to PyBytes_FromStringAndSize

I've confirmed that this works with Python 3.6.1. All the Python versions were installed from homebrew.

I cannot reproduce on linux on windows machines. I've attached a zip file of the ndbm files that python created.

ForestGregg · 2017-05-18T18:16:42Z

The ndbm db's two files (in the attachment) have the following size

tmp___otctx 0 bytes
tmp___otctx.db 12857344 bytes

terryjreedy · 2017-05-19T21:02:05Z

Do you mean that on OSX, you code works with 3.6.1 and fails with 3.5.3?

ForestGregg · 2017-05-19T21:04:31Z

Very sorry for the ambiguity.

The bug appears for both python 3.5 and python 3.6.1 on OS X. I have not tried other versions of python on OS X.

ned-deily · 2017-05-20T17:32:32Z

Can you provide a complete example of how to reproduce the behavior you are seeing? Ideally, it would include producing the shelve object. Also, please provide the output of this command:

otool -L $(python3.6 -c 'import _dbm;print(dbm.__file_)')

ForestGregg · 2017-06-05T13:57:53Z

I have been trying to make a small reproducible example, but haven't been able to isolate it.

Running this script [1] csv_example on mac os x under either py27 or py3 does seem *often* cause this problem [2], [3]:

[1] https://github.com/dedupeio/dedupe-examples/blob/master/csv_example/csv_example.py
[2] dedupeio/dedupe#571
[3] dedupeio/dedupe-examples#54

ned-deily · 2017-06-05T22:12:45Z

It would be helpful if you could provide the output of this command:

otool -L $(python3.6 -c 'import _dbm;print(dbm.__file_)')

if necessary, substituting whatever command name(s) you are using to invoke python when failing.

ForestGregg · 2017-06-06T17:42:24Z

From one user who had problems under both 3.5 and 3.6

otool -L $(python3.5 -c 'import _dbm;print(dbm.__file_)')

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload/_dbm.cpython-35m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 125.2.0)

otool -L $(python3.6 -c 'import _dbm;print(dbm.__file_)')

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload/_
dbm.cpython-36m-darwin.so:

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
125.2.0)

ForestGregg · 2017-06-09T10:48:27Z

A different user:

otool -L $(python3.6 -c 'import _dbm;print(_dbm.file)')
/usr/local/var/pyenv/versions/3.6.1/lib/python3.6/lib-dynload/_dbm.cpython-36m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)

ronaldoussoren · 2020-11-14T08:45:27Z

See also bpo-33074

ronaldoussoren · 2020-11-27T11:32:30Z

I've attached a C program that demonstrates the problem, and have filed an issue with Apple about this: FB8919203

It is not clear to me what we can do about this, other than not building the ndbm binding on macOS. The manpages for the ndbm APIs do not mention limitations on item sizes, and the dbopen manpage mention that there a no limitations on item sizes).

…macOS The system ndbm implementation on macOS has an undocumented limitation on the size of values and can silently corrupt database files when those are exceeded.

ronaldoussoren · 2023-12-21T11:03:31Z

The best we can do is document the issue.

…113354) * gh-74573: document that ndbm can silently corrupt databases on macOS The system ndbm implementation on macOS has an undocumented limitation on the size of values and can silently corrupt database files when those are exceeded. Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

…macOS (pythonGH-113354) * pythongh-74573: document that ndbm can silently corrupt databases on macOS The system ndbm implementation on macOS has an undocumented limitation on the size of values and can silently corrupt database files when those are exceeded. (cherry picked from commit 593b4d8) Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

… macOS (GH-113354) (#113431) gh-74573: document that ndbm can silently corrupt databases on macOS (GH-113354) * gh-74573: document that ndbm can silently corrupt databases on macOS The system ndbm implementation on macOS has an undocumented limitation on the size of values and can silently corrupt database files when those are exceeded. (cherry picked from commit 593b4d8) Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

… macOS (GH-113354) (#113432) gh-74573: document that ndbm can silently corrupt databases on macOS (GH-113354) * gh-74573: document that ndbm can silently corrupt databases on macOS The system ndbm implementation on macOS has an undocumented limitation on the size of values and can silently corrupt database files when those are exceeded. (cherry picked from commit 593b4d8) Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

ronaldoussoren · 2023-12-23T12:50:58Z

The just merged PRs document the problem in main, 3.12 and 3.11. As we cannot fix Apple's code that's the best we can do.

…macOS (python#113354) * pythongh-74573: document that ndbm can silently corrupt databases on macOS The system ndbm implementation on macOS has an undocumented limitation on the size of values and can silently corrupt database files when those are exceeded. Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

ForestGregg mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels May 17, 2017

terryjreedy added the OS-mac label May 19, 2017

ezio-melotti transferred this issue from another repository Apr 10, 2022

ronaldoussoren mentioned this issue Apr 11, 2022

Database corruption with the shelve module #91228

Open

ambv mentioned this issue May 3, 2022

test_dbm and test_dbm_ndbm segfaults in M1 Mac with Homebrew gdbm 1.23 #89452

Closed

bedevere-app bot mentioned this issue Dec 21, 2023

gh-74573: document that ndbm can silently corrupt databases on macOS #113354

Merged

This was referenced Dec 23, 2023

[3.12] gh-74573: document that ndbm can silently corrupt databases on macOS (GH-113354) #113431

Merged

[3.11] gh-74573: document that ndbm can silently corrupt databases on macOS (GH-113354) #113432

Merged

ronaldoussoren closed this as completed Dec 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ndbm can't iterate through values on OS X #74573

ndbm can't iterate through values on OS X #74573

ForestGregg mannequin commented May 17, 2017 •

edited by bedevere-app bot

ForestGregg mannequin commented May 17, 2017

ForestGregg mannequin commented May 18, 2017

terryjreedy commented May 19, 2017

ForestGregg mannequin commented May 19, 2017

ned-deily commented May 20, 2017

ForestGregg mannequin commented Jun 5, 2017

ned-deily commented Jun 5, 2017

ForestGregg mannequin commented Jun 6, 2017

ForestGregg mannequin commented Jun 9, 2017

ronaldoussoren commented Nov 14, 2020

ronaldoussoren commented Nov 27, 2020

ronaldoussoren commented Dec 21, 2023

ronaldoussoren commented Dec 23, 2023

ndbm can't iterate through values on OS X #74573

ndbm can't iterate through values on OS X #74573

Comments

ForestGregg mannequin commented May 17, 2017 • edited by bedevere-app bot

Linked PRs

ForestGregg mannequin commented May 17, 2017

ForestGregg mannequin commented May 18, 2017

terryjreedy commented May 19, 2017

ForestGregg mannequin commented May 19, 2017

ned-deily commented May 20, 2017

ForestGregg mannequin commented Jun 5, 2017

ned-deily commented Jun 5, 2017

ForestGregg mannequin commented Jun 6, 2017

ForestGregg mannequin commented Jun 9, 2017

ronaldoussoren commented Nov 14, 2020

ronaldoussoren commented Nov 27, 2020

ronaldoussoren commented Dec 21, 2023

ronaldoussoren commented Dec 23, 2023

ForestGregg mannequin commented May 17, 2017 •

edited by bedevere-app bot