This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: gdbm reorganize() leaves hanging file descriptor
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: jcea, sam-s, themoken
Priority: normal Keywords:

Created on 2012-02-06 04:13 by themoken, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (7)
msg152719 - (view) Author: Jack Miller (themoken) Date: 2012-02-06 04:13
I've discovered that using either 2.7.2 or 3.2.2 (from Arch) reorganizing gdbm databases leave hanging file descriptors that will cause EAGAIN errors trying to reopen it until the process is killed. For example:

Python 2.7.2 (default, Jan 31 2012, 13:19:49) 
[GCC 4.6.2 20120120 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdbm
>>> o = gdbm.open("test27", "c")
>>> o.reorganize()
>>> o.close()
>>> o = gdbm.open("test27", "w")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
gdbm.error: (11, 'Resource temporarily unavailable')
>>>

By using lsof you can see that after reorganize() test27 has two open file descriptors, and after close() one is still open with no (obvious) way to close it. This happens with freshly created and populated databases and regardless of open mode flags.

I also tested this on 3.2.2 with dbm.gnu and got identical behavior.
msg152720 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-02-06 04:48
I can not reproduce the issue with GDBM 1.9.1, neither GDBM 1.8.3, neither GDBM 1.10, under Solaris.

I don't see the file descriptor leak either.

According to http://sunsite.ualberta.ca/Documentation/Gnu/gdbm-1.8.0/html_node/gdbm_10.html, all file descriptor management is done inside GDBM.

What GDBM are you using?. Could you possibly upgrade?. Beware, 1.8 and 1.9/1.10 fileformats are *NOT* compatible.
msg152721 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-02-06 04:52
Can't reproduce either in Ubuntu 10.04, 64 bits, with GDBM 1.8.3.

Closing bug.

Reporter, please indicate your GDBM version. Can you possibly upgrade it?.
msg152722 - (view) Author: Jack Miller (themoken) Date: 2012-02-06 05:22
I am using gdbm 1.10, the latest available.

pacman says:
core/gdbm 1.10-1 [installed]
    GNU database library

Which also contains the file (libgdbm.so.4) that Python's gdbm.so is linked to (just in case there was some version trickery going on here).

I guess it's possible this is Arch brokenness, but the only patch applied to the 1.10 code in the PKGBUILD is malloc -> calloc for a known bug.
msg153050 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-02-10 14:46
I can't reproduce. Sorry.

Reading the GDBM documentation, all filedescriptors management seem to be inside gdbm, not Python. Anyway, if I can't reproduce, I can not diagnose neither patch. Sorry.
msg339398 - (view) Author: sds (sam-s) Date: 2019-04-03 15:58
I observe the same behavior on Linux 4.18.0-17-generic #18-Ubuntu
with Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0].
gdbm 1.14.1-6:
after

db = dbm.gnu.open("asdf","n")
db["a"] = "124"
db.reorganize()
db.close()

I see

$ lsof ~/asdf 
COMMAND   PID USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
python3 22071  sds mem-W  REG    8,2    16384 4328326 /home/sds/asdf

and

>>> db = dbm.gnu.open("asdf","r")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sds/src/pyapp/tests/fileconv_test.py", line 1053, in <module>
_gdbm.error: [Errno 11] Resource temporarily unavailable
msg339668 - (view) Author: sds (sam-s) Date: 2019-04-08 19:39
Upstream claims that this is fixed by http://git.gnu.org.ua/cgit/gdbm.git/commit/?id=a0d6c1a8
History
Date User Action Args
2022-04-11 14:57:26adminsetgithub: 58155
2019-04-08 19:39:39sam-ssetmessages: + msg339668
2019-04-03 15:59:33sam-ssetversions: + Python 3.6, - Python 3.7
2019-04-03 15:59:05sam-ssettype: behavior
2019-04-03 15:58:44sam-ssetcomponents: + Library (Lib)
versions: + Python 3.7, - Python 2.7, Python 3.2
2019-04-03 15:58:04sam-ssetnosy: + sam-s
messages: + msg339398
2012-02-10 14:46:47jceasetmessages: + msg153050
2012-02-06 05:22:13themokensetmessages: + msg152722
2012-02-06 04:52:03jceasetmessages: + msg152721
stage: resolved
2012-02-06 04:48:24jceasetstatus: open -> closed

nosy: + jcea
messages: + msg152720

resolution: works for me
2012-02-06 04:13:35themokencreate