This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: intermitent failures with test_dbm
Type: Stage:
Components: Versions: Python 3.6, Python 3.4, Python 3.5
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asottile, ethan.furman, jcea
Priority: normal Keywords:

Created on 2013-12-29 16:46 by ethan.furman, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (7)
msg207079 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-12-29 16:46
Following errors occur about half the time:

======================================================================
ERROR: test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 75, in test_anydbm_creation
    self.read_helper(f)
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 117, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 90, in test_anydbm_modification
    self.read_helper(f)
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 117, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 96, in test_anydbm_read
    self.read_helper(f)
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 117, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'
msg207080 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-12-29 16:48
Actually, make that about 1/5 of the time.
msg217630 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2014-04-30 17:05
I can't reproduce on Linux 12.04. I tried the test a thousand times.

Ethan, what is your build environment?
msg217639 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-04-30 18:37
Actually, I haven't had this issue in quite a while now, so closing.

Thanks for taking a look at it, Jesús.
msg285081 - (view) Author: Anthony Sottile (asottile) * Date: 2017-01-09 23:29
I'm seeing this same failure in python3.5 on 16.04 about 20% of the time:

```
$ python3.5 -m test -v test_dbm
== CPython 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609]
==   Linux-4.4.0-57-generic-x86_64-with-Ubuntu-16.04-xenial little-endian
==   hash algorithm: siphash24 64bit
==   /tmp/test_python_8169
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
[1/1] test_dbm
test_keys (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb_ndbm (test.test_dbm.WhichDBTestCase) ... BDB0004 fop_read_meta: @test_8169_tmp_ndbm.db: unexpected file type or format
ok
test_anydbm_access (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.gnu) ... ok
test_error (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_8169_tmp.db: unable to flush: No such file or directory
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_8169_tmp.db: unable to flush: No such file or directory
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
test_error (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.dumb) ... ok
test_error (test.test_dbm.TestCase-dbm.dumb) ... ok

======================================================================
ERROR: test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/test/test_dbm.py", line 74, in test_anydbm_creation
    self.read_helper(f)
  File "/usr/lib/python3.5/test/test_dbm.py", line 115, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/test/test_dbm.py", line 89, in test_anydbm_modification
    self.read_helper(f)
  File "/usr/lib/python3.5/test/test_dbm.py", line 115, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/test/test_dbm.py", line 95, in test_anydbm_read
    self.read_helper(f)
  File "/usr/lib/python3.5/test/test_dbm.py", line 115, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

----------------------------------------------------------------------
Ran 27 tests in 0.093s

FAILED (errors=3)
test test_dbm failed
1 test failed:
    test_dbm
```

Under 3.6 (built from source), it seems to fail 100% of the time

```
$ ./python -m test test_dbm -v
== CPython 3.6.0 (default, Jan 9 2017, 15:00:38) [GCC 5.4.0 20160609]
==   Linux-4.4.0-57-generic-x86_64-with-debian-stretch-sid little-endian
==   hash algorithm: siphash24 64bit
==  cwd: /home/asottile/workspace/cpython/build/test_python_8246
==  encodings: locale=UTF-8, FS=utf-8
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 [1/1] test_dbm
test_keys (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb_ndbm (test.test_dbm.WhichDBTestCase) ... BDB0004 fop_read_meta: @test_8246_tmp_ndbm.db: unexpected file type or format
ok
test_anydbm_access (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.gnu) ... ok
test_error (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_8246_tmp.db: unable to flush: No such file or directory
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_8246_tmp.db: unable to flush: No such file or directory
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
test_error (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.dumb) ... ok
test_error (test.test_dbm.TestCase-dbm.dumb) ... ok

======================================================================
ERROR: test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 73, in test_anydbm_creation
    self.read_helper(f)
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 88, in test_anydbm_modification
    self.read_helper(f)
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 94, in test_anydbm_read
    self.read_helper(f)
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

----------------------------------------------------------------------
Ran 27 tests in 0.084s

FAILED (errors=3)
test test_dbm failed
test_dbm failed

1 test failed:
    test_dbm

Total duration: 112 ms
Tests result: FAILURE
```
msg285082 - (view) Author: Anthony Sottile (asottile) * Date: 2017-01-09 23:37
Stepping through the code, it seems under ndbm it is creating a file with a '.db' extension:

```
(Pdb) list
 47  	             'g': b'intended',
 48  	             }
 49  	
 50  	    def init_db(self):
 51  	        import pdb; pdb.set_trace()
 52  ->	        with dbm.open(_fname, 'n') as f:
 53  	            for k in self._dict:
 54  	                f[k.encode("ascii")] = self._dict[k]
 55  	
 56  	    def keys_helper(self, f):
 57  	        keys = sorted(k.decode("ascii") for k in f.keys())
(Pdb) n
> /home/asottile/workspace/cpython/Lib/test/test_dbm.py(53)init_db()
-> for k in self._dict:
(Pdb) os.listdir('.')
['@test_8371_tmp.db']
(Pdb) _fname
'@test_8371_tmp'
```
msg285083 - (view) Author: Anthony Sottile (asottile) * Date: 2017-01-09 23:48
That doesn't seem to be the problem though, that occurs in both the successful and failure case
History
Date User Action Args
2022-04-11 14:57:56adminsetgithub: 64293
2017-01-09 23:48:20asottilesetmessages: + msg285083
2017-01-09 23:37:13asottilesetmessages: + msg285082
2017-01-09 23:29:05asottilesetnosy: + asottile

messages: + msg285081
versions: + Python 3.5, Python 3.6
2014-04-30 18:37:59ethan.furmansetstatus: open -> closed

messages: + msg217639
2014-04-30 17:05:11jceasetnosy: + jcea
messages: + msg217630
2013-12-29 16:48:41ethan.furmansetmessages: + msg207080
2013-12-29 16:46:38ethan.furmancreate