That's the very weird thing. I've been unable to reproduce this locally but it can be seen on Travis runs.

Here are the results so far (Versions numbers are retrieved with python -VV and python -c "import sys; print(sys._git)"):

    On some versions, both tests pass:
        Python 3.6 and before
        Python 3.7.0a4+ (heads/master:4666ec5, Jan 26 2018, 04:14:24) - [GCC 4.8.4] - ('CPython', 'heads/master', '4666ec5'))

    On some versions, both tests fail:
        Python 3.8.0a1+ (heads/master:8a03ff2, Feb 9 2019, 07:30:26) [GCC 5.4.0 20160609] - ('CPython', 'heads/master', '8a03ff2')

    On some versions, only test_assertRaisesRegex_contextman fails which is what I find confusing:
        Python 3.7.1 (default, Dec 5 2018, 18:09:53) [GCC 5.4.0 20160609] - ('CPython', '', '')
        Python 3.7.2+ (heads/3.7:3fcfef3, Feb 9 2019, 07:30:09) [GCC 5.4.0 20160609] - ('CPython', 'heads/3.7', '3fcfef3')


Corresponding Travis build: https://travis-ci.org/SylvainDe/DidYouMean-Python/builds/491118939 .
Corresponding code on Github: https://github.com/SylvainDe/DidYouMean-Python/blob/issue36_investigation/didyoumean/didyoumean_sugg_tests.py

I'm happy to add more debug information and retrigger builds if needed.