On some Python versions, the following pieces of code have a different behavior which is not something I'd expect:

```

    DESCRIPT_REQUIRES_TYPE_RE = r"descriptor '\w+' requires a 'set' object but received a 'int'"

    ...

    def test_assertRaisesRegex(self):
        self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE, set.add, 0)

    def test_assertRaisesRegex_contextman(self):
        with self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE):
            set.add(0)

```

On impacted Python versions, only test_assertRaisesRegex_contextman fails while test_assertRaisesRegex works fine.

Logs for the failure:


```

======================================================================
FAIL: test_assertRaisesRegex_contextman (didyoumean_sugg_tests.SetAddIntRegexpTests)
----------------------------------------------------------------------
TypeError: descriptor 'add' for 'set' objects doesn't apply to 'int' object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/travis/build/.../didyoumean/didyoumean_sugg_tests.py", line 23, in test_assertRaisesRegex_contextman
    set.add(0)
AssertionError: "descriptor '\w+' requires a 'set' object but received a 'int'" does not match "descriptor 'add' for 'set' objects doesn't apply to 'int' object"


```


Either I am missing something or it looks like a bug to me.

If needed, more details/context can be found on the StackOverflow question I opened: https://stackoverflow.com/questions/54612348/different-error-message-when-unittest-assertraisesregex-is-called-as-a-context-m .
I can provide the details directly here if it is relevant.

On which Python versions do you see this problem? Both tests pass on my system (macOS 10.4.3) with Python 3.7.2, 3.6.3 and 3.5.5.

That's the very weird thing. I've been unable to reproduce this locally but it can be seen on Travis runs.

Here are the results so far (Versions numbers are retrieved with python -VV and python -c "import sys; print(sys._git)"):

    On some versions, both tests pass:
        Python 3.6 and before
        Python 3.7.0a4+ (heads/master:4666ec5, Jan 26 2018, 04:14:24) - [GCC 4.8.4] - ('CPython', 'heads/master', '4666ec5'))

    On some versions, both tests fail:
        Python 3.8.0a1+ (heads/master:8a03ff2, Feb 9 2019, 07:30:26) [GCC 5.4.0 20160609] - ('CPython', 'heads/master', '8a03ff2')

    On some versions, only test_assertRaisesRegex_contextman fails which is what I find confusing:
        Python 3.7.1 (default, Dec 5 2018, 18:09:53) [GCC 5.4.0 20160609] - ('CPython', '', '')
        Python 3.7.2+ (heads/3.7:3fcfef3, Feb 9 2019, 07:30:09) [GCC 5.4.0 20160609] - ('CPython', 'heads/3.7', '3fcfef3')


Corresponding Travis build: https://travis-ci.org/SylvainDe/DidYouMean-Python/builds/491118939 .
Corresponding code on Github: https://github.com/SylvainDe/DidYouMean-Python/blob/issue36_investigation/didyoumean/didyoumean_sugg_tests.py

I'm happy to add more debug information and retrigger builds if needed.

Is this something related to running under coverage? With the failure commit on Travis I built the binary and there are no failures with unittest but the tests fail with coverage. Running 'set.add(0)' under coverage and as normal file generates different error messages. Can you try running with unittest to see if there are any discrepancies? I am adding @nedbat who might have a better explanation.

# Python version

(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ python
Python 3.8.0a1+ (tags/v3.8.0a1-23-g8a03ff2ff4:8a03ff2ff4, Feb 11 2019, 15:51:54)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

# set.add(0) under coverage and as a file

(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ echo "set.add(0)" > foo.py
(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ coverage run foo.py
Traceback (most recent call last):
  File "foo.py", line 1, in <module>
    set.add(0)
TypeError: descriptor 'add' for 'set' objects doesn't apply to 'int' object
(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ python foo.py
Traceback (most recent call last):
  File "foo.py", line 1, in <module>
    set.add(0)
TypeError: descriptor 'add' requires a 'set' object but received a 'int'

# Running the reported case will fail while running under coverage

(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ python -m unittest discover --start-directory=/tmp/suggest --pattern=*.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK
(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ coverage run -m unittest discover --start-directory=/tmp/suggest --pattern=*.py
FF
======================================================================
FAIL: test_assertRaisesRegex (foo.FooTest)
----------------------------------------------------------------------
TypeError: descriptor 'add' for 'set' objects doesn't apply to 'int' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/suggest/foo.py", line 8, in test_assertRaisesRegex
    self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE, set.add, 0)
AssertionError: "descriptor '\w+' requires a 'set' object but received a 'int'" does not match "descriptor 'add' for 'set' objects doesn't apply to 'int' object"

======================================================================
FAIL: test_assertRaisesRegex_contextman (foo.FooTest)
----------------------------------------------------------------------
TypeError: descriptor 'add' for 'set' objects doesn't apply to 'int' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/suggest/foo.py", line 12, in test_assertRaisesRegex_contextman
    set.add(0)
AssertionError: "descriptor '\w+' requires a 'set' object but received a 'int'" does not match "descriptor 'add' for 'set' objects doesn't apply to 'int' object"

----------------------------------------------------------------------
Ran 2 tests in 0.003s

FAILED (failures=2)


# Executed test

(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ cat /tmp/suggest/foo.py
import unittest

DESCRIPT_REQUIRES_TYPE_RE = r"descriptor '\w+' requires a 'set' object but received a 'int'"

class FooTest(unittest.TestCase):

    def test_assertRaisesRegex(self):
        self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE, set.add, 0)

    def test_assertRaisesRegex_contextman(self):
        with self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE):
            set.add(0)

if __name__ == "__main__":
    unittest.main()

This is a brilliant idea! I'll give it a try and keep you posted.

I have no explanation whatsoever but removing the call to coverage seems to remove the issue: https://travis-ci.org/SylvainDe/DidYouMean-Python/builds/491727724 .

I guess we can close this issue - sorry for the inconvenience.

@nedbat: do you want me to open an issue in coverage ?

No problem, after some debugging the statement finally executed is 'exec(compile("set.add(0)", "foo.py", "exec"))' . In Python interpreter this will give the expected error. It also gives the same error before coverage.start() [0] and gives a different one as in the report once the statement is executed after coverage.start(). So this might be something to do with coverage and I have less knowledge about the internals.

I am closing this as third party since I couldn't see any issue with CPython and you might want to follow up coverage issue tracker. Thanks for the report though!

[0] https://github.com/nedbat/coveragepy/blob/f12db7fd4b3ab288f7f770f7138062951feaa6c9/coverage/cmdline.py#L639