classification
Title: Behavior for unittest.assertRaisesRegex differs depending on whether it is used as a context manager
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: SylvainDe, nedbat, ronaldoussoren, xtreak
Priority: normal Keywords:

Created on 2019-02-11 13:04 by SylvainDe, last changed 2019-02-11 17:51 by xtreak. This issue is now closed.

Messages (7)
msg335212 - (view) Author: SylvainDe (SylvainDe) * Date: 2019-02-11 13:04
On some Python versions, the following pieces of code have a different behavior which is not something I'd expect:

```

    DESCRIPT_REQUIRES_TYPE_RE = r"descriptor '\w+' requires a 'set' object but received a 'int'"

    ...

    def test_assertRaisesRegex(self):
        self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE, set.add, 0)

    def test_assertRaisesRegex_contextman(self):
        with self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE):
            set.add(0)

```

On impacted Python versions, only test_assertRaisesRegex_contextman fails while test_assertRaisesRegex works fine.

Logs for the failure:


```

======================================================================
FAIL: test_assertRaisesRegex_contextman (didyoumean_sugg_tests.SetAddIntRegexpTests)
----------------------------------------------------------------------
TypeError: descriptor 'add' for 'set' objects doesn't apply to 'int' object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/travis/build/.../didyoumean/didyoumean_sugg_tests.py", line 23, in test_assertRaisesRegex_contextman
    set.add(0)
AssertionError: "descriptor '\w+' requires a 'set' object but received a 'int'" does not match "descriptor 'add' for 'set' objects doesn't apply to 'int' object"


```


Either I am missing something or it looks like a bug to me.

If needed, more details/context can be found on the StackOverflow question I opened: https://stackoverflow.com/questions/54612348/different-error-message-when-unittest-assertraisesregex-is-called-as-a-context-m .
I can provide the details directly here if it is relevant.
msg335213 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2019-02-11 13:11
On which Python versions do you see this problem? Both tests pass on my system (macOS 10.4.3) with Python 3.7.2, 3.6.3 and 3.5.5.
msg335214 - (view) Author: SylvainDe (SylvainDe) * Date: 2019-02-11 13:15
That's the very weird thing. I've been unable to reproduce this locally but it can be seen on Travis runs.

Here are the results so far (Versions numbers are retrieved with python -VV and python -c "import sys; print(sys._git)"):

    On some versions, both tests pass:
        Python 3.6 and before
        Python 3.7.0a4+ (heads/master:4666ec5, Jan 26 2018, 04:14:24) - [GCC 4.8.4] - ('CPython', 'heads/master', '4666ec5'))

    On some versions, both tests fail:
        Python 3.8.0a1+ (heads/master:8a03ff2, Feb 9 2019, 07:30:26) [GCC 5.4.0 20160609] - ('CPython', 'heads/master', '8a03ff2')

    On some versions, only test_assertRaisesRegex_contextman fails which is what I find confusing:
        Python 3.7.1 (default, Dec 5 2018, 18:09:53) [GCC 5.4.0 20160609] - ('CPython', '', '')
        Python 3.7.2+ (heads/3.7:3fcfef3, Feb 9 2019, 07:30:09) [GCC 5.4.0 20160609] - ('CPython', 'heads/3.7', '3fcfef3')


Corresponding Travis build: https://travis-ci.org/SylvainDe/DidYouMean-Python/builds/491118939 .
Corresponding code on Github: https://github.com/SylvainDe/DidYouMean-Python/blob/issue36_investigation/didyoumean/didyoumean_sugg_tests.py

I'm happy to add more debug information and retrigger builds if needed.
msg335233 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-02-11 16:06
Is this something related to running under coverage? With the failure commit on Travis I built the binary and there are no failures with unittest but the tests fail with coverage. Running 'set.add(0)' under coverage and as normal file generates different error messages. Can you try running with unittest to see if there are any discrepancies? I am adding @nedbat who might have a better explanation.

# Python version

(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ python
Python 3.8.0a1+ (tags/v3.8.0a1-23-g8a03ff2ff4:8a03ff2ff4, Feb 11 2019, 15:51:54)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

# set.add(0) under coverage and as a file

(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ echo "set.add(0)" > foo.py
(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ coverage run foo.py
Traceback (most recent call last):
  File "foo.py", line 1, in <module>
    set.add(0)
TypeError: descriptor 'add' for 'set' objects doesn't apply to 'int' object
(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ python foo.py
Traceback (most recent call last):
  File "foo.py", line 1, in <module>
    set.add(0)
TypeError: descriptor 'add' requires a 'set' object but received a 'int'

# Running the reported case will fail while running under coverage

(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ python -m unittest discover --start-directory=/tmp/suggest --pattern=*.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK
(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ coverage run -m unittest discover --start-directory=/tmp/suggest --pattern=*.py
FF
======================================================================
FAIL: test_assertRaisesRegex (foo.FooTest)
----------------------------------------------------------------------
TypeError: descriptor 'add' for 'set' objects doesn't apply to 'int' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/suggest/foo.py", line 8, in test_assertRaisesRegex
    self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE, set.add, 0)
AssertionError: "descriptor '\w+' requires a 'set' object but received a 'int'" does not match "descriptor 'add' for 'set' objects doesn't apply to 'int' object"

======================================================================
FAIL: test_assertRaisesRegex_contextman (foo.FooTest)
----------------------------------------------------------------------
TypeError: descriptor 'add' for 'set' objects doesn't apply to 'int' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/suggest/foo.py", line 12, in test_assertRaisesRegex_contextman
    set.add(0)
AssertionError: "descriptor '\w+' requires a 'set' object but received a 'int'" does not match "descriptor 'add' for 'set' objects doesn't apply to 'int' object"

----------------------------------------------------------------------
Ran 2 tests in 0.003s

FAILED (failures=2)


# Executed test

(foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ cat /tmp/suggest/foo.py
import unittest

DESCRIPT_REQUIRES_TYPE_RE = r"descriptor '\w+' requires a 'set' object but received a 'int'"

class FooTest(unittest.TestCase):

    def test_assertRaisesRegex(self):
        self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE, set.add, 0)

    def test_assertRaisesRegex_contextman(self):
        with self.assertRaisesRegex(TypeError, DESCRIPT_REQUIRES_TYPE_RE):
            set.add(0)

if __name__ == "__main__":
    unittest.main()
msg335237 - (view) Author: SylvainDe (SylvainDe) * Date: 2019-02-11 16:19
This is a brilliant idea! I'll give it a try and keep you posted.
msg335239 - (view) Author: SylvainDe (SylvainDe) * Date: 2019-02-11 17:00
I have no explanation whatsoever but removing the call to coverage seems to remove the issue: https://travis-ci.org/SylvainDe/DidYouMean-Python/builds/491727724 .

I guess we can close this issue - sorry for the inconvenience.

@nedbat: do you want me to open an issue in coverage ?
msg335243 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-02-11 17:51
No problem, after some debugging the statement finally executed is 'exec(compile("set.add(0)", "foo.py", "exec"))' . In Python interpreter this will give the expected error. It also gives the same error before coverage.start() [0] and gives a different one as in the report once the statement is executed after coverage.start(). So this might be something to do with coverage and I have less knowledge about the internals.

I am closing this as third party since I couldn't see any issue with CPython and you might want to follow up coverage issue tracker. Thanks for the report though!

[0] https://github.com/nedbat/coveragepy/blob/f12db7fd4b3ab288f7f770f7138062951feaa6c9/coverage/cmdline.py#L639
History
Date User Action Args
2019-02-11 17:51:18xtreaksetstatus: open -> closed
resolution: third party
messages: + msg335243

stage: resolved
2019-02-11 17:00:56SylvainDesetmessages: + msg335239
2019-02-11 16:19:20SylvainDesetmessages: + msg335237
2019-02-11 16:06:24xtreaksetnosy: + nedbat, xtreak
messages: + msg335233
2019-02-11 13:15:08SylvainDesetmessages: + msg335214
2019-02-11 13:11:58ronaldoussorensetnosy: + ronaldoussoren
messages: + msg335213
2019-02-11 13:04:40SylvainDecreate