classification
Title: Check against misspellings of assert etc. in mock
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: cjw296, lisroach, mariocj89, michael.foord, rbcollins, rhettinger, terry.reedy, vabr2, veky, xtreak
Priority: normal Keywords:

Created on 2020-09-28 18:19 by vabr2, last changed 2020-10-13 17:10 by vabr2.

Messages (9)
msg377611 - (view) Author: Václav Brožek (vabr2) * Date: 2020-09-28 18:19
Recently we cleaned up the following typos in mocks in unittests of our codebase:

* Wrong prefixes of mock asserts: asert/aseert/assrt -> assert
* Wrong attribute names around asserts: autospect/auto_spec -> autospec, set_spec -> spec_set

Especially the asserts are dangerous, because a misspelled assert_called will fail silently. We found real bugs in production code which were masked by a misspelled assert_called.

There is prior work done to report similar cases of assert misspellings with an AttributeError: https://github.com/testing-cabal/mock/commit/7c530f0d9aa48d2538501761098df7a5a8979a7d, and adding new cases will be an easy change. I suppose that adding similar error signalling for the wrong argument names will not be much harder.

I'm prepared to implement it, if people of this project would be happy to have such checks.
msg377612 - (view) Author: Vedran Čačić (veky) * Date: 2020-09-28 18:41
How about we actually _solve_ the problem, instead of masking it with layer upon layer of obfuscation?

Python has standalone functions. assert_called (and company) should just be functions, they shouldn't be methods, and the problem is solved elegantly. The whole reason the problem exists in the first place is that Java, where Python copied the API from, has no standalone functions. In Pythonic design, the problem shouldn't exist at all.
msg377620 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-09-28 20:12
Václav, I support your suggestion and suggest that you move forward to prepare a PR.

Vedran, this API has been set in stone for a long time, so changing is disruptive.  If you really think the API needs to be redesigned, consider bringing it up on python-ideas.  The proposal would need broad support to move forward.
msg377625 - (view) Author: Vedran Čačić (veky) * Date: 2020-09-28 20:40
It would be a valid argument if the API _worked_. Obviously, it doesn't. Every few years, the same story repeats. "We've found even more missspellings of assert, we need to add them too. They cause real bugs in our tests." I have a strong feeling it will never end.
msg377841 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-10-02 23:43
Vedran, you explained why many use pytest instead of unittest. But we have the latter and a stability policy.

I am not familiar with the existing mock code, but one already invented solution for misspelling tolerance without enumeration is the soundex algorithm.  I have not read the details for over a decade, but I belive soundex(<assert variation>) = 'asrt' for all examples given here.  Perhaps it could be used to broaden the test.
msg378255 - (view) Author: Vedran Čačić (veky) * Date: 2020-10-08 16:19
Sorry, but 

1) Stability policy is great when we hold on to it. If we add new spellings of assert every few years, what kind of stability it is? "My" solution is of the same type: just add one more thing to the API. But that solution is better, because a) it doesn't clash, and b) it must be done only once.

2) Soundex has nothing to do with it. It was invented to mitigate errors when transferring _sounds_ (phonemes) over the telephone wire. It would surprise me very much if those were in any useful sense correlated with errors encountered typing graphemes on the mechanical keyboard.
msg378299 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-10-09 03:12
See also below issues for previous discussion on making the assert helpers as top level functions : 

https://bugs.python.org/issue24651
https://bugs.python.org/issue30949
msg378302 - (view) Author: Vedran Čačić (veky) * Date: 2020-10-09 04:11
Of course, that's why I wrote "my" in quotes above. It's not my solution, it's the idea that many people independently had. Because it is _the_ solution, of course. :-]

I'd just like to point out in the above thread (first link you provided), how _many_ people operate under the assumption that "misspelling problems are now solved". I'm sure many of their opinions would be different if they knew we'd be having this discussion now.

Let's not make the same mistake again. I assure you that in few more years we'll find some other creative ways to misspell arrest_* methods. ;-)
msg378569 - (view) Author: Václav Brožek (vabr2) * Date: 2020-10-13 17:10
Thank you all for the informative replies (and sorry for my long silence, I was sick).

I agree that the general solution (module-level assert) is worth doing, and I just added the current motivation to https://bugs.python.org/issue24651.

I still think that the cost associated with bad misspellings compared to the effort to extend the existing solution (adding patterns to [1]) is strongly in favour of extending the solution: the recent clean-up we had cost us many hours of work and involved several people (especially cases with potential or real bugs being discovered after the fixed typos). Adding a pattern to [1] seems much cheaper than the cost it saves.

The general solution is unlikely to be implemented soon, and even once it is, migrating existing code to use it seems unrealistic from the cost perspective. That's why I think that adding the newly found patterns to [1] makes sense.


[1] https://github.com/python/cpython/blob/master/Lib/unittest/mock.py#L634
History
Date User Action Args
2020-10-13 17:10:31vabr2setmessages: + msg378569
2020-10-09 04:11:48vekysetmessages: + msg378302
2020-10-09 03:12:56xtreaksetnosy: + lisroach, cjw296, xtreak, rbcollins, mariocj89
messages: + msg378299
2020-10-08 16:19:24vekysetmessages: + msg378255
2020-10-02 23:43:15terry.reedysetnosy: + terry.reedy
messages: + msg377841
2020-09-28 20:40:03vekysetmessages: + msg377625
2020-09-28 20:12:58rhettingersetnosy: + rhettinger, michael.foord
messages: + msg377620
2020-09-28 18:41:04vekysetnosy: + veky
messages: + msg377612
2020-09-28 18:19:17vabr2create