This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Mutation tests results of typing.py
Type: behavior Stage:
Components: Tests Versions: Python 3.11
process
Status: open Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: arhadthedev, serhiy.storchaka, sobolevn
Priority: normal Keywords:

Created on 2021-10-29 16:52 by sobolevn, last changed 2022-04-11 14:59 by admin.

Messages (6)
msg405319 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2021-10-29 16:52
I've decided to test `typing.py` with `cosmic-ray` mutation testing framework. It identified 3 potential problems.

Config:

```
[cosmic-ray]
module-path = "Lib/typing.py"
timeout = 15.0
excluded-modules = []
test-command = "./python.exe -m test -v test_typing --failfast"

[cosmic-ray.distributor]
name = "local"
```

Repro steps:
0. pip install cosmic-ray
1. Copy config above as `typing.toml`
2. cosmic-ray init typing.toml typing.sqlite
3. cosmic-ray exec tutorial.toml tutorial.sqlite 

Survived mutants:

1. ```
--- mutation diff ---
--- aLib/typing.py
+++ bLib/typing.py
@@ -1103,7 +1103,7 @@
             if Protocol in bases:
                 return ()
             i = bases.index(self)
-            for b in bases[i+1:]:
+            for b in bases[i*1:]:
                 if isinstance(b, _BaseGenericAlias) and b is not self:
                     return ()
         return (self.__origin__,)
```

2. ```
--- mutation diff ---
--- aLib/typing.py
+++ bLib/typing.py
@@ -1103,7 +1103,7 @@
             if Protocol in bases:
                 return ()
             i = bases.index(self)
-            for b in bases[i+1:]:
+            for b in bases[i//1:]:
                 if isinstance(b, _BaseGenericAlias) and b is not self:
                     return ()
         return (self.__origin__,)
```

3. ```
--- mutation diff ---
--- aLib/typing.py
+++ bLib/typing.py
@@ -1103,7 +1103,7 @@
             if Protocol in bases:
                 return ()
             i = bases.index(self)
-            for b in bases[i+1:]:
+            for b in bases[i**1:]:
                 if isinstance(b, _BaseGenericAlias) and b is not self:
                     return ()
         return (self.__origin__,)
```

I've attached the full report to this issue if it is interesting to anyone: https://gist.github.com/sobolevn/79f1729dcf0b8c4b2a9bcee67027673a
msg405321 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2021-10-29 16:55
I can send a PR to have 100% mutation coverage for `typing.py`.
I think that a single test can do that.
msg405629 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-11-03 16:28
What is the problem actually?
msg405672 - (view) Author: Oleg Iarygin (arhadthedev) * Date: 2021-11-04 10:33
> What is the problem actually?

I guess, Lib/test/test_typing.py has gaps in test coverage. The report provided by the OP is a list of random modifications that corrupt logic of Lib/typing.py but still pass all test cases.

Mutation testing is validation of tests themselves. A mutation framework takes code that the validated test actually tests, makes some random but syntactically correct change, and runs the test that should fail some assertion here. If so, the code is rolled back and another mutation attempt is done. If the validated test catches all thousands of such attempts, we can consider it a reliable safety net for any refactoring. Otherwise, we open the report and start to supplement tests and probably package documentation to cover missed scenarios.

Actually, something like cosmic-ray may be added into a CI pipeline, but I suspect that preliminary fixing of test coverage will be a task of the OP themselves. There are no people with enough free time to sit down and sort all tests out, being overwhelmed with new language features, bug fixing and already pending epics of, for example, closing Python internals from extension modules to stop constantly breaking backward compatibility, or fixing startup modules to properly freeze them for speedup of Python launching.
msg405676 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-11-04 10:50
The tests are passed because this modification does not affect behavior, it just makes the code slightly less efficient. Replacing i+1 with i just adds one iteration:

    b = bases[i]  # == self
    if isinstance(b, _BaseGenericAlias) and b is not self:
        return ()

Since b is self, the condition is always false.

It is impossible to catch this change in tests because both codes are equivalent.
msg405682 - (view) Author: Oleg Iarygin (arhadthedev) * Date: 2021-11-04 11:53
> because this modification does not affect behavior

Unfortunately, this one (that I missed by not reading the report thoroughly) makes the framework totally unsuitable for CI.

However, looking at such false positives allows to muse about reasons behind design choises. For example (in a snippet you cited), why Python runtime stores bases as a tuple, not as a linked list. Anyway, application of the method resolution order scans them sequentially, they are not of a primitive type (so tuples give no optimization), a call of bases.index() may be replaced with a reference property inside class internals, etc.

Though, I perfectly understand that such treatment leads to immense amount of refactoring with no performance gain, probably counterweighted with removal of couple of internal packages that will become unused after it. That's why I wrote about "a task of the OP themselves".
History
Date User Action Args
2022-04-11 14:59:51adminsetgithub: 89835
2021-11-04 11:53:01arhadthedevsetmessages: + msg405682
2021-11-04 10:50:16serhiy.storchakasetresolution: not a bug
messages: + msg405676
2021-11-04 10:33:32arhadthedevsetnosy: + arhadthedev
messages: + msg405672
2021-11-03 16:28:08serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg405629
2021-10-29 16:55:10sobolevnsetmessages: + msg405321
2021-10-29 16:52:34sobolevncreate