This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: `test_functools` unexpected failures when C `_functoolsmodule` is missing
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.11, Python 3.10
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, rhettinger, shihai1991, sobolevn
Priority: normal Keywords: patch

Created on 2022-02-05 10:46 by sobolevn, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 31141 closed sobolevn, 2022-02-05 11:11
Messages (5)
msg412565 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2022-02-05 10:46
Reproduction steps:

1. Add to `Setup.local`:

```
*disabled*
_functoolsmodule
```

2. `.configure && make -j`. Then, ensure that this module is not available:

```
» ./python.exe -c 'import _functools'                                         
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '_functools'
```

3. Run `test_functools`:

```
======================================================================
ERROR: test_bad_cmp (test.test_functools.TestCmpToKeyC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 905, in test_bad_cmp
    key = self.cmp_to_key(cmp1)
          ^^^^^^^^^^^^^^^^^^^^^
TypeError: cmp_to_key() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_cmp_to_key (test.test_functools.TestCmpToKeyC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 869, in test_cmp_to_key
    key = self.cmp_to_key(cmp1)
          ^^^^^^^^^^^^^^^^^^^^^
TypeError: cmp_to_key() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_cmp_to_key_arguments (test.test_functools.TestCmpToKeyC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 885, in test_cmp_to_key_arguments
    key = self.cmp_to_key(mycmp=cmp1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cmp_to_key() got multiple values for argument 'mycmp'

======================================================================
ERROR: test_hash (test.test_functools.TestCmpToKeyC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 941, in test_hash
    key = self.cmp_to_key(mycmp)
          ^^^^^^^^^^^^^^^^^^^^^^
TypeError: cmp_to_key() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_obj_field (test.test_functools.TestCmpToKeyC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 920, in test_obj_field
    key = self.cmp_to_key(mycmp=cmp1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cmp_to_key() got multiple values for argument 'mycmp'

======================================================================
ERROR: test_sort_int (test.test_functools.TestCmpToKeyC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 926, in test_sort_int
    self.assertEqual(sorted(range(5), key=self.cmp_to_key(mycmp)),
                                          ^^^^^^^^^^^^^^^^^^^^^^
TypeError: cmp_to_key() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_sort_int_str (test.test_functools.TestCmpToKeyC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 934, in test_sort_int_str
    values = sorted(values, key=self.cmp_to_key(mycmp))
                                ^^^^^^^^^^^^^^^^^^^^^^
TypeError: cmp_to_key() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_pickle (test.test_functools.TestPartialC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 258, in test_pickle
    f_copy = pickle.loads(pickle.dumps(f, proto))
                          ^^^^^^^^^^^^^^^^^^^^^^
_pickle.PicklingError: Can't pickle <class 'functools.partial'>: it's not the same object as functools.partial

======================================================================
ERROR: test_recursive_pickle (test.test_functools.TestPartialC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 343, in test_recursive_pickle
    pickle.dumps(f, proto)
    ^^^^^^^^^^^^^^^^^^^^^^
_pickle.PicklingError: Can't pickle <class 'functools.partial'>: it's not the same object as functools.partial

======================================================================
ERROR: test_iterator_usage (test.test_functools.TestReduceC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 843, in test_iterator_usage
    self.assertEqual(self.reduce(add, SequenceClass(5)), 10)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/functools.py", line 249, in reduce
    it = iter(sequence)
         ^^^^^^^^^^^^^^
TypeError: 'builtin_function_or_method' object is not iterable

======================================================================
ERROR: test_reduce (test.test_functools.TestReduceC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 794, in test_reduce
    self.assertEqual(self.reduce(add, ['a', 'b', 'c'], ''), 'abc')
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: reduce() takes from 2 to 3 positional arguments but 4 were given

======================================================================
FAIL: test_disallow_instantiation (test.test_functools.TestCmpToKeyC)
----------------------------------------------------------------------
TypeError: type() takes 1 or 3 arguments

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 955, in test_disallow_instantiation
    support.check_disallow_instantiation(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/test/support/__init__.py", line 2121, in check_disallow_instantiation
    testcase.assertRaisesRegex(TypeError, msg, tp, *args, **kwds)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "cannot create 'type' instances" does not match "type() takes 1 or 3 arguments"

======================================================================
FAIL: test_attributes_unwritable (test.test_functools.TestPartialC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 402, in test_attributes_unwritable
    self.assertRaises(AttributeError, setattr, p, 'func', map)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: AttributeError not raised by setattr

======================================================================
FAIL: test_attributes_unwritable (test.test_functools.TestPartialCSubclass)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_functools.py", line 402, in test_attributes_unwritable
    self.assertRaises(AttributeError, setattr, p, 'func', map)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: AttributeError not raised by setattr

----------------------------------------------------------------------
Ran 249 tests in 0.690s

FAILED (failures=3, errors=11)
test test_functools failed
test_functools failed (11 errors, 3 failures)

== Tests result: FAILURE ==

1 test failed:
    test_functools

Total duration: 1.3 sec
Tests result: FAILURE
```

List of individual problems:

1. This function is defined assuming that `c_functools` always has `.lru_cache`: https://github.com/python/cpython/blob/fea7290a0ecee09bbce571d4d10f5881b7ea3485/Lib/test/test_functools.py#L1860-L1862
2. `TestLRUC` is never skipped: https://github.com/python/cpython/blob/fea7290a0ecee09bbce571d4d10f5881b7ea3485/Lib/test/test_functools.py#L1879-L1881 I think it should be, because there's no need to test `_lru_cache_wrapper` twice for just python implementation (default if `_functools` is missing)
3. All similar modules tend to use `fresh=` in `import_fresh_module`, for example: https://github.com/python/cpython/blob/fea7290a0ecee09bbce571d4d10f5881b7ea3485/Lib/test/test_typing.py#L43-L44 But, `test_functools` does not do this: https://github.com/python/cpython/blob/fea7290a0ecee09bbce571d4d10f5881b7ea3485/Lib/test/test_functools.py#L30 So, even if `_functools` is missing, `c_functools` will not be `None`, it will still be `functools.py` module! And this causes multiple unexpected test failures above


Related:
- https://github.com/python/cpython/pull/23405
- https://github.com/python/cpython/pull/23407

I will send a patch for this in a moment.
msg412572 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-02-05 13:39
_functoolsmodule is a core bootstrap module and defined in Modules/Setup.bootstrap. It has no external dependencies. There is no reason and no point to disable the module.

We can safely assume that all modules defined in Modules/Setup.bootstrap are always available. (Maybe except for pwd, but that is a different story.)

I propose to close the bug and PR as wontfix.
msg412573 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2022-02-05 13:44
Cristian, in this case - is there a reason to keep `skipUnless(c_functools)` around? 

If we are sure that it is always available - then it should be always tested. 

We either should have:
1. Cleanly defined skips that work (this PR)
2. Unconditional coverage

Or do you think that some middle-ground is possible?
msg412767 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2022-02-07 16:44
> _functoolsmodule is a core bootstrap module and defined in Modules/Setup.bootstrap. It has no external dependencies. There is no reason and no point to disable the module.

+1. 

> Cristian, in this case - is there a reason to keep `skipUnless(c_functools)` around? 
> If we are sure that it is always available - then it should be always tested. 

Hm. Personally, I suggest to keep the `skipUnless` to aovid the code churn.
Or maybe you have other cases to show the functools module will missing unexpectly?
msg412809 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2022-02-08 04:43
> Or maybe you have other cases to show the functools module will missing unexpectly?

No, I can't think of any :)

Your argument about code churn also makes sense.
But, if we ever are going to refactor this test module, this is something to remember of.

Thanks everyone!
History
Date User Action Args
2022-04-11 14:59:55adminsetgithub: 90805
2022-02-08 04:43:59sobolevnsetstatus: open -> closed
resolution: wont fix
messages: + msg412809

stage: patch review -> resolved
2022-02-07 16:44:19shihai1991setmessages: + msg412767
2022-02-05 13:44:16sobolevnsetmessages: + msg412573
2022-02-05 13:39:14christian.heimessetnosy: + christian.heimes
messages: + msg412572
2022-02-05 11:11:30sobolevnsetkeywords: + patch
stage: patch review
pull_requests: + pull_request29318
2022-02-05 10:46:54sobolevncreate