This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: regrtest fails to start on missing performance counter names
Type: Stage: resolved
Components: Tests, Windows Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, jkloth, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2022-02-18 16:36 by jkloth, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (9)
msg413493 - (view) Author: Jeremy Kloth (jkloth) * Date: 2022-02-18 16:36
When attempting to run the test harness, I receive the following:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Public\Devel\cpython\main\Lib\test\__main__.py", line 2, in <module>
    main()
    ^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\main.py", line 736, in main
    Regrtest().main(tests=tests, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\support\os_helper.py", line 396, in temp_dir
    yield path
    ^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\support\os_helper.py", line 427, in change_cwd
    yield os.getcwd()
    ^^^^^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\support\os_helper.py", line 449, in temp_cwd
    yield cwd_dir
    ^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\main.py", line 658, in main
    self._main(tests, kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\main.py", line 704, in _main
    self.win_load_tracker = WindowsLoadTracker()
                            ^^^^^^^^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\win_utils.py", line 41, in __init__
    self.start()
    ^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\win_utils.py", line 70, in start
    counter_name = self._get_counter_name()
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\win_utils.py", line 90, in _get_counter_name
    system = counters_dict['2']
             ~~~~~~~~~~~~~^^^^^
KeyError: '2'

This is due to my machine missing the localized names for the performance counters.  Other performance monitoring tools operate just fine.

While I have been working around this issue for some time, it has become difficult to seperate the workarounds from actually changes in the test harness.

The PR (https://github.com/python/cpython/pull/26578) from https://bugs.python.org/issue44336 also solves this issue by accessing the counters directly instead of relying on their localized names.
msg415696 - (view) Author: Jeremy Kloth (jkloth) * Date: 2022-03-21 18:59
OK, I know it has been a busy month for Python, but this issue is really hampering my bug fixing efforts.  It makes the complete regrtest useless for me.  I am required to run each affected test directly so it is possible to miss side-effects of changes made.

The original PR is now 9 months old.
msg415713 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-03-21 21:42
You're closer to this stuff than I am, so if you say it works for your scenarios and it doesn't break the existing ones, I'll merge it.

Do we need to backport to 3.8 at this point?
msg415719 - (view) Author: Jeremy Kloth (jkloth) * Date: 2022-03-21 22:56
With 3.8 so close to security only, I would doubt it is worth it anymore.  I've just run the PR against HEAD and it still works as is, so should be good to go.
msg415726 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-03-21 23:34
"So close" being some months ago ;)

I need to retrigger address sanitiser so that it passes before merging, but after that, I'll merge (and hopefully the backport is clean).

Copying Eryk Sun's comment here for posterity:

This would be less fragile and more useful in general if support were added to _winapi for PdhOpenQueryW(), PdhAddEnglishCounterW(), PdhCollectQueryData(), PdhCollectQueryDataEx(), PdhGetFormattedCounterValue(), and PdhCloseQuery(). The English counter name is "\System\Processor Queue Length" -- no need to worry about localized names, hard-coded values, or hard-coded struct sizes and offsets.
msg415727 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2022-03-21 23:50
I was just wondering whether it's worth implementing it using the API. To be clear, I wasn't implying to hold off on applying Jeremy's PR. The existing code is causing him problems, and he has a working solution.
msg415731 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-03-22 00:08
Jeremy - backports need to be done manually. Can I leave that with you and just ping me on here when you're ready for a merge?
msg415775 - (view) Author: Jeremy Kloth (jkloth) * Date: 2022-03-22 14:39
Backports state that they are ready...  I'm just a little uneasy as I've never used cherry_picker before.  3.10 went smooth, but 3.9 required manual merging.
msg415799 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-03-22 17:22
Backports have been merged
History
Date User Action Args
2022-04-11 14:59:56adminsetgithub: 90944
2022-03-22 19:51:55eryksunsetfiles: - loadtracker.py
2022-03-22 19:51:47eryksunsetmessages: - msg415781
2022-03-22 17:22:32steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg415799

stage: backport needed -> resolved
2022-03-22 14:55:59eryksunsetfiles: + loadtracker.py

messages: + msg415781
2022-03-22 14:39:18jklothsetmessages: + msg415775
2022-03-22 00:08:13steve.dowersetmessages: + msg415731
stage: backport needed
2022-03-21 23:50:54eryksunsetnosy: + eryksun
messages: + msg415727
2022-03-21 23:34:53steve.dowersetmessages: + msg415726
2022-03-21 22:56:02jklothsetmessages: + msg415719
versions: - Python 3.8
2022-03-21 21:42:04steve.dowersetmessages: + msg415713
2022-03-21 18:59:24jklothsetmessages: + msg415696
2022-02-18 16:36:49jklothsetnosy: + vstinner
2022-02-18 16:36:09jklothcreate