This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Improving _removeHandlerRef for a very long _handlerList
Type: performance Stage: resolved
Components: Library (Lib) Versions: Python 3.11
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Yonatan Goldschmidt, miss-islington, vinay.sajip
Priority: normal Keywords: patch

Created on 2021-05-24 01:12 by Yonatan Goldschmidt, last changed 2022-04-11 14:59 by admin. This issue is now closed.

File name Uploaded Description Edit Yonatan Goldschmidt, 2021-05-24 01:12
Pull Requests
URL Status Linked Edit
PR 26325 merged Yonatan Goldschmidt, 2021-05-24 01:20
Messages (4)
msg394229 - (view) Author: Yonatan Goldschmidt (Yonatan Goldschmidt) * Date: 2021-05-24 01:12
We have an application which creates 10,000s of logging.Logger & logging.Handler objects. I encountered a major slowdown during GCs which happen to collect dead Handler objects, calling logging._removeHandlerRef() along the way.
That function looks like:

    def _removeHandlerRef(wr):
        acquire, release, handlers = _acquireLock, _releaseLock, _handlerList
        if acquire and release and handlers:
                if wr in handlers:

and is called by a weakref, attached to each Handler created in _addHandlerRef().

The slowdown occurs in the "in" operator, and the remove() call. These operations on very long lists are expensive. My suggestion is that, without modifying the observed behavior of this function, it can be changed to perform only one lookup in the list, by simply calling remove() and ignoring the ValueError if it gets raised.

Attached is a small script which demonstrates the issue. It creates N1 Handler objects, then deletes the strong reference to the last N2 handler objects, and triggers a GC, measuring the time it takes. Additionally, we can measure its entire execution (since all remaining Handler objects are cleaned up on exit, and eventually _removeHandlerRef() is called for each, and it drastically slows down the interpreter shutdown).

Running the demo with N1=30k and N2=2k:

    8aafe9d1308b">root@8aafe9d1308b:/# python -V
    Python 3.9.2
    8aafe9d1308b">root@8aafe9d1308b:/# time python 30000 2000
    gc.collect() took 1.552838139992673s

    real    0m12.626s
    user    0m12.618s
    sys 0m0.008s

then applying the improving patch...

    8aafe9d1308b">root@8aafe9d1308b:/# (cd /usr/local/lib/python3.9/ && patch -p2 < /patch )
    patching file logging/
    Hunk #1 succeeded at 826 (offset -19 lines).

and trying again:

    8aafe9d1308b">root@8aafe9d1308b:/# time python 30000 2000
    gc.collect() took 0.8691686679958366s

    real    0m7.134s
    user    0m7.130s
    sys 0m0.004s

Almost a 2x speedup.

I also tried removing the acquire/release locks (now that the remove operation is atomic...). I'm not sure it maintains the semantics, and it didn't make too much of a positive effect on the run-time anyway, so I didn't continue looking into it.
msg394246 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2021-05-24 10:46
Seems a bit of a pathological case - out of interest, why do you need to create tens of thousands of loggers and handlers? Each logger is supposed to represent an area of the application. If you want to track e.g. client IP requests or similar, there are better ways. Ditto, each handler is supposed to represent a different audience for logs. Does your application have tens of thousands of distinct modules and/or audiences for logs?
msg394271 - (view) Author: Yonatan Goldschmidt (Yonatan Goldschmidt) * Date: 2021-05-24 22:41
That system has many (>10k) objects of a certain type. Each of them has a separate Logger object + Handlers. That was done, among other reasons, to allow for easy consuming of the logs relating to a particular object in a file-based manner.

I agree that it's not common at all (other Python projects I've worked on don't exhibit this behavior) but since this change is harmless I thought we might as well make it.
msg394406 - (view) Author: miss-islington (miss-islington) Date: 2021-05-25 22:40
New changeset 156699bca02dd2def844d03e26fc16a831336635 by Yonatan Goldschmidt in branch 'main':
bpo-44222: Improve _removeHandlerRef() for a very long _handlerList (GH-26325)
Date User Action Args
2022-04-11 14:59:46adminsetgithub: 88388
2021-05-25 22:57:57Yonatan Goldschmidtsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-05-25 22:40:31miss-islingtonsetnosy: + miss-islington
messages: + msg394406
2021-05-24 22:41:04Yonatan Goldschmidtsetmessages: + msg394271
2021-05-24 10:46:42vinay.sajipsetnosy: + vinay.sajip
messages: + msg394246
2021-05-24 01:20:40Yonatan Goldschmidtsetkeywords: + patch
stage: patch review
pull_requests: + pull_request24918
2021-05-24 01:12:25Yonatan Goldschmidtcreate