This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: .python_history file causes considerable slowdown
Type: performance Stage:
Components: Build Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: bytecookie, steven.daprano
Priority: normal Keywords:

Created on 2020-08-16 11:42 by bytecookie, last changed 2022-04-11 14:59 by admin.

Messages (5)
msg375505 - (view) Author: bytecookie (bytecookie) Date: 2020-08-16 11:42
The seemingly unlimited growth of the ".python_history"-file seems to cause a massive slowdown when starting or exiting python, the same after running commands.
On one machine python took about 3 minutes to start and I found out that while it is seemingly doing nothing, it was constantly aceesing the ".python_history"-file, which was 130 Mb large. After deleting the file, anything was back to normal.
msg375506 - (view) Author: bytecookie (bytecookie) Date: 2020-08-16 11:50
Using Python 3.8.3 on Windows 10 1803
msg375538 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2020-08-17 10:12
How very odd. I use the Python interactive interpreter extensively, and have done so for years. My history file is only 500 lines.

Did you happen to inspect the file before deleting it to see if it contained something odd?

What does this print for you?

    import readline
    readline.get_history_length()
msg375539 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2020-08-17 10:26
> My history file is only 500 lines.

*slaps forehead*

Of course it is, I'm running a customer history hook that has a limit of 500 lines in the history file.

It looks to me that by default the history feature is set to unlimited lines, so I guess that implies that this isn't a bug and it is your responsibility to set a maximum history length.

You can put these two lines in your Python startup file:

    import readline
    readline.set_history_length(1000)  # or any number you like



Personally, I don't think that having a default setting that allows the history file to grow to 130MB is a good idea.
msg375540 - (view) Author: bytecookie (bytecookie) Date: 2020-08-17 10:33
> it is your responsibility to set a maximum history length.

No, sorry. The problem is not the history file, is the massive slow down it causes. And if you have to utilize a process monitoring tool to find out that the history file is the cause, its not a matter of responsibility .
You can only be responsible for something you know of.

Still I wonder why this isn't an already widely known problem, or is this a new feature? (I am an seasoned developer but very new to python)
History
Date User Action Args
2022-04-11 14:59:34adminsetgithub: 85735
2020-08-17 10:33:46bytecookiesetmessages: + msg375540
2020-08-17 10:26:27steven.dapranosetmessages: + msg375539
2020-08-17 10:12:03steven.dapranosetnosy: + steven.daprano
messages: + msg375538
2020-08-16 11:50:33bytecookiesetmessages: + msg375506
2020-08-16 11:42:32bytecookiecreate