classification
Title: webbrowser module import has heavy side effects
Type: performance Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: louielu, ncoghlan, serhiy.storchaka, terry.reedy
Priority: normal Keywords: patch

Created on 2017-02-25 09:26 by serhiy.storchaka, last changed 2017-03-24 22:40 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
webbrowser-delayed-initialization.patch serhiy.storchaka, 2017-02-25 11:48
webbrowser-delayed-initialization-2.patch serhiy.storchaka, 2017-02-25 22:50
Pull Requests
URL Status Linked Edit
PR 484 merged serhiy.storchaka, 2017-03-05 21:38
PR 703 larry, 2017-03-17 21:00
Messages (10)
msg288551 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-02-25 09:26
`import webbrowser` has heavy side effects. It searches a number of executables. On X Window it also runs external program xdg-settings.

Cold start:

$ time ./python -c ''

real	0m1.719s
user	0m0.088s
sys	0m0.036s

$ time ./python -c 'import webbrowser'

real	0m5.713s
user	0m0.308s
sys	0m0.196s

Hot start:

$ time ./python -c ''

real	0m0.094s
user	0m0.072s
sys	0m0.020s

$ time ./python -c 'import webbrowser'

real	0m1.026s
user	0m0.284s
sys	0m0.100s
msg288561 - (view) Author: Louie Lu (louielu) * Date: 2017-02-25 11:38
What is the different of Cold start and Hot start? It that CPU speed or something else.

In Linux 4.9.11 with i7-2k, I can't reproduce the significant real time you gave:

# CPU gov powersave
$ time ./python -c 'import webbrowser'
0.16s user 0.02s system 93% cpu 0.200 total

# CPU gov performance
$ time ./python -c 'import webbrowser'
0.08s user 0.00s system 82% cpu 0.093 total
msg288562 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-02-25 11:47
Following patch makes searching of all platform browsers delayed until it is needed. In addition it makes webbrowser.register() thread safe.

Cold start:

$ time ./python -c 'import webbrowser'

real	0m2.851s
user	0m0.224s
sys	0m0.056s

Hot start:

$ time ./python -c 'import webbrowser'

real	0m0.259s
user	0m0.232s
sys	0m0.024s
msg288563 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-02-25 11:55
> What is the different of Cold start and Hot start?

Disk caches are dropped before cold start.

$ echo 3 | sudo tee /proc/sys/vm/drop_caches
msg288566 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-02-25 14:52
Nice, this is much cleaner than the current approach!

The one thing I would suggest is a new test case that:

- asserts webbrowser._tryorder is None
- asserts webbrowser._browsers is empty
- calls webbrowser.get()
- asserts webbrowser._tryorder is non-empty
- asserts webbrowser._browsers is non-empty

I wouldn't worry about explicitly testing the thread safety. That's just a normal double-checked locking pattern, so I think code review is sufficient to address that - the only way for it to break is for something to go horribly wrong in threading.RLock().
msg288585 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-02-25 22:50
Added tests.
msg288587 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-02-26 03:43
Patch LGTM.

Serhiy, did you want to take this as a chance to run through the new GitHub PR workflow?

Current details are at https://docs.python.org/devguide/pullrequest.html
msg288589 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-02-26 07:14
The new GitHub PR workflow still looks cumbersome and unclear to me. It needs using a lot of git commands different for different branches (and some command are failed or don't work as I expected when I try to repeat sequences from the devguide). How to convert a patch to a pull request? How to get a list of added and modified files? What is the best way to revert all changes and changes in selected files? How to retrieve changes from pull request for local testing? Is there a way to edit otherpeople's pull request (add an entry in Misc/NEWS, etc) before merging it in the main repository? I wait until the devguide be more comprehensive.
msg288930 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 21:16
This will help IDLE startup (though webbrowser should not be imported on windows, where os.startfile is used instead).

This is a somewhat separate issue, but should the Windows code be modified for Win10 and Microsoft Edge?
msg290260 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-24 22:40
New changeset a7cba27aea138311117e2ab1d91584efcfeac4ec by Serhiy Storchaka in branch 'master':
bpo-29645: Speed up importing the webbrowser module. (#484)
https://github.com/python/cpython/commit/a7cba27aea138311117e2ab1d91584efcfeac4ec
History
Date User Action Args
2017-03-24 22:40:47serhiy.storchakasetmessages: + msg290260
2017-03-17 21:00:36larrysetpull_requests: + pull_request626
2017-03-08 15:20:37serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-03-05 21:38:08serhiy.storchakasetpull_requests: + pull_request397
2017-03-03 21:16:16terry.reedysetnosy: + terry.reedy
messages: + msg288930
2017-02-26 07:14:06serhiy.storchakasetmessages: + msg288589
2017-02-26 03:43:29ncoghlansetmessages: + msg288587
2017-02-25 22:50:54serhiy.storchakasetfiles: + webbrowser-delayed-initialization-2.patch

messages: + msg288585
2017-02-25 14:52:10ncoghlansetmessages: + msg288566
2017-02-25 11:55:16serhiy.storchakasetmessages: + msg288563
2017-02-25 11:48:58serhiy.storchakasetfiles: + webbrowser-delayed-initialization.patch
keywords: + patch
2017-02-25 11:47:58serhiy.storchakasetnosy: + ncoghlan

messages: + msg288562
stage: patch review
2017-02-25 11:38:23louielusetnosy: + louielu
messages: + msg288561
2017-02-25 09:26:58serhiy.storchakacreate