classification
Title: optimize v4 & v6 netmask parsing
Type: performance Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: 16531 Superseder:
Assigned To: Nosy List: ncoghlan, pitrou, pmoody, python-dev, serhiy.storchaka
Priority: low Keywords: patch

Created on 2014-05-12 18:24 by pitrou, last changed 2014-05-15 19:06 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
v4_netmask.patch pitrou, 2014-05-12 18:24 review
netmask.patch pitrou, 2014-05-12 18:57 review
Messages (8)
msg218344 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-12 18:24
Here is a patch to optimize ipv4 netmask parsing by maintaining a cache (there are not many valid ipv4 netmask representations). This should be especially useful with #16531.
msg218349 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-12 18:57
Updated patch, also optimizing v6 netmask parsing (same principle).

Before patch:

$ ./python -m timeit -s "import ipaddress" "net = ipaddress.IPv6Network(('2001:db8::', 96))"
10000 loops, best of 3: 26.1 usec per loop

$ ./python -m timeit -s "import ipaddress" "net = ipaddress.IPv4Network(('10.0.0.0', 23))"
100000 loops, best of 3: 17 usec per loop

After patch:

$ ./python -m timeit -s "import ipaddress" "net = ipaddress.IPv6Network(('2001:db8::', 96))"
100000 loops, best of 3: 13.8 usec per loop

$ ./python -m timeit -s "import ipaddress" "net = ipaddress.IPv4Network(('10.0.0.0', 23))"
100000 loops, best of 3: 14.3 usec per loop
msg218481 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-05-13 20:03
Why not just use functools.lru_cache?
msg218482 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-13 20:04
Because that would incur the cost of LRU logic and locking, which we don't need here.
msg218504 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-05-14 06:33
With C implementation (issue14373) functools.lru_cache is so fast as manually written specialized code.
msg218609 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-05-15 12:43
What I want to say, the patch LGTM, but after committing issue14373 we should simplify the code by using functools.lru_cache().
msg218610 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-15 12:49
Actually, using lru_cache(maxsize=None) would enable a simple infinite cache like in the patch. But it's not like a lot of code would be saved.
msg218621 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-15 18:19
New changeset 2158614e1607 by Antoine Pitrou in branch 'default':
Issue #21486: Optimize parsing of netmasks in ipaddress.IPv4Network and ipaddress.IPv6Network.
http://hg.python.org/cpython/rev/2158614e1607
History
Date User Action Args
2014-05-15 19:06:11pitrousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2014-05-15 18:19:51python-devsetnosy: + python-dev
messages: + msg218621
2014-05-15 12:49:10pitrousetmessages: + msg218610
2014-05-15 12:43:42serhiy.storchakasetmessages: + msg218609
2014-05-14 06:33:22serhiy.storchakasetmessages: + msg218504
2014-05-13 20:04:50pitrousetmessages: + msg218482
2014-05-13 20:03:00serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg218481
2014-05-12 18:57:34pitrousetfiles: + netmask.patch

messages: + msg218349
title: optimize v4 netmask parsing -> optimize v4 & v6 netmask parsing
2014-05-12 18:24:56pitrousetdependencies: + Allow IPNetwork to take a tuple
2014-05-12 18:24:45pitroucreate