classification
Title: Limiting data copy in xmlrpclib
Type: resource usage Stage: test needed
Components: Library (Lib) Versions: Python 3.4
process
Status: pending Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Thomas Fenzl, kilobug, loewis, pitrou, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2007-07-04 09:54 by kilobug, last changed 2016-01-20 09:57 by serhiy.storchaka.

Files
File name Uploaded Description Edit
xmlrpclib.patch kilobug, 2007-07-04 09:54
xmlrpc_less_copy.diff Thomas Fenzl, 2013-03-21 01:43 review
xmlrpc_less_copy-1.diff Thomas Fenzl, 2013-03-26 09:41 review
Messages (12)
msg52808 - (view) Author: Gael Le Mignot (kilobug) Date: 2007-07-04 09:54
Data going through the xmlrpclib are copied many times in Python. This creates problem when submitting blob of data in XML-RPC (I know the protocol is not aimed for that, but sometimes it's useful).

This little patch try to limit memory usage of xmlrpclib by adding a few optimizations.

I did the patch on Python 2.4 because it's what our customer is using, but it should work on Python 2.5 or more recent too.
msg110892 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-20 13:28
The code in the patch has been removed from py3k.  This can only go forward if a unit test is provided.  Change assigned to as per maintainers list.
msg110900 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-07-20 14:09
Martin has recently removed his name from py3k/Misc/maintainers.rst.
Also, he has stated that assigning bugs to him might mean that they
get less attention (due to the large number of issues that he's involved
in).
msg110920 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-20 16:52
Could somebody update here please http://svn.python.org/view/*checkout*/python/branches/py3k/Misc/maintainers.rst
msg184825 - (view) Author: Thomas Fenzl (Thomas Fenzl) * Date: 2013-03-21 01:43
Adapted the patch to python3.3
It may be useful with large amounts of transfered data avoiding a copy and freeing memory earlier.
The functionality is tested with the existing unit tests.
msg184949 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-03-22 09:50
Well, str.join() already optimizes this case to avoid copies:

>>> s = "x" * 50
>>> id(s)
139712615414000
>>> id(''.join([s]))
139712615414000
msg184950 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-03-22 09:51
I would close this as out of date, unless you have other suggestions to improve memory consumption.
msg184961 - (view) Author: Gael Le Mignot (kilobug) Date: 2013-03-22 11:32
Great that join does the optimisation by itself now, but the last issue of the patch (cleaning the _data array before calling f() so the memory of _data can be collected earlier) still seems meaningful today.
msg185264 - (view) Author: Thomas Fenzl (Thomas Fenzl) * Date: 2013-03-26 09:41
I removed the unnecessary check on single-element arrays. 
No strong opinion on usefulness, as I don't use xmlrpc a lot...
msg189963 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-25 14:43
Do you have any benchmarks?
msg190134 - (view) Author: Gael Le Mignot (kilobug) Date: 2013-05-27 08:20
It's not something that can be easily benched because it depends a lot of the use case. If some conditions are present (big amount of data sent to XML-RPC, the XML-RPC server taking a long time to answer, end either Python giving back the memory to the OS or another thread reusing the garbage collected memory) the gain in memory can be very significant, in most other cases, it won't change anything. 

Do you me to craft a simple example where the difference can be seen ?
msg190136 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-27 09:55
Yes, it would be nice if you provided an example.
History
Date User Action Args
2016-01-20 09:57:29serhiy.storchakasetstatus: open -> pending
2014-05-13 22:11:17skrahsetnosy: - skrah
2014-02-03 17:10:26BreamoreBoysetnosy: - BreamoreBoy
2013-05-27 09:55:28serhiy.storchakasetmessages: + msg190136
2013-05-27 08:20:40kilobugsetmessages: + msg190134
2013-05-25 14:43:04serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg189963
2013-03-26 09:41:19Thomas Fenzlsetfiles: + xmlrpc_less_copy-1.diff

messages: + msg185264
2013-03-22 11:32:37kilobugsetmessages: + msg184961
2013-03-22 09:51:49pitrousetmessages: + msg184950
2013-03-22 09:50:33pitrousetnosy: + pitrou
messages: + msg184949
2013-03-21 01:43:21Thomas Fenzlsetfiles: + xmlrpc_less_copy.diff
versions: + Python 3.4, - Python 2.7
nosy: + Thomas Fenzl

messages: + msg184825
2010-07-20 16:52:15BreamoreBoysetmessages: + msg110920
2010-07-20 16:39:17loewissetassignee: loewis ->
2010-07-20 14:09:14skrahsetnosy: + skrah
messages: + msg110900
2010-07-20 13:28:38BreamoreBoysetversions: - Python 3.1
nosy: + loewis, BreamoreBoy

messages: + msg110892

assignee: loewis
2009-04-06 10:30:55ajaksu2setstage: test needed
type: resource usage
versions: + Python 3.1, Python 2.7
2007-07-04 09:54:37kilobugcreate