classification
Title: pulldom cannot handle xml file with large external entity properly
Type: resource usage
Components: XML Versions: Python 2.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: hanselda
Priority: Keywords:

Created on 2008-05-11 13:32 by hanselda, last changed 2008-05-11 13:32 by hanselda.

Messages
msg66628 (view) Author: Luyang Han (hanselda) Date: 2008-05-11 13:32 
when use xml.dom.pulldom module to parse a large xml file, if all the 
information is saved in one xml file, the module can handle it in the 
following way without construction the whole DOM:

events = xml.dom.pulldom.parse('file.xml')
for (event, node) in events:
    process(event, node)

But if 'file.xml' contains some large external entities, for example:

<!ENTITY file_external SYSTEM "others.xml">
<body>&file_external;</body>

Then using the same python snippet above leads to enormous memory 
usage. I did not perform a concrete benchmark, in one case a 3M 
external xml file drained about 1 GB memory. I think in this case it 
might be the whole DOM structure is constructed.
History
Date User Action Args
2008-05-11 13:32:17hanseldacreate