msg191894 - (view) |
Author: Bryce Nesbitt (brycenesbitt) |
Date: 2013-06-26 03:48 |
ElementTree offers a wonderful and easy API for parsing XML... but if there is a namespace involved it suddenly gets ugly. This is a proposal to fix that. First an example:
------------------
!/usr/bin/python
# Demonstrate awkward behavior of namespaces in ElementTree
import xml.etree.cElementTree as ET
xml_sample_one = """\
<?xml version="1.0"?>
<presets>
<thing stuff="some stuff"/>
<thing stuff="more stuff"/>
</presets>
"""
root = ET.fromstring(xml_sample_one)
for child in root.iter('thing'):
print child.tag
xml_sample_two = """\
<?xml version="1.0"?>
<presets xmlns="http://josm.openstreetmap.de/tagging-preset-1.0">
<thing stuff="some stuff"/>
<thing stuff="more stuff"/>
</presets>
"""
root = ET.fromstring(xml_sample_two)
for child in root.iter('{http://josm.openstreetmap.de/tagging-preset-1.0}thing'):
print child.tag
------------------
Because of the namespace in the 2nd example, a {namespace} name keeps {namespace} getting {namespace} in {namespace} {namespace} the way.
Online there are dozens of question on how to deal with this, for example: http://stackoverflow.com/questions/11226247/python-ignore-xmlns-in-elementtree-elementtree
With wonderfully syntactic solutions like 'item.tag.split("}")[1][0:]'
-----
How about if I could set any root to have an array of namespaces to suppress:
root = ET.fromstring(xml_sample_two)
root.xmlns_at_root.append('{namespace}')
Or even just a boolean that says I'll take all my namespaces without qualification?
|
msg194896 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2013-08-11 15:53 |
FWIW, lxml.etree supports wildcards like '{*}tag' in searches, and this is otherwise quite rarely a problem in practice.
I'm -1 on the proposed feature and wouldn't mind rejecting this all together. (At least change the title to something more appropriate.)
|
msg194899 - (view) |
Author: Eli Bendersky (eli.bendersky) * |
Date: 2013-08-11 16:05 |
I was planning to look more closely at the namespace support in ET at some point, but haven't found the time yet.
[changing the title to be more helpful]
|
msg194901 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2013-08-11 16:17 |
There's also the QName class which can be used to split qualified tag names. And it's pretty trivial to pre-process the entire tree by stripping all namespaces from it the intention is really to do namespace agnostic processing. However, in my experience, most people who want to do that haven't actually understood namespaces (although, admittedly, sometimes it's those who designed the XML format who didn't understand namespaces ...).
|
msg194906 - (view) |
Author: Eli Bendersky (eli.bendersky) * |
Date: 2013-08-11 17:41 |
> (although, admittedly, sometimes it's those who designed the XML format
who didn't understand >namespaces ...).
I fully concur. The design of XML, in general, is not the best
demonstration of aesthetics in programming. But namespaces always seem to
me to be one further step in the WTF direction. This is precisely why I
didn't reject this issue right away: perhaps it's not a bad idea to provide
Python programmers with *some* way to ease namespace-related tasks (even if
they go against the questionable design principles behind XML).
|
msg194919 - (view) |
Author: Bryce Nesbitt (brycenesbitt) |
Date: 2013-08-12 04:06 |
The mere existence of popular solutions like
'item.tag.split("}")[1][0:]' argues something is wrong. What could
lmxl do to make this cleaner (even if the ticket proposal is junk).
|
msg194920 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2013-08-12 04:21 |
Please leave the title as it is now.
|
msg194923 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2013-08-12 04:34 |
As I already suggested for lxml, you can use the QName class to process qualified names, e.g.
QName(some_element.tag).localname
Or even just
QName(some_element).localname
It appears that ElementTree doesn't support this. It lists the QName type as "opaque". However, it does provide a "text" attribute that contains the qualified tag name.
http://docs.python.org/2/library/xml.etree.elementtree.html#xml.etree.ElementTree.QName
Here is the corresponding documentation from lxml:
http://lxml.de/api/lxml.etree.QName-class.html
QName instances in lxml provide the properties "localname", "namespace" and "text".
|
msg216768 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2014-04-18 01:12 |
FWIW, I would like to have a way to ignore namespaces.
For many day-to-day problems (parsing Microsoft Excel
files saved in an XML format or parsing RSS feeds),
this would be a nice simplification.
I teach Python for a living and have found that it is
common for namespaces to be an obstacle for people
trying to get something done.
Giving them the following answer is unsatisfactory response to legitimate needs:
"""
And it's pretty trivial to pre-process the entire tree by stripping all namespaces from it the intention is really to do namespace agnostic processing. However, in my experience, most people who want to do that haven't actually understood namespaces (although, admittedly, sometimes it's those who designed the XML format who didn't understand namespaces ...).
"""
|
msg216774 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2014-04-18 05:08 |
You can already use iterparse for this.
it = ET.iterparse('somefile.xml')
for _, el in it:
el.tag = el.tag.split('}', 1)[1] # strip all namespaces
root = it.root
As I said, this would be a little friendlier with support in the QName class, but it's not really complex code. Could be added to the docs as a recipe, with a visible warning that this can easily lead to incorrect data processing and therefore should not be used in systems where the input is not entirely under control.
Note that it's unclear what the "right way to do it" is, though. Is it better to 1) alter the data by stripping all namespaces off, or 2) let the tree API itself provide a namespace agnostic mode? Depends on the use case, but the more generic way 2) should be fairly involved in terms of implementation complexity, for just a minor use case. 1) would be ok in most cases where this "feature" is useful, I guess, and can be done as shown above.
In fact, the advantage of doing it explicitly with iterparse() is that instead of stripping all namespaces, only the expected namespaces can be discarded. And errors can be raised when finding unnamespaced elements, for example. This allows for a safety guard that prevents the code from completely misinterpreting input. There is a reason why namespace were added to XML at some point.
|
msg235724 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2015-02-11 04:05 |
See Issue 8583 for a proposal that would apparently allow all namespaces to be ignored
|
msg271448 - (view) |
Author: jonathan miller (jjmiller50) |
Date: 2016-07-27 13:00 |
A flexible and pretty simple way opf loosening up handling namespaces would be to OPTIONALLY change what is done at parse time:
1. Don't handle xmlns declarations specially. Leave them as normal attributes, and the Element.attrib would have a normal entry for each.
2. Leave the abbreviation colon-separated prefix in front of the element
tags as they come in.
If the using code wants, it can walk the ElementTree contents making dictionaries of the active namespace declarations, tucking a dict reference into each Element. Maybe put in an ElementTree method that does this, why not?
I'm interested in this topic because I wish to handle xml from a variety of different tools, some of which had their XML elements defined without namespaces. They can use element names which are really common - like 'project' - and have no namespace definitions. Worse: if you put one in, the tool that originally used the element breaks.
Doing things as suggested gives the user the opportunity to look for matches using the colonized names, to shift namespace abbrevs easily, and to write out nicely namespaced code with abbrevs on the elements easily.
This would be OPTIONAL: the way etree does it now, full prefixing of URI, is the safe way and should be retained as the default.
|
msg271466 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2016-07-27 17:59 |
Here is a proposed patch for a new function "strip_namespaces(tree)" that discards all namespaces from tags and attributes in a (sub-)tree, so that subsequent processing does not have to deal with them.
The "__all__" test is failing (have to figure out how to fix that), and docs are missing (it's only a proposal for now). Comments welcome.
|
msg271472 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2016-07-27 19:47 |
On second thought, I think it should be supported (also?) in the parser. Otherwise, using it with an async parser would be different from (and more involved than) one-shot parsing. That seems wrong.
|
msg271475 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2016-07-27 23:22 |
Perhaps it would make more sense to use rpartition() or rstrip(). It seems possible to have a closing curly bracket in a namespace, but not in a element tag or attribute name.
My guess is the __all__ failure is just a sign to add the new function to the __all__ variable at the top of the module.
|
msg305405 - (view) |
Author: Tim Chambers (Tim Chambers) |
Date: 2017-11-02 08:33 |
I suggest adding the option to keep the namespace prefixes in the tree when reading in (will it need to set a tree wide variable for the instance?). I haven't looked at the etree internals in detail.
Add a function to ElementTree that returns the tag using the namespace prefix (eg. treenode.tagpre). Namespaces and prefixes are cached and used to expand the prefix only when absolutely required. Some XML/xpath search operations currently assume the full expanded namespaces not prefixes which may lead to side-effects.
You can leave the default behaviour for compatibility.
Using prefixes in the tree storage and searches would reduce memory and CPU time (when no expansion is required).
|
msg341000 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2019-04-27 16:58 |
Coming back to this issue after a while, I think it's still a relevant problem in some use cases. However, it's not currently clear what an improved solution would look like. The fully qualified tag names in Clark notation are long, sure, but also extremely convenient in being explicit and fully self-contained.
One thing I noticed is that the examples used in this and other reports usually employ the .find() methods. Maybe issues 28238 and 30485 can reduce the pain to an acceptable level?
Regarding the specific proposals:
> root.xmlns_at_root.append('{namespace}')
This cannot work since searches from a child element would then not know about the prefix. Elements in ElementTree do not have a global tree state.
> 2. Leave the abbreviation colon-separated prefix in front of the element tags as they come in.
Note that prefixes are only indirections. They do not have a meaning by themselves and multiple prefixes can (and often do) refer to the same namespace. Moving the prefix resolution from the parser to the users seems to make the situation worse instead of better.
|
msg341001 - (view) |
Author: Stefan Behnel (scoder) * |
Date: 2019-04-27 17:00 |
I was referring to issue 28238 and issue 30485.
|
msg389391 - (view) |
Author: Lars Hammarstrand (lars.hammarstrand) |
Date: 2021-03-23 15:12 |
Any update regarding this?
We switched to lxml to make life easier but it would be useful if this functionality also was implemented in the standard library.
Wishlist:
1. Flag to ignore all namespaces during find().
2. Ability to set default namespace the during find().
3. Clear existing namespaces similar to lxml cleanup_namespaces.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:47 | admin | set | github: 62504 |
2021-03-23 15:12:20 | lars.hammarstrand | set | nosy:
+ lars.hammarstrand messages:
+ msg389391
|
2019-04-27 17:00:59 | scoder | set | messages:
+ msg341001 |
2019-04-27 16:58:56 | scoder | set | messages:
+ msg341000 versions:
+ Python 3.8, - Python 3.6 |
2019-04-27 16:32:07 | scoder | link | issue8583 superseder |
2017-11-02 08:33:30 | Tim Chambers | set | nosy:
+ Tim Chambers messages:
+ msg305405
|
2016-07-28 04:37:21 | martin.panter | set | title: ElementTree -- provide a way to ignore namespace in tags and seaches -> ElementTree -- provide a way to ignore namespace in tags and searches |
2016-07-27 23:22:48 | martin.panter | set | messages:
+ msg271475 |
2016-07-27 19:47:37 | scoder | set | messages:
+ msg271472 |
2016-07-27 17:59:04 | scoder | set | files:
+ etree_strip_namespaces.patch keywords:
+ patch messages:
+ msg271466
versions:
+ Python 3.6, - Python 3.4 |
2016-07-27 13:00:50 | jjmiller50 | set | nosy:
+ jjmiller50 messages:
+ msg271448
|
2015-02-11 04:05:25 | martin.panter | set | messages:
+ msg235724 |
2014-04-18 05:08:06 | scoder | set | messages:
+ msg216774 |
2014-04-18 01:12:08 | rhettinger | set | nosy:
+ rhettinger messages:
+ msg216768
|
2014-04-17 23:14:32 | martin.panter | set | nosy:
+ martin.panter
|
2014-04-11 09:00:07 | pocek | set | nosy:
+ pocek
|
2013-08-12 04:34:27 | scoder | set | messages:
+ msg194923 |
2013-08-12 04:21:23 | scoder | set | messages:
+ msg194920 title: ElementTree gets awkward to use if there is an xmlns -> ElementTree -- provide a way to ignore namespace in tags and seaches |
2013-08-12 04:06:42 | brycenesbitt | set | messages:
+ msg194919 title: ElementTree -- provide a way to ignore namespace in tags and seaches -> ElementTree gets awkward to use if there is an xmlns |
2013-08-11 17:41:00 | eli.bendersky | set | messages:
+ msg194906 |
2013-08-11 16:17:22 | scoder | set | messages:
+ msg194901 |
2013-08-11 16:05:44 | eli.bendersky | set | messages:
+ msg194899 title: ElementTree gets awkward to use if there is an xmlns -> ElementTree -- provide a way to ignore namespace in tags and seaches |
2013-08-11 15:53:23 | scoder | set | versions:
+ Python 3.4, - Python 2.7 nosy:
+ scoder, eli.bendersky
messages:
+ msg194896
components:
+ Library (Lib), XML, - Extension Modules |
2013-06-26 03:48:17 | brycenesbitt | create | |