Following the discussion on python-dev [1], this issue will track the re-organization of Lib/xml/etree to expose the C implementation (_elementtree) by default when importing ElementTree. The test suite will also have to be updated - it's required that it exercises both the C and the Python implementations.

I would like to make the import "magic" simple. Thus, the idea I currently plan to pursue is:

* xml/etree/ElementTree.py will be a simple facade that attempts to 'import *' from _elementtree, and on failure does 'import *' from pyElementTree
* The current contents of xml/etree/ElementTree.py will move to xml/etree/pyElementTree.py
* xml/etree/cElementTree.py disappears.

The test suite will be modified accordingly.


I'll be working on creating a patch for this. Any help, ideas, comments and discussions are more than welcome.


[1] http://mail.python.org/pipermail/python-dev/2012-February/116278.html

Oh, and not to forget: the documentation has to be updated to just not mention cElementTree any longer. For the user, the fact that a fast C library is invoked underneath should be invisible.

Eli Bendersky, 10.02.2012 15:52:
> * The current contents of xml/etree/ElementTree.py will move to xml/etree/pyElementTree.py

IIRC, there is a well specified way how accelerator modules should be used
by Python modules. I recall a lengthy discussion on python-dev (or the py3k
list?) back in the old pre-3.0 days, maybe there's even a PEP?

> * xml/etree/cElementTree.py disappears.

Careful with backwards compatibility here. It's the accelerator module
(_elementtree.so, IIRC) which is to be moved behind ElementTree.py.

I don't see a compelling enough reason to break imports in existing code by
removing the cElementTree module, so we should not do that.

Stefan

>> IIRC, there is a well specified way how accelerator modules should be used
by Python modules. I recall a lengthy discussion on python-dev (or the py3k
list?) back in the old pre-3.0 days, maybe there's even a PEP?

If there's a convention, I'll happily follow it. It would be great if someone could dig up the relevant details (I'll try to look for them myself).

>> I don't see a compelling enough reason to break imports in existing code by
removing the cElementTree module, so we should not do that.

Agreed. Perhaps it should just be deprecated?

Hmm, that may be PEP 399:

  If an acceleration module is provided it is to be named the same as the module it is accelerating with an underscore attached as a prefix, e.g., _warnings for warnings. The common pattern to access the accelerated code from the pure Python implementation is to import it with an import *, e.g., from _warnings import *. This is typically done at the end of the module to allow it to overwrite specific Python objects with their accelerated equivalents. 

However, it's hardly a rule, just describing a "common pattern". I wonder why this approach is preferable to the one I proposed (explicit facade module)?

Eli Bendersky, 10.02.2012 16:43:
>>> I don't see a compelling enough reason to break imports in existing code by
>>> removing the cElementTree module, so we should not do that.
> 
> Agreed. Perhaps it should just be deprecated?

Given that its mere existence is currently almost undocumented (except for
one tiny sentence in the docs), I vote for clearly documenting it as
deprecated, yes, with a mention to the fact that it's automatically used by
xml.etree.ElementTree starting with 3.3.

I wouldn't want to see more than that done, though, specifically not a
visible warning when it's being imported. There's far too much code out
there that uses it in previous Python versions. Such a warning would strike
even if it's only being imported conditionally with a try-except, which is
the most common way of doing it. So it would hit most users and require an
awful lot of code to be touched to fix it, for basically zero benefit.

Stefan

A note in the doc is easy to miss IMHO, and since DeprecationWarnings are silenced by default, I don't think they will affect the final users.

A different "problem" is that developers will have to check for the Python version if they want to use ElementTree on Python >=3.3 and keep using cElementTree on <3.3 (unless another way is provided).

If possible I would avoid pyElementTree, and keep ElementTree that imports from _elementtree and the deprecated cElementTree (until it can be removed).

> If possible I would avoid pyElementTree,

I suppose it's possible, but I'm genuinely interested in a technical reason for doing so. The approach suggested in PEP 399 is useful for module in which part of the functionality is implemented in C, and then augmented in Python. ElementTree is different - it's pretty much two separate implementations of the same API.

So, I think there's little question in terms of simplicity and clarity. Having pyElementTree and cElementTree (keeping it for backwards compat), and a facade named ElementTree that chooses between them is simple, clean and intuitive.

From a performance point of view, consider the (by far) common case - where _elementtree *is* successfully imported. 

Option 1: from _elementtree import *, at the end of the Python implementation in ElementTree.py - so for each invocation, the whole import of the Python code has to be done, just to reach the overriding import * at the end.

Option 2: ElementTree is a facade that attempts to import _elementtree first. So the Python implementation in pyElementTree doesn't even have to be parsed and imported

> If possible I would avoid pyElementTree,

Me too:
 - __name__ and __qualname__ would be less confusing
 - the cElementTree accelerator uses large parts of Python implementation

> ElementTree is different - it's pretty much two separate implementations of the same API.

Not fully separated... there's some python code hidden in the C module.

> From a performance point of view, consider the (by far) common case
> - where _elementtree *is* successfully imported.
> ... for each invocation, the whole import of the Python code has
> to be done, just to reach the overriding import * at the end.

This point is wrong... the _elementtree.c accelerator imports Python ElementTree already.

As you can see on lines 2938 to 2945, the change could lead to an import cycle:
http://hg.python.org/cpython/file/705b56512287/Modules/_elementtree.c#l2938

Trying to sort this out, it already gives me a headache.
I would like to remove the Python bootstrap code from the C module and try to do it differently, in a more standard way.

>> From a performance point of view, consider the (by far) common case
>> - where _elementtree *is* successfully imported.
>> ... for each invocation, the whole import of the Python code has
>> to be done, just to reach the overriding import * at the end.
>
> This point is wrong... the _elementtree.c accelerator imports Python ElementTree already.
>
> As you can see on lines 2938 to 2945, the change could lead to an import cycle:
> http://hg.python.org/cpython/file/705b56512287/Modules/_elementtree.c#l2938
>
> Trying to sort this out, it already gives me a headache.
> I would like to remove the Python bootstrap code from the C module and try to do it differently, in a more standard way.
>

The Python code inside _elementtree could be moved to Python code,
which would then import the Python stuff it needs from pyElementTree.
Since pyElementTree doesn't import _elementtree, there will be
circular dependencies.

So this is a point *in favor* of pyElementTree being pure-Python :-)

In other words:

In xml/etree there is:

- ElementTree: tries to import cElementTree. On success, done. On
ImportError, imports pyElementTree
- pyElementTree: the pure Python implementation
- cElementTree: sets up the bootstrap Python code and tries to import
_elementtree. In case of an error, propagates an ImportError up.

Would that work?

Oops, in last message:

s/there will be circular dependencies/there will not be circular dependencies/

> In xml/etree there is:
> - ElementTree: tries to import cElementTree. On success, done. On
>   ImportError, imports pyElementTree
> - pyElementTree: the pure Python implementation
> - cElementTree: sets up the bootstrap Python code and tries to import
>   _elementtree. In case of an error, propagates an ImportError up.

What I had in mind is more like:
 - ElementTree: defines the python code and if _elementtree is available overrides part of it with the functions imported from it;
 - cElementTree: at this point it could just be a deprecated alias for ElementTree

> What I had in mind is more like:
>  - ElementTree: defines the python code and if _elementtree is available overrides part of it with the functions imported from it;

The problem with this is the bootstrap Python code executed by
_elementtree. That should not be executed when _elementtree (the C
parts) can't be imported. Keeping this code in ElementTree will
probably complicate matters since it will add import conditions.

>>  - ElementTree: defines the python code and if _elementtree is 
>>  available overrides part of it with the functions imported from it;

> The problem with this is the bootstrap Python code executed by
_elementtree.

This might become unnecessary if ElementTree becomes the main module and _elementtree only contains a few faster functions/classes that are supposed to replace the ones written in Python.
So basically you only have a single fully functional Python module (ElementTree) plus an optional C module (_elementtree) that only provides faster replacements for ElementTree.

> That should not be executed when _elementtree (the C parts) can't be
> imported.

We are assuming that _elementtree might be missing, but what are the cases where this might actually happen? Other implementations like PyPy? Exotic platforms that can't compile _elementtree?

> Keeping this code in ElementTree will probably complicate 
> matters since it will add import conditions.

Wouldn't that as simple as having in ElementTree.py:
...
full python code here...
...
try:
    # override a few functions/classes with the faster versions
    from _elementtree import *
except ImportError:
    # _elementtree is missing, so we just keep the "slow" versions
    pass
else:
    # do the rest here if at all needed (e.g. plug the faster
    # versions in the right places)

I'm not familiar with ElementTree (I just looked at the bootstrap bit quickly), so what I'm saying might not be applicable here, but I've seen other modules doing something similar to what I'm proposing (json, heapq, maybe even warning and others).

The first step is to strip out the cElementTree bootstrap code from the C module.
I did it in the attached patch (plus removal of obsolete code for copy() in Python 2.4).
This passes the unmodified tests "test_xml_etree" and "test_xml_etree_c".

Then I think the right approach is to fold completely cElementTree behind ElementTree.
The cElementTree alias can be simply declared in Lib/xml/etree/__init__.py.

Ezio,

>> We are assuming that _elementtree might be missing, but what are the cases where this might actually happen? Other implementations like PyPy? Exotic platforms that can't compile _elementtree?

I guess both. To make the stdlib work on PyPy without changes, it has to be able to load the pure Python modules in a fallback.

As for platforms that can't compile _elementtree, keep in mind that there's also expat which _elementtree uses, so it's a lot of code to compile. Python works on some embedded systems, I'm not sure all of them can compile this code.

Florent, thanks for the patch - at this point code is more useful than talk :-)

Anyhow, I tried to apply it and a few tests in test_xml_etree_c fail, because it can't find fromstring and fromstringlist. This gets fixed when I import fromstringlist in cElementTree.py from ElementTree, and in the same file assign:

  fromstring = XML

Which is similar to what ElementTree itself does.

In general, I agree that a good first step would be to refactor the code to extract the boostrapping from _elementtree.c to cElementTree.py. As long as the tests pass, this can be committed regardless of this issue's original intent.

However, why did you leave some bootstrapping code inside? Can't all of it go away?

I strongly feel that existing code importing ElementTree or cElementTree should not be broken.  Let’s add transparent import from _elementtree to ElementTree without breaking existing uses of cET.

I think that 3.2 and 2.7 should get a doc note about cET, do we have a bug for this?

> I strongly feel that existing code importing ElementTree or cElementTree should not be broken.  Let’s add transparent import from _elementtree to ElementTree without breaking existing uses of cET.
>

AFAICS there's currently no disagreement on this point. The import
from cElementTree will keep working in 3.3 as it always had. However,
the explicit mention of cElementTree should be removed from the
documentation of ElementTree. The only remaining question is whether a
silent deprecation warning should be added in cElementTree.

> I think that 3.2 and 2.7 should get a doc note about cET, do we have a bug for this?
>

What doc note? Something in the spirit of: "Note that in 3.3, the
accelerated C implementation will be provided by default when
importing ElementTree" - or do you mean something else?

I don't think there's an open bug for this.

The more I think about it, the more the bootstrap code in _elementtree.c annoys me. It's the only instance of calling PyRun_String in Modules/ !

It's hackish and causes ugly import problems. If the C code needs stdlib functionality like copy.deepcopy, it should use PyImport_ImportModule like everyone else and not through a PyRun_String hack.

Since we've already decided to do some refactoring, I suggest all trace of the bootstrap is removed from _elementtree.c

Eli Bendersky, 11.02.2012 09:08:
> The more I think about it, the more the bootstrap code in _elementtree.c
> annoys me. It's the only instance of calling PyRun_String in Modules/ !
> 
> It's hackish and causes ugly import problems. If the C code needs stdlib
> functionality like copy.deepcopy, it should use PyImport_ImportModule
> like everyone else and not through a PyRun_String hack.

I find it perfectly legitimate to run Python code from a C module.
Certainly not a hack. We all know that most non-trivial functionality can
be expressed much easier in Python than in C, that's why we use Python
after all. In particular, defining a class with attributes and methods is a
couple of lines of code in Python, but a huge amount of code in C. Avoiding
the complexity of writing everything in C, or even of splitting the code in
a harder to understand way, is worth it.

That being said, I think it's worth removing any clear *redundancy* with
ET.py, as Florent's patch did. The goal is to keep _elementtree.c a pure
accelerator module that improves plain ElementTree, and redundancy is
counterproductive in this context. But if the implementation differs for
some reason, I would tend towards leaving it as is.

Stefan

>> I find it perfectly legitimate to run Python code from a C module.
Certainly not a hack. We all know that most non-trivial functionality can
be expressed much easier in Python than in C, that's why we use Python
after all. In particular, defining a class with attributes and methods is a couple of lines of code in Python, but a huge amount of code in C. Avoiding the complexity of writing everything in C, or even of splitting the code in a harder to understand way, is worth it.
<<

There can be arguments both way, but if we follow the lead of existing standard extension modules, the tendency is clearly not to use PyRun_String. Many C extensions use functionality from Python, but none does it the "bootstrap way". Why is that? Is there a good reason, or is it just convention?

> Anyhow, I tried to apply it and a few tests in test_xml_etree_c fail,
> because it can't find fromstring and fromstringlist.

Ooops, I cut some redundancy after running the tests, and I forgot to re-add the import. You're right.

> However, why did you leave some bootstrapping code inside?
> It's the only instance of calling PyRun_String in Modules/

I just tried to cut the import cycle and import it the other way.
I think it was done like that historically, for some reason, when
the module was first developped (for Python 1.5 maybe ...)
It is not necessary to remove all the Python code at once, and I am better at Python than at C.
We can delay this additional clean-up at a later time, it does not
block the PEP399 implementation.

Updated patch:
 - fixed missing import and missing alias
 - moved the XMLTreeBuilder alias to the Python module

New changeset 31dfb4be934d by Florent Xicluna in branch 'default':
Issue #13988: move the python bootstrap code to cElementTree.py, and remove obsolete code for Python 2.4 and 2.5.
http://hg.python.org/cpython/rev/31dfb4be934d

I've pushed this first part, which is just a code refactoring.

I tried to work out a patch for the second part.
The tricky thing is because of xml.etree still using doctests.
The patch for the tests seems to be enough small and readable.

We have small differences between C and Python, about the warnings beeing raised. In general the C implementation do not raise the deprecation warnings. IMHO, this could be fixed later.

Still missing is the patch for the documentation.

Another random cleanup idea:

ElementTree.py has this code:

try:
    from . import ElementPath
except ImportError:
    ElementPath = _SimpleElementPath()

Since in the stdlib ElementPath.py is always there, this is meaningless, so I'd say this try... except ImportError contraption can be removed, as well as _SimpleElementPath, and just replaced by:

from . import ElementPath

>> I think that 3.2 and 2.7 should get a doc note about cET
> What doc note?

I was referring to one of the points raised in the email thread: the docs don’t tell people that they can import a faster ET version, cET.

>> I was referring to one of the points raised in the email thread: the docs don’t tell people that they can import a faster ET version, cET.

Well, they *do*, but very modestly :-) I agree that should be improved and emphasized a bit, perhaps even mentioning (at least in the 3.2 doc) that in 3.3 it's going to be done by default. Is that accepted practice? I mean to mention in docs of 3.x some upcoming change in version 3.x+1 ?

Florent,

The issue13988_fold_cET_behind_ET.diff patch looks good to me. Unless there are objections from others, you can commit! Even the tests became simpler now, that's awesome ;-)

Now, what's left for this issue:

1. Update the documentation
2. Update "what's new"
3. Discuss and possibly implement the deprecation of cElementTree

I'll send a patch for 1 & 2 soon. What about 3?

Attaching a patch for Doc/library/xml.etree.elementtree.rst and Misc/NEWS.

The doc notice is modeled after a similar notice in the doc of 'pickle'. Note that I've also removed the mention that effbot's site is the home of the development version of the library, since it's no longer formally true.

Florent,

Just something minor I noticed in the new cElementTree.py:

  # Wrapper module for _elementtree

  from xml.etree.ElementTree import *
  # Not in __all__
  from xml.etree.ElementTree import ElementPath, XMLID, register_namespace

The "wrapper" comment should be removed, and possibly replaced with "This module exists for backwards compatibility with releases earlier than 3.3; Please import xml.etree.ElementTree directly"

Also, regarding the names not in __all__. Any good reason for them not to be? Seems like an omission to me.

> perhaps even mentioning (at least in the 3.2 doc) that in 3.3 it's going
> to be done by default. Is that accepted practice?
I don’t think we ever do that, and it would be confusing.

-1 to mentioning _elementtree, an implementation detail.  What I was talking about is cElementTree.

Éric,

Could you please open a new issue (with a dependency on this one) and explain there clearly what you want/mean?

There was a discussion in December which ruled out some CPython too specific implementation details from the documentation.
http://mail.python.org/pipermail/python-dev/2011-December/115063.html

Maybe it's better to remove these 2 lines about the "transparent optimizer". Then the "versionchanged" tag can be changed a little:
.. versionchanged:: 3.3
   This module will use a fast implementation whenever available.
   The module :mod:`xml.etree.cElementTree` is deprecated.

Probably we'll add few words in the Doc/whatsnew/3.3.rst too.

Updated patch:
 - add 'XMLID' and 'register_namespace' to the ElementTree.__all__
 - the comment says explicitly that cElementTree is deprecated
 - exercise the deprecated module with the tests

Florent,

Your updated patch looks good. I think that the explicit import of _namespace_map into cElementTree is just to satisfy some weird magic in the tests and can probably be removed as well (along with the weird magic :-), but that's not really important and can be left for later cleanups.

Regarding the documentation, alright let's not mention the implementation detail, and your "versionchanged" addition makes sense. I don't think adding directly to whatsnew/3.3.rst is necessary, updating Misc/NEWS is enough.

I'll apply the documentation patch after you apply the code patch. Or if you want, you can apply it yourself, I don't mind.

Thanks for the cooperation!

By the way, I see that if the explicit import of _namespace_map is commented out, the test_xml_etree_c test fails because it's not in the __all__ list. So the test can just import it directly with:

from xml.etree.ElementTree import _namespace_map

And the import in cElementTree won't be necessary. After all, _namespace_map is definitely not a public API!

This will keep cElementTree an nice-and-clean:

from xml.etree.ElementTree import *

> from xml.etree.ElementTree import _namespace_map
>
> And the import in cElementTree won't be necessary.
> After all, _namespace_map is definitely not a public API!

Because of the interaction of the support.import_fresh_module with the CleanContext context manager, it's not so easy to remove black magic.
I don't find better than:

        if hasattr(ET, '_namespace_map'):
            _namespace_map = ET._namespace_map
        else:
            from xml.etree.ElementTree import _namespace_map


This is why I kept the import in the deprecated "cElementTree" at first.
It does not hurt (it's private API), and it makes the test easier.

( If you have doubts, try ./python -m test test_xml_etree{,_c} or variants. )


I will probably commit code and documentation at once. It makes things easier regarding traceability.

Alright, it's not really important at this point and can be cleaned up
later.

>
> I will probably commit code and documentation at once. It makes things
> easier regarding traceability.
>

Sounds good

FWIW the JSON doc doesn't even mention the acceleration module _json, but since people here are used to import cElementTree I think it should be mentioned that it's now deprecated and accelerations are used automatically, so something like this would work:

.. versionchanged:: 3.3
   The :mod:`xml.etree.cElementTree` module is now deprecated.
   A fast implementation will be used automatically whenever available.

I also agree with Éric that there's no need to mention _elementtree (people might try to import that instead, and other implementations might use a different name).

Lib/test/test_xml_etree_c.py could also be removed, and the other tests could import cElementTree too (even though I'm not sure that works too well with doctests).

Shouldn't cElementTree raise an error when _elementtree is missing?
A DeprecationWarning should be added too.

New changeset 65fc79fb4eb2 by Florent Xicluna in branch 'default':
Issue #13988: cElementTree is deprecated and the _elementtree accelerator is automatically used whenever available.
http://hg.python.org/cpython/rev/65fc79fb4eb2

New changeset e9cf34d56ff1 by Florent Xicluna in branch 'default':
Fix xml_etree_c test error (follow up of issue #13988).
http://hg.python.org/cpython/rev/e9cf34d56ff1

Now the merge is done. Thank you Eli for the effort, and to the other contributors for the review.

Following topics may need further work:

 - add a Deprecation warning for cElementTree? it will annoy the package maintainers which support both 3.2 and >= 3.3, because either they'll use the non-accelerated version in 3.2, or they will have the Deprecation warning in 3.3 ... IMHO, it's better to do nothing, and just keep the mention in the documentation that it is deprecated.

 - raise the Deprecation warnings for the functions and methods which are marked as deprecated in the documentation (the Python code does it, but not the C accelerator)

 - convert _elementtree.c Python bootstrap code to C

 - refactor the test suite


These topics are not high priority. A specific issue should be opened if any of them require some attention.

I would add to the TODO - improve the documentation of the module. Opened issue 14006 for this.

I started going over the deprecated methods in ElementTree and ran into a more serious problem. XmlParser.doctype() is listed as deprecated, and indeed ElementTree (the Python version) issues a deprecation warning. However, the C implementation doesn't have doctype() at all so I get AttributeError.

DeprecationWarnings aren't that annoying anymore now that they're silent by default. It should at least have a PendingDeprecationWarning

Opened issue 14007 to track the doctype() problem

Emitting a deprecation warning on importing cElementTree has been rejected in the pydev list. The other remaining tasks have new issues on them, so this issue is done now.

I'm still not sure that's the best option.  Without deprecation people will keep using cElementTree and we will have to keep it around forever (or at least until Python4 and then have a 3to4 to fix the import).
This might be fine, but as a developer I would still like Python to tell me "You can just import ElementTree now, there's no need to use cElementTree".  Maybe the deprecation can be added to 3.4?

P.S. I'm fine with keeping it around for several more versions, but if we eventually have to remove it, we would still have to warn the users beforehand.  The more we wait, the more users will be still using cElementTree by the time we will actually remove it.

I don’t see benefits in removing cET.

Hi,

the C implementation of ElementTree do not support namespaces for find/all/... .

To me this is a serious regression, as I rely on ElementTree namespace support, and 3.3 would break it with this change.
Breaking namespace support is a fundamental problem.

Please reconsider this therefore.

Code to reproduced attached - works fine with python 3.2.

As the C implementation of ElementTree and Element lack the namespace keyword for findall (and *all* the other methods), 
where namespaces are very important when dealing with xml, 
and it is not possible to prevent using the v implementation of ElementTree without changing the python install, 
I propose to revert this change.

Until the C implementation can do namespaces as well.

The file was bad, sorry.
re-attached

Markus (cmn): Please file a separate issue, which will be a release blocker for 3.3 release. (It's not the only regression.)

Temporary very ugly workaround (before importing xml.etree.ElementTree) is:

import sys
sys.modules["_elementtree"] = None

It seems to me that namespaces are actually supported, but they are accepted only as positional args and not keyword args, so this should be easy to fix.

As advised I opened a new bug on this:
http://bugs.python.org/issue14818

New bug - C implementation of ElementTree: Inheriting from Element breaks text member
http://bugs.python.org/issue14849

Note: last traces of Python bootstrap code were removed from _elementtree in changeset 652d148bdc1d