This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Remove old-deprecated ElementTree features (part 2)
Type: enhancement Stage: resolved
Components: XML Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: christian.heimes, eli.bendersky, hroncok, miss-islington, petr.viktorin, rahul-kumi, scoder, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2019-04-06 16:06 by serhiy.storchaka, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12707 merged serhiy.storchaka, 2019-04-06 16:13
PR 18937 merged rahul-kumi, 2020-03-11 17:51
PR 19108 merged serhiy.storchaka, 2020-03-22 09:03
PR 19188 merged hroncok, 2020-03-27 10:39
PR 19920 closed serhiy.storchaka, 2020-05-05 11:51
PR 19921 closed christian.heimes, 2020-05-05 11:57
PR 20117 merged serhiy.storchaka, 2020-05-15 18:34
PR 20780 merged miss-islington, 2020-06-10 15:39
Messages (27)
msg339533 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-04-06 16:06
The proposed PR removes old-deprecated ElementTree features.

* Methods Element.getchildren(), Element.getiterator() and ElementTree.getiterator() deprecated in 2.7 and 3.2. They were deprecated in the documentation only, and started to emit a warning in 3.8. Use list(elem) or iteration instead of getchildren(), methods iter() instead of getiterator().

* The xml.etree.cElementTree module deprecated in 3.3. It was deprecated documentation only because adding a runtime warning will cause more harm than removing it because of the common idiom of using it since Python 2:

    try:
        import xml.etree.cElementTree as ET
    except ImportError:
        import xml.etree.ElementTree as ET

TODO: Add a What's New entry after the start of developing 3.9.
msg339535 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-04-06 16:16
See also issue29209.
msg350938 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-09-01 08:18
New changeset f02ea6225bc3b71bd5fe66224d199a6e3e23b14d by Serhiy Storchaka in branch 'master':
bpo-36543: Remove old-deprecated ElementTree features. (GH-12707)
https://github.com/python/cpython/commit/f02ea6225bc3b71bd5fe66224d199a6e3e23b14d
msg364802 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-03-22 12:31
New changeset b33e52511a59c6da7132c226b7f7489b092a33eb by Serhiy Storchaka in branch 'master':
bpo-36543: Remove the xml.etree.cElementTree module. (GH-19108)
https://github.com/python/cpython/commit/b33e52511a59c6da7132c226b7f7489b092a33eb
msg365190 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-03-27 19:59
New changeset 6467134307cf01802c9f1c0384d8acbebecbd400 by Miro Hrončok in branch 'master':
bpo-36543: What's new: Document how to replace xml.etree.cElementTree (GH-19188)
https://github.com/python/cpython/commit/6467134307cf01802c9f1c0384d8acbebecbd400
msg368083 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-04 21:34
For the record, I'm quite unhappy that xml.etree.cElementTree was removed without going through a proper active deprecation cycle with plenty of head start. The removal came as a surprise to me -- and I'm a core dev, author of PEP 594 and owner of the defusedxml package. xml.etree.cElementTree module was also never added to PEP 4 and is still not mentioned as removed in PEP 4.

This changeset broke defusedxml, a package with over 10M downloads a month and 57k dependencies on Github. I'm sure that the removal is going to cause more trouble for other packages, too.

In PEP 594 I deliberately included a long deprecation phase for all packages with at least one release deprecation warnings. I gave users plenty of time to plan for removal.
msg368105 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-05-05 05:55
Christian, I understand your complaint, but I've actually never seen code in the wild that didn't provide a fallback for the import, usually something like what Serhiy spelled out and explained above:

    try:
        import xml.etree.cElementTree as ET
    except ImportError:
        import xml.etree.ElementTree as ET

This makes it inconvenient for users to emit a deprecation warning for the import, because it would impact perfectly future proof code like the above, without a good way for users to work around it by adapting their code, because the above import order is actually what they want. And there's a lot of such code.

I personally believe that the breakage will be quite limited. But that's a belief, not a fact. I don't have numbers to back it.
msg368108 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-05 06:31
xml.etree.cElementTree should be treated in the same way as cStringIO and cPickle -- they where separate modules in the past, but now the acceleration is used by default in io.StringIO and pickle. It looks an oversign that it was not removed in 3.0.

xml.etree.cElementTree was deprecated since 3.3. I think 6 releases (3.3-3.8) is long enough period. It was not possible to emit a deprecation warning at runtime for reasons mentioned above. And we did not have deprecation warnings for cStringIO and cPickle for the same reasons.

defusedxml is the only known to me package which does not fall back to xml.etree.ElementTree. I hope it is easy to fix it. In Python xml.etree.cElementTree is the same as xml.etree.ElementTree, so you can just import xml.etree.ElementTree. And you can use the above mentioned idiom if you still support Python 2.
msg368111 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-05-05 08:42
Some stats: See https://bugzilla.redhat.com/showdependencytree.cgi?id=PYTHON39&hide_resolved=0 and Ctrl+F for cElementTree.
msg368113 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-05-05 08:51
Thanks Miro, that's beautiful. Good to know that I'm not the only one who doesn't read documentation. ;-)

So, how do we deal with this? We can't warn about the deprecation without annoying everyone. We can't just remove the empty module without breaking lots of code. We also can't really let the module stick around forever and let people write more code that uselessly imports it.

Any ideas?
msg368118 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-05 09:06
Thank you Miro for information. It looks as awesome work!

The simplest solution would be to restore xml.etree.cElementTree and keep it forever. It will not harm anyone, it is just outdated import here and there.

We could also write an import hook which would analyze the code of the importer and warn only if xml.etree.cElementTree is imported without a common idiom. But it looks slightly overcomplicated to me.

And we can just let the users to fix they code for 3.9.
msg368120 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-05 09:20
Stefan wrote:
> Christian, I understand your complaint, but I've actually never seen code in the wild that didn't provide a fallback for the import, usually something like what Serhiy spelled out

My package defusedxml broke with the change. Python's documention recommends my package for parsing XML from untrusted sources. defusedxml 0.7.0rc1 addresses the problem by removing defusedxml.cElementTree in 3.9, too. The fix is going to break other packages, too.

Serhiy wrote:
> xml.etree.cElementTree should be treated in the same way as cStringIO and cPickle -- they where separate modules in the past, but now the acceleration is used by default in io.StringIO and pickle. It looks an oversign that it was not removed in 3.0.

I completely agree with you. It should have been removed with 3.0. But this fact doesn't grant a carte blanche to remove the module without a deprecation phase with a proper deadline.

Stefan wrote:
> So, how do we deal with this? We can't warn about the deprecation without annoying everyone. We can't just remove the empty module without breaking lots of code. We also can't really let the module stick around forever and let people write more code that uselessly imports it.

I prefer annoyance over getting broking by surprise. If the current deprecation warning system doesn't work for us, then we need to come up with a better plan. In the mean time please give users one release head star to adjust their code. I propose that feature shall not be removed in 3.x unless the documentation of 3.(x-1).0 release has explicitly stated the removal of a feature (unless other special circumstances and general consense require it).

* Add deprecation and removal warning for xml.etree.cElementTree to whatsnew/3.9.rst
* Remove xml.etree.cElementTree in 3.10
msg368121 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-05 09:28
IMHO we don't have to spend extra work on an import hook. It might be a good idea to raise the problem with https://github.com/PyCQA/, though. Large projects tend to use code checkers and linters.
msg368122 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-05-05 09:30
It's not like the deprecation was hidden – the ET docs have this line right at the top:

"Changed in version 3.3: This module will use a fast implementation whenever available. The xml.etree.cElementTree module is deprecated."

https://docs.python.org/3/library/xml.etree.elementtree.html

But as I said, people don't read docs.

I agree that it's best to keep the module "for now", and maybe leave it to IDEs and code checkers to spread the deprecation notice for us. As Serhiy said, it doesn't hurt us much to have it.
msg368123 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-05 09:43
3.3.0 was released in 2012. That's almost a lifetime ago. I assume that majority of users either forgot, missed the note, or no longer took the deprecation serious. After seven, eight years we should do our users a service and point out the deprecation with a concrete deadline.
msg368127 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-05-05 10:43
>> But as I said, people don't read docs.

> After seven, eight years we should do our users a service and point out the deprecation with a concrete deadline.

¯\_(ツ)_/¯

Honestly, let's just keep it. Maybe we can add an invisible PendingDeprecationWarning and wait if some static analysers pick it up. I wouldn't want to invest any more work into it than that.
msg368135 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-05 12:18
I do not think that it is right to emit a warning at import time. We should not punish people which do things right: fallback to xml.etree.ElementTree and test with -We.
msg371210 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-06-10 15:39
New changeset ec88e1bca81a167e6d5c0ac635e22f84298cb1df by Serhiy Storchaka in branch 'master':
bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." (GH-20117)
https://github.com/python/cpython/commit/ec88e1bca81a167e6d5c0ac635e22f84298cb1df
msg371212 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-06-10 16:01
New changeset 3b97d1becbe08cf56c58d9c740a4622cbf6285fd by Miss Islington (bot) in branch '3.9':
bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." (GH-20117) (GH-20780)
https://github.com/python/cpython/commit/3b97d1becbe08cf56c58d9c740a4622cbf6285fd
msg371214 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-06-10 16:04
The xml.etree.cElementTree module is restored. We'll see about what to do with it later. It hurts less to keep it around than to delete it.

I think this ticket can be closed.
msg384390 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-01-05 12:42
Let me just point out there's an official process for making incompatible changes now: https://www.python.org/dev/peps/pep-0387/#making-incompatible-changes

A warning must be emitted (not just mentioned in docs) for two releases before a feature is removed.
msg384399 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-01-05 14:11
This process is not new. The problem is that in this particular case warning will do more harm than removing the module. If we cannot just remove it, we should not introduce warning. It will be kept forever.
msg384400 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-01-05 14:14
Should the process be changed?
msg384401 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-01-05 14:16
The process is fine. A deprecation warning will give users time to update code before it hits production.
msg384412 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-01-05 16:05
Deprecation warning may help users of old lower-quality code at the cost of harming users of old higher-quality code. It does not look right to me.
msg384415 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-01-05 17:06
Apparently my defusedxml package is low-quality code...

Anyhow PEP 387 has clear rules how to handle incompatible changes. Every core dev has to follow these rules. You cannot simply ignore the rule, because you don't like it. If you want to change the rules or get an exception for a special case, please open a discussion, lobby for an amendment to PEP 387, and get approval from the SC.
msg384419 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-01-05 17:46
I do not understand what this discussion in a long-closed article is about. If you propose to add warning, I am against it. You are against removing the module. Well, the module will not be removed, the warning is no longer needed. The issue is closed.

I do not propose to change the rule. I say that the procedure cannot be executed in this case, because it will cause more harm than benefit. If we cannot follow the procedure for removing module, we cannot remove that module. It is fine to me. If you want to change this, propose changes to the procedure, but it should be discussed not here.
History
Date User Action Args
2022-04-11 14:59:13adminsetgithub: 80724
2021-01-05 17:46:41serhiy.storchakasetmessages: + msg384419
2021-01-05 17:06:18christian.heimessetmessages: + msg384415
2021-01-05 16:05:12serhiy.storchakasetmessages: + msg384412
2021-01-05 14:16:48christian.heimessetmessages: + msg384401
2021-01-05 14:14:48petr.viktorinsetmessages: + msg384400
2021-01-05 14:11:18serhiy.storchakasetmessages: + msg384399
2021-01-05 12:42:52petr.viktorinsetnosy: + petr.viktorin
messages: + msg384390
2020-06-10 16:04:24scodersetstatus: open -> closed
resolution: fixed
messages: + msg371214

stage: patch review -> resolved
2020-06-10 16:01:33scodersetmessages: + msg371212
2020-06-10 15:39:26miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19976
2020-06-10 15:39:19scodersetmessages: + msg371210
2020-05-15 18:34:01serhiy.storchakasetpull_requests: + pull_request19421
2020-05-05 12:18:22serhiy.storchakasetmessages: + msg368135
2020-05-05 11:57:40christian.heimessetpull_requests: + pull_request19237
2020-05-05 11:51:02serhiy.storchakasetstage: resolved -> patch review
pull_requests: + pull_request19235
2020-05-05 10:43:24scodersetmessages: + msg368127
2020-05-05 09:43:20christian.heimessetmessages: + msg368123
2020-05-05 09:30:34scodersetmessages: + msg368122
2020-05-05 09:28:24christian.heimessetmessages: + msg368121
2020-05-05 09:20:19christian.heimessetmessages: + msg368120
2020-05-05 09:06:16serhiy.storchakasetmessages: + msg368118
2020-05-05 08:51:24scodersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg368113
2020-05-05 08:42:22hroncoksetmessages: + msg368111
2020-05-05 06:31:18serhiy.storchakasetmessages: + msg368108
2020-05-05 05:55:13scodersetmessages: + msg368105
2020-05-04 21:34:10christian.heimessetnosy: + christian.heimes
messages: + msg368083
2020-03-27 19:59:06serhiy.storchakasetmessages: + msg365190
2020-03-27 10:39:35hroncoksetnosy: + hroncok

pull_requests: + pull_request18547
2020-03-22 12:31:48serhiy.storchakasetmessages: + msg364802
2020-03-22 09:03:37serhiy.storchakasetpull_requests: + pull_request18469
2020-03-11 17:51:20rahul-kumisetnosy: + rahul-kumi

pull_requests: + pull_request18289
2019-09-01 08:19:14serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-09-01 08:18:43serhiy.storchakasetmessages: + msg350938
2019-04-06 16:16:22serhiy.storchakasetmessages: + msg339535
2019-04-06 16:13:29serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request12631
2019-04-06 16:06:22serhiy.storchakacreate