classification
Title: DeprecationWarning for doctype() method when subclassing _elementtree.XMLParser
Type: behavior Stage: resolved
Components: Extension Modules, XML Versions: Python 3.6, Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Arfrever, berker.peksag, eli.bendersky, ezio.melotti, martin.panter, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-10-06 06:42 by martin.panter, last changed 2015-06-29 20:50 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
inherit-doctype.patch martin.panter, 2014-12-17 01:48 review
doctype-remove.patch martin.panter, 2014-12-18 06:42 review
doctype-remove.v2.patch martin.panter, 2015-03-31 06:02 Remove instead of fixing review
inherit-doctype.v2.patch martin.panter, 2015-03-31 06:28 Fix without removing review
doctype-remove.v3.patch martin.panter, 2015-03-31 07:10 Remove instead of fixing review
doctype-remove.v4.patch serhiy.storchaka, 2015-05-16 16:26 review
inherit-doctype.v3.patch serhiy.storchaka, 2015-06-26 17:45 For 3.4 review
Messages (22)
msg199031 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2013-10-06 06:42
I am using the C version of Element Tree via the main ElementTree module. I have subclassed XMLParser, and created my own target object. I am not that interested in XML doctypes, but the following simplified code raises a DeprecationWarning:

$ python3.3 -Wall
Python 3.3.2 (default, May 16 2013, 23:40:52) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from xml.etree.ElementTree import XMLParser
>>> class CustomParser(XMLParser): pass
... 
>>> CustomParser().feed("<!DOCTYPE blaua>")
__main__:1: DeprecationWarning: This method of XMLParser is deprecated.  Define doctype() method on the TreeBuilder target.

Looking at the C code, the logic is wrong. Subclasses of XMLParser will normally always have a doctype() method, at least by inheritance. So the code should compare the method with the XMLParser.doctype() base method rather than just checking that it exists. The native Python version seems to get it right.

http://hg.python.org/cpython/file/50ea4dccb03e/Modules/_elementtree.c#l3091

It looks like this may not be an issue for Python 3.4 because according to Issue 13248 the deprecated doctype() method is due to be removed.
msg201812 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-10-31 12:59
Thanks for the report, Martin. I'll take a look once I get some time
msg214165 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-03-20 01:27
Actually I am still seeing this in 3.4.0. Looks like the doctype() method was not removed after all.
msg232791 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-12-17 01:48
Since the doctype() method doesn’t appear to be removed from the “default” (3.5?) branch either, here is a patch that avoids the deprecation warning when inheriting method + a test.

Hopefully this will be useful when you get some time to look at it. I’m not sure if the PyCFunction_ calls are a good idea; they don’t seem to be documented, but I couldn’t think of a better way to check if the method was overridden.
msg232860 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-12-18 06:42
Here is another patch that removes the method instead, as suggested in the review
msg232862 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-12-18 07:37
doctype-remove.patch is acceptable for both 3.4 and 3.5 or only 3.5?
If only 3.5, then please apply inherit-doctype.patch in 3.4.
msg238783 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-21 10:23
Looks as there is yet one difference in the behavior of Python and C implementations. In Python implementation either self.target.doctype or self.doctype are called. But in C implementation both are called.
msg239671 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-31 06:02
The difference of calling XMLParser.doctype() between the implementations is another argument for removing it completely. With all these bugs, and no opposition that I know of, I think it should be okay to remove the deprecated doctype() method in 3.5.

doctype-remove.v2.patch adds a tweaked version of the original test for no DeprecationWarning from the other patch.
msg239673 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-31 06:28
inherit-doctype.v2.patch inverts the logic in the C module. This patch may still be useful if people want to apply it to 3.4, or do not want to remove the deprecated method from 3.5.
msg239674 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-31 07:10
doctype-remove.v3.patch includes an entry on the What’s New page.
msg239757 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-03-31 22:33
doctype-remove.v3.patch LGTM.
msg239772 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2015-04-01 06:38
Also please apply inherit-doctype.v2.patch in 3.4 branch.
msg243330 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-05-16 16:26
Updated to the tip. If Eli doesn't have objections, I'll commit it.
msg243332 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2015-05-16 16:42
I don't know how important this is to really warrant removal. Removal means potentially breaking working code when trying to run it with Python 3.5, and my impression was that the core devs are somewhat alergic to this, at least while the transition to Python 3 is ongoing.

Unless it has already been discussed (sorry if I missed it, haven't had much time for CPython recently, unfortunately), may be worth a quick check with python-dev
msg243466 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-05-18 08:42
Then what to do with the discrepancy between Python and C implementations (msg238783)?
msg243470 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2015-05-18 11:57
I'm not sure. This is why I'm proposing asking on python-dev
msg245047 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-06-09 08:03
Started topic on Python-Dev: http://comments.gmane.org/gmane.comp.python.devel/153655 .
msg245205 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-12 02:54
Ideally I guess the Python native behaviour is better: only call target.doctype() if available. It might allow you to easily implement doctype() in both the old and new versions of the API, without worrying about the API being called twice, and without experiencing any DeprecationWarning. But I do not have a real-world use case to demonstrate this, and I don’t think this decision should hold up committing inherit-doctype.v2.patch. They are separate bugs.

BTW: Another difference between the implementations is that the C version accepts my <!DOCTYPE blaua> test case, but the Python version ignores it. It only works with a more elaborate case like <!DOCTYPE blaua SYSTEM "dtd">. I’m no expert, but I think my original XML should be allowed.
msg245857 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-06-26 17:45
Here is a patch that also fixes other issues with doctype.

1) Direct call of doctype() issues a warning.
2) Parser's doctype() is not called if target's doctype() is called.
msg245945 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-29 15:38
inherit-doctype.v3.patch looks good to me
msg245963 - (view) Author: Roundup Robot (python-dev) Date: 2015-06-29 20:12
New changeset 75571407dcd3 by Serhiy Storchaka in branch '3.4':
Issue #19176: Fixed doctype() related bugs in C implementation of ElementTree.
https://hg.python.org/cpython/rev/75571407dcd3

New changeset 6ae8842e9b60 by Serhiy Storchaka in branch '3.5':
Issue #19176: Fixed doctype() related bugs in C implementation of ElementTree.
https://hg.python.org/cpython/rev/6ae8842e9b60

New changeset d792dc240456 by Serhiy Storchaka in branch 'default':
Issue #19176: Fixed doctype() related bugs in C implementation of ElementTree.
https://hg.python.org/cpython/rev/d792dc240456
msg245969 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-06-29 20:50
In 2.7 XMLParser is a function, not a class, and therefore can't be subclassed.
History
Date User Action Args
2015-06-29 20:50:13serhiy.storchakasetstatus: open -> closed
versions: - Python 2.7
messages: + msg245969

resolution: fixed
stage: commit review -> resolved
2015-06-29 20:12:28python-devsetnosy: + python-dev
messages: + msg245963
2015-06-29 15:38:47martin.pantersetmessages: + msg245945
2015-06-26 17:46:36serhiy.storchakasetversions: + Python 2.7, Python 3.4, Python 3.6
2015-06-26 17:45:26serhiy.storchakasetfiles: + inherit-doctype.v3.patch

messages: + msg245857
2015-06-12 02:58:27martin.pantersetcomponents: + Extension Modules
2015-06-12 02:58:07martin.pantersetcomponents: + XML, - Extension Modules
2015-06-12 02:54:04martin.pantersetmessages: + msg245205
2015-06-09 08:03:57serhiy.storchakasetmessages: + msg245047
2015-05-18 11:57:04eli.benderskysetmessages: + msg243470
2015-05-18 08:42:45serhiy.storchakasetmessages: + msg243466
2015-05-16 16:42:17eli.benderskysetmessages: + msg243332
2015-05-16 16:26:21serhiy.storchakasetfiles: + doctype-remove.v4.patch

messages: + msg243330
2015-05-16 16:12:40serhiy.storchakasetassignee: serhiy.storchaka
2015-04-01 06:38:15Arfreversetmessages: + msg239772
2015-03-31 22:33:09berker.peksagsetversions: - Python 3.4
nosy: + berker.peksag

messages: + msg239757

stage: patch review -> commit review
2015-03-31 07:10:13martin.pantersetfiles: + doctype-remove.v3.patch

messages: + msg239674
2015-03-31 06:28:25martin.pantersetfiles: + inherit-doctype.v2.patch

messages: + msg239673
2015-03-31 06:02:39martin.pantersetfiles: + doctype-remove.v2.patch

messages: + msg239671
2015-03-21 10:23:02serhiy.storchakasetversions: - Python 3.3
nosy: + serhiy.storchaka

messages: + msg238783

stage: patch review
2014-12-18 07:37:58Arfreversetmessages: + msg232862
2014-12-18 06:42:51martin.pantersetfiles: + doctype-remove.patch

messages: + msg232860
2014-12-17 01:48:57martin.pantersetfiles: + inherit-doctype.patch
keywords: + patch
messages: + msg232791

versions: + Python 3.5
2014-03-20 01:27:50martin.pantersetmessages: + msg214165
versions: + Python 3.4
2013-11-01 00:46:52Arfreversetnosy: + Arfrever
2013-10-31 12:59:03eli.benderskysetmessages: + msg201812
2013-10-14 00:24:07ezio.melottisetnosy: + ezio.melotti
2013-10-06 22:08:03pitrousetnosy: + eli.bendersky
2013-10-06 06:42:30martin.pantercreate