I originally reported this as a bug in PyPy, but it turns out that CPython's C implementation (_elementtree) behaves differently than the pure Python version (b/c it sets specified_attributes = 1).

PyPy issue with example code: https://foss.heptapod.net/pypy/pypy/-/issues/3333

The patch looks right. I'm not sure if this can still be changed in Py3.8, though, since that has been around for quite a while now.

Admittedly, few people will disable the C accelerator module and thus whitness this issue, but for them, this is a breaking change, and some code might rely on the current behaviour. I have no way to tell how much, and whether it intentionally relies on it.

I'd definitely change this for 3.9 and later. Maybe for 3.8, but it's at least a bit of a risk, given that there will only be very few more minor releases for it, and given that this is how things have been working for years. So, rather not, unless there is a convincing argument for backporting the change.

specified_attributes = True is also set in xml.dom.expatbuilder.

That's good to know and should perhaps be addressed as well.

Should not it be set to true in the C implementation of ElementTree?

That would break existing code. Including mine.

I also think the current behaviour of the C implementation makes a lot more sense, especially as there is currently no way to request the alternative.

I think using specified_attributes=False as the default behaviour for both implementations is the best solution. But I certainly would not oppose adding e.g. a keyword argument to override the default behaviour for those who would prefer the alternative.

In general, since the C accelerator is enabled by default, and few people would consider disabling it explicitly, I generally consider the behaviour of the C implementation to be "right", if both implementations differ.

As a single data point, the reason why the difference was found in this case was differing behaviour in PyPy (which uses only the Python implementation). It was only later found to be a problem on the CPython side.

Changing the behaviour of the C implementation would certainly break a lot more code than changing the Python implementation.

Is there an owner for the XML module that can make a decision? The PR has a test that shows this fix brings the python implementation into sync with the C implementation, which is, unintuitively, the "reference implementation".

Changing the behaviour of the C implementation would
certainly break a lot more code than changing the Python
implementation.

+1 for changing only the Python implementation.

PyPy issue https://foss.heptapod.net/pypy/pypy/-/issues/3181 shows another problem with the pure-python ElementTree implementation, that again is not reflected in the C implementation. Is there a code owner for this stdlib module?

New changeset 1f43340 by Felix C. Stegerman in branch 'master':
bpo-42151: don't set specified_attributes=1 in pure Python ElementTree (GH-22987)
1f43340

@obfusk
Thank you Felix for reporting and contributing!