In the documentation for xml.sax.xmlreader.InputSource objects
(section 8.12.4 of the Library Reference) we find that
users of InputSource objects should use the following
sequence to get their input data:

1. If the InputSource has a character stream, use that.
2. Otherwise, if the InputSource has a byte stream, use that.
3. Otherwise, open a URI connection to the system ID.

The parse() method of IncrementalParser skips step 1.

In addition, we need to add a method
getSourceEncoding() to the XMLReader interface;
if non-null, it will indicate to the parser that
the input is a byte stream in the given encoding.

The documentation should indicate what the parser
should do if the XML itself announces that its
encoding is something else. I propose that the parser should
be required to raise an error in that case.

See also #1483.

See also: #1483 and #2175.

Hmm. When getSourceEncoding() is None, there needs to be some
way for the parser to distinguish between the cases where it
is getting pre-decoded Unicode through a character stream,
or where it is getting a byte stream with an unspecified
encoding. In the latter case, it will have to look in the
XML for an encoding declaration, or use UTF-8 by default).

Note that expat only can handle the latter case.

So I think there are two possibilities:

1. Use a special value for getSourceEnconding(),
like "unicode", to indicate that this is a
unicode character stream and not a byte stream.

2. Provide yet another method in the XMLReader
interface: sourceIsCharacterStream(), returning
a bool.

There is a more drastic option:

3. Since expat doesn't support this stuff
anyway, and perhaps not too many people
have written parsers that do support it,
dumb down the InputSource interface.

Specifically, deprecate setCharacterStream(),
getCharacterStream(), setEncoding() and
getEncoding(), none of which are used by
expat. Parsers should read the XML from
the byte stream and use that to determine
the encoding.

That may upset some implementors of XML
libraries though. They would each have to go
to some trouble to provide their own
proprietary and possibly incompatible
mechanisms for this, if they need it.

Perhaps a compromise fourth path would
be to have subclasses of InputSource for
the two cases of character stream and
byte stream.

Subclass of XMLReader would be needed, not InputStream.

It's certainly arguable that the current behavior is a bug, though I
suspect it shouldn't be considered major since I've not seen any prior
complaints about this.

It should be easy to fix the bug you describe by taking the character
stream and encoding it before feeding it to the XML parser; Expat can
certainly be forced to take a known encoding, ignoring what's in the XML
declaration.

On the other hand, it's not at all clear that changing this is
worthwhile.  This API borrows quite literally from the Java SAX APIs;
perhaps this separation of the character stream from the byte stream
makes sense for some of the Java XML parsers, but I don't know that
there are any Python parsers that benefit from that separation.