On Mon, Aug 18, 2008 at 1:51 PM, Antoine Pitrou <report@bugs.python.org>wrote:

>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> From the discussion on the python-3000, it looks like it would be nice
> if sax.parser handled both bytes and unicode streams.
>

> Edward, does your simple fix make sax.parser work entirely well with
> byte streams?

No. The sax.parser seems to have other problems.  Here is what I *think* I
know ;-)

1. A smallish .leo file (an xml file) containing a single non-ascii (utf-8)
encoded character appears to have been read correctly with Python 3.0.

2. A larger .leo file fails as follows (it's possible that the duplicate
error messages are a Leo problem):

Traceback (most recent call last):
Traceback (most recent call last):

  File "C:\leo.repo\leo-30\leo\core\leoFileCommands.py", line 1283, in
parse_leo_file
    parser.parse(theFile) # expat does not support parseString
  File "C:\leo.repo\leo-30\leo\core\leoFileCommands.py", line 1283, in
parse_leo_file
    parser.parse(theFile) # expat does not support parseString

  File "c:\python30\lib\xml\sax\expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "c:\python30\lib\xml\sax\expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)

  File "c:\python30\lib\xml\sax\xmlreader.py", line 121, in parse
    buffer = file.read(self._bufsize)
  File "c:\python30\lib\xml\sax\xmlreader.py", line 121, in parse
    buffer = file.read(self._bufsize)

  File "C:\Python30\lib\io.py", line 1670, in read
    eof = not self._read_chunk()
  File "C:\Python30\lib\io.py", line 1670, in read
    eof = not self._read_chunk()

  File "C:\Python30\lib\io.py", line 1499, in _read_chunk
    self._set_decoded_chars(self._decoder.decode(input_chunk, eof))
  File "C:\Python30\lib\io.py", line 1499, in _read_chunk
    self._set_decoded_chars(self._decoder.decode(input_chunk, eof))

  File "C:\Python30\lib\io.py", line 1236, in decode
    output = self.decoder.decode(input, final=final)
  File "C:\Python30\lib\io.py", line 1236, in decode
    output = self.decoder.decode(input, final=final)

  File "C:\Python30\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
  File "C:\Python30\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 74:
character maps to <undefined>
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 74:
character maps to <undefined>

The same calls to sax read the file correctly on Python 2.5.

It would be nice to have a message pinpoint the line and character offset of
the problem.

My vote would be for the code to work on both kinds of input streams. This
would save the users considerable confusion if sax does the (tricky)
conversions automatically.

Imo, now would be the most convenient time to attempt this--there is a
certain freedom in having everything be partially broken :-)

Edward
--------------------------------------------------------------------
Edward K. Ream email: edreamleo@gmail.com
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------