The patch removes the basestring type from Python 3.0. PyString and
PyUnicode are subclasses of PyBaseObject_Type. Each occurrence of
basestring was replaces with str, mostly isinstance(egg, basestring)
with a few exceptions. PyObject_TypeCheck(args, &PyBaseString_Type) is
replaced with a check for PyUnicode and PyString.

Thanks, evaluating!

I see 10 failing tests:

    test_ctypes test_email test_httplib test_inspect test_os test_re
    test_subprocess test_sys test_xml_etree test_xml_etree_c

test_ctypes: works for me
test_email: need some help from an email expoert
test_httplib: __file__ has a wrong type str8. I'm looking into it.
test_inspect: same issue as httplib
test_os: same issue
test_re: I had the failing test before my changes
File "Lib/test/test_re.py", line 622, in test_empty_array
ValueError: bad typecode (must be b, B, u, h, H, i, I, l, L, f or d)
test_subprocess: I don't understand why it fails. The traceback is
missing a line
test_sys: related to __file__
test_xml_etree / test_xml_etree_c: a str8 / io error that may be related
to __file__

On 10/10/07, Christian Heimes <report@bugs.python.org> wrote:
>
> Christian Heimes added the comment:
>
> test_ctypes: works for me

Did you svn up, make clean and rebuild?

> test_email: need some help from an email expoert

Which test is failing?

> test_httplib: __file__ has a wrong type str8. I'm looking into it.

Yes, __file__ always has that type. Fixing it is messy because it
requires using the default filesystem encoding. Can you try that as a
separate patch?

> test_inspect: same issue as httplib
> test_os: same issue
> test_re: I had the failing test before my changes

But it passes for me.

> File "Lib/test/test_re.py", line 622, in test_empty_array
> ValueError: bad typecode (must be b, B, u, h, H, i, I, l, L, f or d)

Hm. It passes for me.

> test_subprocess: I don't understand why it fails. The traceback is
> missing a line
> test_sys: related to __file__
> test_xml_etree / test_xml_etree_c: a str8 / io error that may be related
> to __file__

Thanks for looking into these!!

Guido van Rossum wrote:
> Did you svn up, make clean and rebuild?

The ctypes package didn't change since my last rebuild an hour ago. I'm
on Linux (Ubuntu i386)
> 
>> test_email: need some help from an email expoert
> 
> Which test is failing?

test_decoded_generator()
The generator tries to print a str8 to a text file.

> Yes, __file__ always has that type. Fixing it is messy because it
> requires using the default filesystem encoding. Can you try that as a
> separate patch?

I'm already working on it. Can I introduce a new function
_PyUnicode_AsDefaultFSEncodedString that encodes unicode using
Py_FileSystemDefaultEncoding or UTF-8?

>> File "Lib/test/test_re.py", line 622, in test_empty_array
>> ValueError: bad typecode (must be b, B, u, h, H, i, I, l, L, f or d)
> 
> Hm. It passes for me.

I'm going to look into the issue later.

On 10/10/07, Christian Heimes <report@bugs.python.org> wrote:
>
> Christian Heimes added the comment:
>
> Guido van Rossum wrote:
> > Did you svn up, make clean and rebuild?
>
> The ctypes package didn't change since my last rebuild an hour ago. I'm
> on Linux (Ubuntu i386)

Odd. I'll investigate when I have more time.

> >> test_email: need some help from an email expoert
> >
> > Which test is failing?
>
> test_decoded_generator()
> The generator tries to print a str8 to a text file.

Thought so. I have a tentative fix that I want approved by Barry
Warsaw before checking; you can see if it works for you too:

--- Lib/email/generator.py      (revision 58412)
+++ Lib/email/generator.py      (working copy)
@@ -288,7 +288,7 @@
         for part in msg.walk():
             maintype = part.get_content_maintype()
             if maintype == 'text':
-                print(part.get_payload(decode=True), file=self)
+                print(part.get_payload(decode=False), file=self)
             elif maintype == 'multipart':
                 # Just skip this
                 pass

> > Yes, __file__ always has that type. Fixing it is messy because it
> > requires using the default filesystem encoding. Can you try that as a
> > separate patch?
>
> I'm already working on it. Can I introduce a new function
> _PyUnicode_AsDefaultFSEncodedString that encodes unicode using
> Py_FileSystemDefaultEncoding or UTF-8?

That's a rather long name... I don't think it needs a leading
underscore. How about

PyUnicode_AsFSString()?

Here is an updated patch which applies cleanly and fixes some additional
unit tests and removes one that doesn't make sense any more (re.compile
doesn't accept bytes).

The unit tests profile, cProfile and doctest fail w/ and w/o this patch.
They seem to suffer from the latest changes of our previous patch and
additional calls to utf_8_decode().

Hm? This is a one-word patch to email/generator.py.

On 10/15/07, Christian Heimes <report@bugs.python.org> wrote:
>
> Christian Heimes added the comment:
>
> Here is an updated patch which applies cleanly and fixes some additional
> unit tests and removes one that doesn't make sense any more (re.compile
> doesn't accept bytes).
>
> The unit tests profile, cProfile and doctest fail w/ and w/o this patch.
> They seem to suffer from the latest changes of our previous patch and
> additional calls to utf_8_decode().

> Hm? This is a one-word patch to email/generator.py.

Yes, I already noticed it and I'm creating a new patch now. I saw your
fix for the email generator problem in the bug report and wanted to add
it to my patch. I accidentally replaced the patch with the one liner.

Here is the new patch

> The unit tests profile, cProfile and doctest fail w/ and w/o this patch.
> They seem to suffer from the latest changes of our previous patch and
> additional calls to utf_8_decode().

Any details on those? They don't fail for me.

I'll check this in as soon as there's agreement on the list about this.

Not that I expect disagreement, but I just realized it was never brought
up and it isn't in PEP 3137 (yet).

> Any details on those? They don't fail for me.

Here you are.

$ ./python Lib/test/test_cProfile.py

         121 function calls (101 primitive calls) in 1.000 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    1.000    1.000 <string>:1(<module>)
        8    0.064    0.008    0.080    0.010
test_cProfile.py:103(subhelper)
       28    0.028    0.001    0.028    0.001
test_cProfile.py:115(__getattr__)
        1    0.270    0.270    1.000    1.000 test_cProfile.py:30(testfunc)
     23/3    0.150    0.007    0.170    0.057 test_cProfile.py:40(factorial)
       20    0.020    0.001    0.020    0.001 test_cProfile.py:53(mul)
        2    0.040    0.020    0.600    0.300 test_cProfile.py:60(helper)
        4    0.116    0.029    0.120    0.030 test_cProfile.py:78(helper1)
        2    0.000    0.000    0.140    0.070
test_cProfile.py:89(helper2_indirect)
        8    0.312    0.039    0.400    0.050 test_cProfile.py:93(helper2)
        1    0.000    0.000    0.000    0.000 utf_8.py:15(decode)
        1    0.000    0.000    0.000    0.000 {_codecs.utf_8_decode}
        1    0.000    0.000    1.000    1.000 {exec}
       12    0.000    0.000    0.012    0.001 {hasattr}
        4    0.000    0.000    0.000    0.000 {method 'append' of 'list'
objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}
        4    0.000    0.000    0.000    0.000 {sys.exc_info}

   Ordered by: standard name

Function                                          called...
                                                      ncalls  tottime
cumtime
<string>:1(<module>)                              ->       1    0.270
 1.000  test_cProfile.py:30(testfunc)
test_cProfile.py:103(subhelper)                   ->      16    0.016
 0.016  test_cProfile.py:115(__getattr__)
test_cProfile.py:115(__getattr__)                 ->
test_cProfile.py:30(testfunc)                     ->       1    0.014
 0.130  test_cProfile.py:40(factorial)
                                                           2    0.040
 0.600  test_cProfile.py:60(helper)
test_cProfile.py:40(factorial)                    ->    20/3    0.130
 0.147  test_cProfile.py:40(factorial)
                                                          20    0.020
 0.020  test_cProfile.py:53(mul)
test_cProfile.py:53(mul)                          ->
test_cProfile.py:60(helper)                       ->       4    0.116
 0.120  test_cProfile.py:78(helper1)
                                                           2    0.000
 0.140  test_cProfile.py:89(helper2_indirect)
                                                           6    0.234
 0.300  test_cProfile.py:93(helper2)
test_cProfile.py:78(helper1)                      ->       4    0.000
 0.004  {hasattr}
                                                           4    0.000
 0.000  {method 'append' of 'list' objects}
                                                           4    0.000
 0.000  {sys.exc_info}
test_cProfile.py:89(helper2_indirect)             ->       2    0.006
 0.040  test_cProfile.py:40(factorial)
                                                           2    0.078
 0.100  test_cProfile.py:93(helper2)
test_cProfile.py:93(helper2)                      ->       8    0.064
 0.080  test_cProfile.py:103(subhelper)
                                                           8    0.000
 0.008  {hasattr}
utf_8.py:15(decode)                               ->       1    0.000
 0.000  {_codecs.utf_8_decode}
{_codecs.utf_8_decode}                            ->
{exec}                                            ->       1    0.000
 1.000  <string>:1(<module>)
                                                           1    0.000
 0.000  utf_8.py:15(decode)
{hasattr}                                         ->      12    0.012
 0.012  test_cProfile.py:115(__getattr__)
{method 'append' of 'list' objects}               ->
{method 'disable' of '_lsprof.Profiler' objects}  ->
{sys.exc_info}                                    ->

   Ordered by: standard name

Function                                          was called by...
                                                      ncalls  tottime
cumtime
<string>:1(<module>)                              <-       1    0.000
 1.000  {exec}
test_cProfile.py:103(subhelper)                   <-       8    0.064
 0.080  test_cProfile.py:93(helper2)
test_cProfile.py:115(__getattr__)                 <-      16    0.016
 0.016  test_cProfile.py:103(subhelper)
                                                          12    0.012
 0.012  {hasattr}
test_cProfile.py:30(testfunc)                     <-       1    0.270
 1.000  <string>:1(<module>)
test_cProfile.py:40(factorial)                    <-       1    0.014
 0.130  test_cProfile.py:30(testfunc)
                                                        20/3    0.130
 0.147  test_cProfile.py:40(factorial)
                                                           2    0.006
 0.040  test_cProfile.py:89(helper2_indirect)
test_cProfile.py:53(mul)                          <-      20    0.020
 0.020  test_cProfile.py:40(factorial)
test_cProfile.py:60(helper)                       <-       2    0.040
 0.600  test_cProfile.py:30(testfunc)
test_cProfile.py:78(helper1)                      <-       4    0.116
 0.120  test_cProfile.py:60(helper)
test_cProfile.py:89(helper2_indirect)             <-       2    0.000
 0.140  test_cProfile.py:60(helper)
test_cProfile.py:93(helper2)                      <-       6    0.234
 0.300  test_cProfile.py:60(helper)
                                                           2    0.078
 0.100  test_cProfile.py:89(helper2_indirect)
utf_8.py:15(decode)                               <-       1    0.000
 0.000  {exec}
{_codecs.utf_8_decode}                            <-       1    0.000
 0.000  utf_8.py:15(decode)
{exec}                                            <-
{hasattr}                                         <-       4    0.000
 0.004  test_cProfile.py:78(helper1)
                                                           8    0.000
 0.008  test_cProfile.py:93(helper2)
{method 'append' of 'list' objects}               <-       4    0.000
 0.000  test_cProfile.py:78(helper1)
{method 'disable' of '_lsprof.Profiler' objects}  <-
{sys.exc_info}                                    <-       4    0.000
 0.000  test_cProfile.py:78(helper1)

####################################
$ ./python Lib/test/test_doctest.py

doctest (doctest) ... 66 tests with zero failures
**********************************************************************
File "/home/heimes/dev/python/py3k/Lib/test/test_doctest.py", line 1570,
in test.test_doctest.test_debug
Failed example:
    try: doctest.debug_src(s)
    finally: sys.stdin = real_stdin
Expected:
    > <string>(1)<module>()
    (Pdb) next
    12
    --Return--
    > <string>(1)<module>()->None
    (Pdb) print(x)
    12
    (Pdb) continue
Got:
    > /home/heimes/dev/python/py3k/Lib/encodings/utf_8.py(16)decode()
    -> return codecs.utf_8_decode(input, errors, True)
    (Pdb) next
    --Return--
    >
/home/heimes/dev/python/py3k/Lib/encodings/utf_8.py(16)decode()->('<string>',
8)
    -> return codecs.utf_8_decode(input, errors, True)
    (Pdb) print(x)
    *** NameError: NameError("name 'x' is not defined",)
    (Pdb) continue
    12
**********************************************************************
1 items had failures:
   1 of   4 in test.test_doctest.test_debug
***Test Failed*** 1 failures.
Traceback (most recent call last):
  File "Lib/test/test_doctest.py", line 2422, in <module>
    test_main()
  File "Lib/test/test_doctest.py", line 2406, in test_main
    test_support.run_doctest(test_doctest, verbosity=True)
  File "/home/heimes/dev/python/py3k/Lib/test/test_support.py", line
569, in run_doctest
    raise TestFailed("%d of %d doctests failed" % (f, t))
test.test_support.TestFailed: 1 of 414 doctests failed

####################################
$ ./python Lib/test/test_email.py

Traceback (most recent call last):
  File "Lib/test/test_email.py", line 13, in <module>
    test_main()
  File "Lib/test/test_email.py", line 10, in test_main
    test_support.run_unittest(suite())
  File "/home/heimes/dev/python/py3k/Lib/test/test_support.py", line
541, in run_unittest
    _run_suite(suite)
  File "/home/heimes/dev/python/py3k/Lib/test/test_support.py", line
524, in _run_suite
    raise TestFailed(err)
test.test_support.TestFailed: Traceback (most recent call last):
  File "/home/heimes/dev/python/py3k/Lib/email/test/test_email.py", line
1445, in test_same_boundary_inner_outer
    msg = self._msgobj('msg_15.txt')
  File "/home/heimes/dev/python/py3k/Lib/email/test/test_email.py", line
67, in _msgobj
    return email.message_from_file(fp)
  File "/home/heimes/dev/python/py3k/Lib/email/__init__.py", line 46, in
message_from_file
    return Parser(*args, **kws).parse(fp)
  File "/home/heimes/dev/python/py3k/Lib/email/parser.py", line 68, in parse
    data = fp.read(8192)
  File "/home/heimes/dev/python/py3k/Lib/io.py", line 1240, in read
    readahead, pending = self._read_chunk()
  File "/home/heimes/dev/python/py3k/Lib/io.py", line 1136, in _read_chunk
    pending = self._decoder.decode(readahead, not readahead)
  File "/home/heimes/dev/python/py3k/Lib/codecs.py", line 291, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbe in position 86:
unexpected code byte

BTW we need a 2to3 fixer for this.  Should be trivial -- just replace
*all* occurrences of basestring with str.

Even before this patch, the re module doesn't work very well on byte
strings. IMO this should be fixed.  I've filed a separate bug to remind
us: bug 1282.

Guido van Rossum wrote:
> BTW we need a 2to3 fixer for this.  Should be trivial -- just replace
> *all* occurrences of basestring with str.

I believe you that it's trivial for *you* but I've never dealt with the
fixers or the grammar. Fortunately for me I was able to copy the fixer
for standarderror. It toke just some minor tweaks :)

Let's see if the mail interface can handle attachments.

Committed revision 58495.

Thanks Christian!!!

2007/10/15, Christian Heimes <report@bugs.python.org>:
>
> Christian Heimes added the comment:
>
> Guido van Rossum wrote:
> > BTW we need a 2to3 fixer for this.  Should be trivial -- just replace
> > *all* occurrences of basestring with str.
>
> I believe you that it's trivial for *you* but I've never dealt with the
> fixers or the grammar. Fortunately for me I was able to copy the fixer
> for standarderror. It toke just some minor tweaks :)
>
> Let's see if the mail interface can handle attachments.

It did. :-) I renamed it to fix_basestring and submitted it.  See:

Committed revision 58644.