The write performance into text files is substantially slower (5x-8x)
than that of python 2.5. This makes python 3.0 unsuited to any
application that needs to write larger amounts of data.

------------test code follows -----------------------

import time

lo, hi, step = 10**5, 10**6, 10**5

# writes increasingly more lines to a file
for N in range(lo, hi, step):
    fp = open('foodata.txt', 'wt')
    start = time.time()
    for i in range( N ):
        fp.write( '%s\n' % i)
    fp.close()
    stop = time.time()
    print ( "%s\t%s" % (N, stop-start) )

This is expected: the I/O stack has been completely rewritten... in 
almost pure-python code.

There is a project to rewrite it in C. It started at
http://svn.python.org/view/sandbox/trunk/io-c/

Well I would strongly dispute that anyone other than the developers
expected this. The release documentation states:

"The net result of the 3.0 generalizations is that Python 3.0 runs the
pystone benchmark around 10% slower than Python 2.5."

There is no indication of an order of magnitudes in read/write slowdown.
I believe that this issue is extremely serious! IO is an essential part
of a program, and today we live in the world of gigabytes of data. I am
reading reports of even more severe io slowdowns than what I saw:

http://bugs.python.org/issue4561

Java has had a hard time getting rid of the "it is very slow" stigma
even after getting a JIT compiler, so there is a danger there for a
lasting negative impression.

Hi Amaury,

> There is a project to rewrite it in C

Thanks for publicizing this. I'm a bit surprised by the adopted
approach. It seems you are merely translating the Python code into C. I
think the proper approach for the buffered IO classes would be to use a
fixed-size buffer which never gets reallocated.

If you look at bufferedwriter2.patch in #3476, I had rewritten
BufferedWriter using a fixed-size buffer (although not for performance
reasons), I think it would be a good starting point for a C implementation.

For more bug reports see #4533 and #4561.

I suggest we close this bug report as duplicate and keep the discussion
in #4561.

> I'm a bit surprised by the adopted approach. 
> It seems you are merely translating the Python code into C. 
> I think the proper approach for the buffered IO classes would be 
> to use a fixed-size buffer which never gets reallocated.

You are certainly right, but the code io.py is already difficult to understand and 
maintain; the corresponding C code adds one level of complexity;
had I changed the buffering strategy at the same time, it would have been impossible 
to ensure a correct implementation.
Now that my C implementation of the Buffered classes seems correct (all tests pass, 
except a few about destructors) we could try alternative approaches.

We can't solve this for 3.0.1, downgrading to critical.

The work to rewrite the IO stack in C will solve this problem as it will
probably solve most performance-related IO problems in py3k.

Amaury and I have been progressing a lot, the rewrite is now a real
branch in SVN at branches/io-c/. On this very issue, it is only 30%
slower than 2.x, which is quite good given the layered nature of the IO
stack and the fact that text IO does a lot more than in 2.x (it
translates newlines and encodes the text).

(actually, if I add an explicit .encode('utf8') call to the 2.x version
of the script, it becomes slower than our io-c rewrite)

Issue depends on #4967 which blocks use of memoryview objects with the
_ssl module.

This is basically going to be the killer feature in 3.1 ;). Therefore,
these are steps I think we need before we can merge the branch:

- Fix the dependencies. (#4967)
- Resolve all outstanding issues with the IO lib on the io-c branch.
- Rewrite the rest of StringIO in C?
- Anything else I forgot?

After "rewrite the rest of StringIO in C", there's "sanitize the
destructor behaviour of IOBase (if at all possible)".

Oh, and "what to do of the now unused pure Python implementations in io.py"?
Easiest would be to dump them, as they will probably get hopelessly out
of sync, but perhaps there's some genuine portability/educational
advantage to keep them?

I think we should just drop the Python implementations. There's no point
in trying to keep two implementations around.

Besides, if we don't backport IO in C, we can maintain them in the trunk. :)

> Oh, and "what to do of the now unused pure Python implementations in
io.py"?  Easiest would be to dump them, as they will probably get
hopelessly out of sync, but perhaps there's some genuine
portability/educational advantage to keep them?

The test suite should be run against both implementations.  That way
tested behavior will always be the same for both.  And all of its
behavior is tested, right? ;)

The value in the Python implementation is manifold.  For example:

  * It eases testing of new features/techniques.  Rather than going
straight to the C version when someone has an idea for a feature, it can
be implemented and tried out in Python.  If it's cool, then the extra
effort of porting to C can be undertaken.
  * It helps other Python implementations immensely.  PyPy, IronPython,
and Jython are all going to have to provide this library eventually (one
supposes).  Forcing them each to re-implement it will mean it will be
that much longer before they support it.

On Thu, Feb 19, 2009 at 1:57 PM, Jean-Paul Calderone
<report@bugs.python.org> wrote:
>
> Jean-Paul Calderone <exarkun@divmod.com> added the comment:
>
>> Oh, and "what to do of the now unused pure Python implementations in
> io.py"?  Easiest would be to dump them, as they will probably get
> hopelessly out of sync, but perhaps there's some genuine
> portability/educational advantage to keep them?
>
> The test suite should be run against both implementations.  That way
> tested behavior will always be the same for both.  And all of its
> behavior is tested, right? ;)
>
> The value in the Python implementation is manifold.  For example:
>
>  * It eases testing of new features/techniques.  Rather than going
> straight to the C version when someone has an idea for a feature, it can
> be implemented and tried out in Python.  If it's cool, then the extra
> effort of porting to C can be undertaken.
>  * It helps other Python implementations immensely.  PyPy, IronPython,
> and Jython are all going to have to provide this library eventually (one
> supposes).  Forcing them each to re-implement it will mean it will be
> that much longer before they support it.

We don't maintain any other features in two languages for those
purposes. IMO, it will just be more of a burden to fix bugs in two
different places as compared to the advantages you mention.

> We don't maintain any other features in two languages for those
purposes. IMO, it will just be more of a burden to fix bugs in two
different places as compared to the advantages you mention.

Surely the majority of the burden is imposed by the C implementation.  I
expect that 90% of the time spent fixing bugs will be spent fixing them
in C.  So for only a slightly increased maintenance cost, a massive
advantage is gained for other Python implementations.  If the general
well-being and popularity of Python isn't a concern of CPython
developers, then perhaps the benefits can still be preserved at minimal
cost to the CPython developers by letting some Jython, IronPython, or
PyPy developers maintain the Python implementation of the io library in
the CPython source tree (rather than making them copy it elsewhere where
it will more frequently get out of sync, and where
Jython/IronPython/PyPy might waste effort in duplicating maintenance).

Or maybe none of them will care or object to the removal of the Python
version from CPython.  It might at least be worth asking first, though.

Hello JP,

> Surely the majority of the burden is imposed by the C implementation.  I
> expect that 90% of the time spent fixing bugs will be spent fixing them
> in C.

Hmm, it depends. It's probably true in general, but I suspect a fair
amount of work also went into getting the Python implementation correct,
since there are things in there that are tricky regardless of the
implementation language (I'm especially thinking of the TextIOWrapper
seek() and tell() methods).
(and there are still bugs in the Python implementation btw.)

> If the general
> well-being and popularity of Python isn't a concern of CPython
> developers, then perhaps the benefits can still be preserved at minimal
> cost to the CPython developers by letting some Jython, IronPython, or
> PyPy developers maintain the Python implementation of the io library in
> the CPython source tree

Well, if it is part of the CPython source tree, we (CPython developers)
can't realistically ignore it by saying it's someone else's job.

> Or maybe none of them will care or object to the removal of the Python
> version from CPython.  It might at least be worth asking first, though.

In any case, it must first be asked on python-dev. We're not gonna dump
the code without telling anybody anything :)

cheers

Antoine.

[Benjamin Peterson]
> I think we should just drop the Python implementations. There's no point
> in trying to keep two implementations around.

I disagree.  I've found great value in keeping a pure python version
around for things I've converted to C.   The former serves as
documentation, as a tool for other implementations (like PyPy
IronPython, and Jython), and as a precise spec.  The latter case
is especially valuable (otherwise, the spec becomes whatever
CPython happens to do). 

Also, I've found that once the two are in-sync, keeping it that way
isn't hard.  And, there effort for keeping them in-sync is a good
way to find bugs.

In the heapqmodule, we do a little magic in the test suite to
make sure the tests are run against both.  It's not hard.


Raymond

Hi Antoine,

> > Surely the majority of the burden is imposed by the C
implementation.  I expect that 90% of the time spent fixing bugs will be
spent fixing them in C.

> Hmm, it depends. It's probably true in general, but I suspect a fair
amount of work also went into getting the Python implementation correct,
since there are things in there that are tricky regardless of the
implementation language (I'm especially thinking of the TextOWrapper
seek() and tell() methods). (and there are still bugs in the Python
implementation btw.)

Indeed, I'm sure a lot of work went into the Python implementation - and
hopefully that work *saved* a huge amount of work when doing the C
implementation.  That's why people prototype things in Python, right? :)
 So it seems to me that keeping the Python implementation is useful for
CPython, since if it made working on the C implementation easier in the
past, it will probably do so again in the future.

Basically, my point is that maintaining C and Python versions is
*cheaper* than just maintaining the C version alone.  The stuff I said
about other VMs is true too, but it doesn't seem like anyone here is
going to be convinced by it ;)  (and I haven't spoked to any developers
for other VMs about whether they really want it, anyway).

> Basically, my point is that maintaining C and Python 
> versions is *cheaper* than just maintaining the C 
> version alone.

Well said.

+1 to setting it up so that unit tests are always run against both and 
keeping both.

> +1 to setting it up so that unit tests are always run against both and 
> keeping both.

If this is the way forward I recommend putting the pure Python versions
into a separate module, eg pyio.py (although the name is not very
elegant). It will make the separation clean and obvious.

(and perhaps it will have the side-effect of improving startup time,
although I'm not really worried about this)

It seems the decision of Python-dev is to keep both implementations.
We'll stuff the python one in _pyio and rewrite the tests to test both.
I'll see if I can get to this this weekend.

The StringIO rewrite is finished now.

Ok. I've split the Python io implementation into the _pyio module and
rewritten the tests. All the C ones are passing, but some Python
implementation ones are failing.

Ok. I've fixed all the tests except
PyBufferedRandomTest.testFlushAndPeek and the garbage collections ones.

What should we do about test_fileio, test_file and test_bufio?

On Sun, Feb 22, 2009 at 1:50 PM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> What should we do about test_fileio, test_file and test_bufio?

I changed test_file and test_bufio to test the open() implementations
of each library. test_fileio should be fine because the implementation
is the same for _pyio and io.

There's also test_univnewlines, I think.

Oh, and test_largefile and test_debussy as well :)

Le dimanche 22 février 2009 à 23:00 +0000, Antoine Pitrou a écrit :
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> There's also test_univnewlines, I think.

test_largefile is done.
One more question: what shall we do with _pyio.OpenWrapper?
Should it become the default exported "open" object?

On Mon, Feb 23, 2009 at 2:26 PM, Antoine Pitrou <report@bugs.python.org> wrote:
> test_largefile is done.

Thanks.

> One more question: what shall we do with _pyio.OpenWrapper?
> Should it become the default exported "open" object?

No, I think it was just meant to be used when _pyio is the builtin
open implementation.

We also have to figure out how to make the C IOBase a ABC, so people can
implement it.

> We also have to figure out how to make the C IOBase a ABC, so people can
> implement it.

Mmmh, I know absolutely nothing about the ABC implementation.

I just took a quick look at Lib/abc.py and there's no way *I*'ll
reimplement it in C :)

The only workable approach would be:
1. rename the current would-be ABCs (IOBase, RawIOBase, etc.) with a
leading underscore (_IOBase, _RawIOBase, etc.)
2. call abc.ABCMeta() with the right arguments to create heap-types 
derived from those base types
3. call XXXIOBase.register() with each of the concrete classes
(BufferedReader, etc.) to register them with the ABCs created in 2

That is, do something like the following:

>>> IOBase = abc.ABCMeta("IOBase", (_io.IOBase,), {})
>>> RawIOBase = type("RawIOBase", (_io.RawIOBase, IOBase), {})
>>> RawIOBase.register(_io.FileIO)
>>> TextIOBase = type("TextIOBase", (_io.TextIOBase, IOBase), {})
>>> TextIOBase.register(_io.TextIOWrapper)

Which gives:
>>> f = open('foobar', 'wb', buffering=0)
>>> isinstance(f, RawIOBase)
True
>>> isinstance(f, IOBase)
True
>>> f = open('foobar', 'w')
>>> isinstance(f, IOBase)
True
>>> isinstance(f, TextIOBase)
True
>>> isinstance(f, RawIOBase)
False


As you see, RawIOBase inherits both from IOBase (the ABC, for ABC-ness)
and _RawIOBase (the concrete non-ABC implementation). Implementation
classes like FileIO don't need to explicitly inherit the ABCs, only to
register with them.

Also, writing a Python implementation still inherits the
close-on-destroy behaviour:

>>> class S(RawIOBase):
...   def close(self):
...     print("closing")
... 
>>> s = S()
>>> del s
closing
>>> 

Perhaps we could even do all this in Python in io.py?

On Wed, Feb 25, 2009 at 10:15 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> I just took a quick look at Lib/abc.py and there's no way *I*'ll
> reimplement it in C :)

I don't blame you for that. :)

>
> The only workable approach would be:
> 1. rename the current would-be ABCs (IOBase, RawIOBase, etc.) with a
> leading underscore (_IOBase, _RawIOBase, etc.)
> 2. call abc.ABCMeta() with the right arguments to create heap-types
> derived from those base types
> 3. call XXXIOBase.register() with each of the concrete classes
> (BufferedReader, etc.) to register them with the ABCs created in 2

I think this is the best solution. We could also just move the Python
ABC's from _pyio to io.py and register() all the C IO classes, but
that would prevent the C implementation of IOBase from being used.

Ok, so the ABC stuff is done now.
Remaining:
- fix the test failures with the Python implementation
- the _ssl bug

I just fixed the last failing test_io.

(I'm listing as dependencies issues we can close after the branch is
merged.)

These StringIO bugs should be dealt with:

#5264
#5265
#5266

Reviewers: ,

Description:
The diff between the py3k and io-c branches, for whoever wants to review
it.

Please review this at http://codereview.appspot.com/22061

Affected files:
   Doc/library/io.rst
   Lib/_pyio.py
   Lib/importlib/__init__.py
   Lib/importlib/_bootstrap.py
   Lib/io.py
   Lib/test/test_bufio.py
   Lib/test/test_descr.py
   Lib/test/test_file.py
   Lib/test/test_fileio.py
   Lib/test/test_io.py
   Lib/test/test_largefile.py
   Lib/test/test_memoryio.py
   Lib/test/test_univnewlines.py
   Lib/test/test_uu.py
   Makefile.pre.in
   Modules/Setup.dist
   Modules/_bufferedio.c
   Modules/_bytesio.c
   Modules/_fileio.c
   Modules/_iobase.c
   Modules/_iomodule.h
   Modules/_stringio.c
   Modules/_textio.c
   Modules/io.c
   PC/VC6/pythoncore.dsp
   PC/config.c
   PCbuild/pythoncore.vcproj
   Python/pythonrun.c
   setup.py

A couple of typos in the Python implementation.

http://codereview.appspot.com/22061/diff/1/11
File Lib/_pyio.py (right):

http://codereview.appspot.com/22061/diff/1/11#newcode266
Line 266: fp is closed after the suite of the with statment is complete:
statment -> statement

http://codereview.appspot.com/22061/diff/1/11#newcode844
Line 844: self._reset_read_buf()
Setting "_read_buf" and "_read_pos" directly on __init__ may help
introspection tools.

http://codereview.appspot.com/22061/diff/1/11#newcode963
Line 963: DEAFULT_BUFFER_SIZE. If max_buffer_size is omitted, it
defaults to
DEAFULT_BUFFER_SIZE -> DEFAULT_BUFFER_SIZE

http://codereview.appspot.com/22061/diff/1/11#newcode1728
Line 1728: decoder = self._decoder or self._get_decoder()
'decoder' isn't used in this method, is this here for an useful
side-effect?

http://codereview.appspot.com/22061/diff/1/11#newcode1784
Line 1784: more_line = ''
This seems unused.

http://codereview.appspot.com/22061

2009/3/3 Daniel Diniz <report@bugs.python.org>:
> A couple of typos in the Python implementation.

Thanks for taking a look! Fixed these things in r70135.

> http://codereview.appspot.com/22061/diff/1/11#newcode844
> Line 844: self._reset_read_buf()
> Setting "_read_buf" and "_read_pos" directly on __init__ may help
> introspection tools.

Perhaps, but I think it duplicates too much of _reset_read_buf(). And
it wouldn't damage introspection, just static analysis.

> http://codereview.appspot.com/22061/diff/1/11#newcode1728
> Line 1728: decoder = self._decoder or self._get_decoder()
> 'decoder' isn't used in this method, is this here for an useful
> side-effect?

Yes, it's for side affect, but it needn't be in a variable.

And the io-c branch has been merged in r70152.