New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rewrite the IO stack in C #48815
Comments
The write performance into text files is substantially slower (5x-8x) ------------test code follows ----------------------- import time lo, hi, step = 10**5, 10**6, 10**5 # writes increasingly more lines to a file
for N in range(lo, hi, step):
fp = open('foodata.txt', 'wt')
start = time.time()
for i in range( N ):
fp.write( '%s\n' % i)
fp.close()
stop = time.time()
print ( "%s\t%s" % (N, stop-start) ) |
This is expected: the I/O stack has been completely rewritten... in There is a project to rewrite it in C. It started at |
Well I would strongly dispute that anyone other than the developers "The net result of the 3.0 generalizations is that Python 3.0 runs the There is no indication of an order of magnitudes in read/write slowdown. http://bugs.python.org/issue4561 Java has had a hard time getting rid of the "it is very slow" stigma |
Hi Amaury,
Thanks for publicizing this. I'm a bit surprised by the adopted If you look at bufferedwriter2.patch in bpo-3476, I had rewritten |
You are certainly right, but the code io.py is already difficult to understand and |
We can't solve this for 3.0.1, downgrading to critical. |
The work to rewrite the IO stack in C will solve this problem as it will Amaury and I have been progressing a lot, the rewrite is now a real (actually, if I add an explicit .encode('utf8') call to the 2.x version |
Issue depends on bpo-4967 which blocks use of memoryview objects with the |
This is basically going to be the killer feature in 3.1 ;). Therefore,
|
After "rewrite the rest of StringIO in C", there's "sanitize the |
Oh, and "what to do of the now unused pure Python implementations in io.py"? |
I think we should just drop the Python implementations. There's no point Besides, if we don't backport IO in C, we can maintain them in the trunk. :) |
The test suite should be run against both implementations. That way The value in the Python implementation is manifold. For example:
|
On Thu, Feb 19, 2009 at 1:57 PM, Jean-Paul Calderone
We don't maintain any other features in two languages for those |
Surely the majority of the burden is imposed by the C implementation. I Or maybe none of them will care or object to the removal of the Python |
Hello JP,
Hmm, it depends. It's probably true in general, but I suspect a fair
Well, if it is part of the CPython source tree, we (CPython developers)
In any case, it must first be asked on python-dev. We're not gonna dump cheers Antoine. |
[Benjamin Peterson]
I disagree. I've found great value in keeping a pure python version Also, I've found that once the two are in-sync, keeping it that way In the heapqmodule, we do a little magic in the test suite to Raymond |
Hi Antoine,
Indeed, I'm sure a lot of work went into the Python implementation - and Basically, my point is that maintaining C and Python versions is |
Well said. |
+1 to setting it up so that unit tests are always run against both and |
If this is the way forward I recommend putting the pure Python versions (and perhaps it will have the side-effect of improving startup time, |
It seems the decision of Python-dev is to keep both implementations. |
The StringIO rewrite is finished now. |
Ok. I've split the Python io implementation into the _pyio module and |
Ok. I've fixed all the tests except |
What should we do about test_fileio, test_file and test_bufio? |
On Sun, Feb 22, 2009 at 1:50 PM, Antoine Pitrou <report@bugs.python.org> wrote:
I changed test_file and test_bufio to test the open() implementations |
There's also test_univnewlines, I think. |
Oh, and test_largefile and test_debussy as well :) Le dimanche 22 février 2009 à 23:00 +0000, Antoine Pitrou a écrit :
|
test_largefile is done. |
On Mon, Feb 23, 2009 at 2:26 PM, Antoine Pitrou <report@bugs.python.org> wrote:
Thanks.
No, I think it was just meant to be used when _pyio is the builtin |
We also have to figure out how to make the C IOBase a ABC, so people can |
Mmmh, I know absolutely nothing about the ABC implementation. |
I just took a quick look at Lib/abc.py and there's no way *I*'ll The only workable approach would be:
That is, do something like the following:
Which gives:
>>> f = open('foobar', 'wb', buffering=0)
>>> isinstance(f, RawIOBase)
True
>>> isinstance(f, IOBase)
True
>>> f = open('foobar', 'w')
>>> isinstance(f, IOBase)
True
>>> isinstance(f, TextIOBase)
True
>>> isinstance(f, RawIOBase)
False As you see, RawIOBase inherits both from IOBase (the ABC, for ABC-ness) Also, writing a Python implementation still inherits the >>> class S(RawIOBase):
... def close(self):
... print("closing")
...
>>> s = S()
>>> del s
closing
>>> Perhaps we could even do all this in Python in io.py? |
On Wed, Feb 25, 2009 at 10:15 AM, Antoine Pitrou <report@bugs.python.org> wrote:
I don't blame you for that. :)
I think this is the best solution. We could also just move the Python |
Ok, so the ABC stuff is done now.
|
I just fixed the last failing test_io. (I'm listing as dependencies issues we can close after the branch is |
Reviewers: , Description: Please review this at http://codereview.appspot.com/22061 Affected files: |
A couple of typos in the Python implementation. http://codereview.appspot.com/22061/diff/1/11 http://codereview.appspot.com/22061/diff/1/11#newcode266 http://codereview.appspot.com/22061/diff/1/11#newcode844 http://codereview.appspot.com/22061/diff/1/11#newcode963 http://codereview.appspot.com/22061/diff/1/11#newcode1728 http://codereview.appspot.com/22061/diff/1/11#newcode1784 |
2009/3/3 Daniel Diniz <report@bugs.python.org>:
Thanks for taking a look! Fixed these things in r70135.
Perhaps, but I think it duplicates too much of _reset_read_buf(). And
Yes, it's for side affect, but it needn't be in a variable. |
And the io-c branch has been merged in r70152. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: