This is the implementation of codecs.Streamwriter.writelines for all
Python versions that I've checked:

    def writelines(self, list):

    """ Writes the concatenated list of strings to the stream
        using .write().
    """
    self.write(''.join(list))

This may be a problem if the 'list' parameter is a generator. The
generator may be returning millions of values, which the join will
concatenate in memory. It can surprise the programmer with large
memory use. I think join should only be used when you know the size of
your input, and this method does not know this. I think the safe
implementation of this method would be:

    def writelines(self, list):

    """ Writes the concatenated list of strings to the stream
        using .write().
    """
    write = self.write
    for value in list:
        write(value)

If a caller knows that it's input list would use a reasonable amount
of memory, it can get the same functionality as before by doing
stream.write(''.join(list)).

When fixing this, note that the builtin name "list" should not be
overwritten by the argument name.

For the common case where list is in fact a sequence of strings, the
used implementation is a lot faster and more efficient than the one you
propose.

Note that the method doesn't pretend to support generators for the list
argument, so adding support for iterators would be a feature request,
not a bug report.

Furthermore, the StreamWriter method can easily be overridden by the
codec's own implementation (the version in codecs.py is the base method
defining the interface and providing a default implementation), so
adding support to the base method will not necessarily mean that all
codecs then support iterators as parameter to .writelines().

IMHO, it's better to use the .write() method in a for-loop of your
application if you want to support iterators that generate a lot of output.

In Python's file protocol, readlines and writelines is a protocol for
iterating over a file. In Python's file protocol, if one doesn't want
to iterate over the file, one calls read() with no argument in order
to read the whole file in, or one calls write() with the complete
contents you want to write.

If writelines is using join, then if one passes an iterator as the
parameter to writelines, it will not iteratively write to the file, it
will accumulate everything in memory until the iterator raises
StopIteration, and then write to the file. So, if one is tailing the
output file, one is not going to see anything in the file until the
end, instead of iteratively seeing content. So, it's breaking the
promise of the file protocol's writelines meaning iteratively write.

I think following the protocol is more important than performance. If
the application is having performance problems, it's up to the
application to buffer the data in memory and make a single write call.

However, here is an alternative implementation that is slightly more
complicated, but possibly has better performance for the passed-a-list
case. It covers three cases:

1. Passed an empty sequence; do not call self.write at all.
2. Passed a sequence with a length. That implies that all the data is
   available immediately, so one can concantenate and write with one
   self.write call.
3. Passed a sequence with no length. That implies that all the data
   is not available immediately, so iteratively write it.

    def writelines(self, sequence):

    """ Writes the sequence of strings to the stream
        using .write().
    """
    try:
        sequence_len = len(sequence)
    except TypeError:
        write = self.write
        for value in sequence:
            write(value)
        return
    if sequence_len:
        self.write(''.join(sequence))

I'm not sure which is better. But one last point is that Python is
moving more in the direction of using iterators; e.g., in Py3K,
replacing dict's keys, values, and items with the implementation of
iterkeys, itervalues, and iteritems.

Let me give an example of why it's important that writelines
iteratively writes. For:

rows = (line[:-1].split('\t') for line in in_file)
projected = (keep_fields(row, 0, 3, 7) for row in rows)
filtered = (row for row in projected if row[2]=='1')
out_file.writelines('\t'.join(row)+'\n' for row in filtered)

For a large input file, for a regular out_file object, this will work.
For a codecs.StreamWriter wrapped out_file object, this won't work,
because it's not following the file protocol that writelines should
iteratively write.

On 2009-03-10 16:36, Daniel Lescohier wrote:

Daniel Lescohier <daniel.lescohier@cbs.com> added the comment:

Let me give an example of why it's important that writelines
iteratively writes. For:

rows = (line[:-1].split('\t') for line in in_file)
projected = (keep_fields(row, 0, 3, 7) for row in rows)
filtered = (row for row in projected if row[2]=='1')
out_file.writelines('\t'.join(row)+'\n' for row in filtered)

For a large input file, for a regular out_file object, this will work.
For a codecs.StreamWriter wrapped out_file object, this won't work,
because it's not following the file protocol that writelines should
iteratively write.

Of course, it's possible to have a generator producing lots of data,
but that does not warrant making most uses of that method slow.

If you'd like to see such support in .writelines(), please provide
an implementation that follows the approach of the file object
implementation in Python 2.x: it writes the lines in chunks of
1000 lines each, if it finds that the input object is not a sequence
(actually, it's stricter than that for some: it requires a Python
list).

The standard case of passing a list of strings to that method
should not get slower because of this.

BTW: I am not aware of any .writelines() file protocol. If there
were such a protocol, .readlines() would also have to return
an iterator (which it doesn't). The idea behind .writelines()
is to be able to write back a list generated with .readlines().

Thanks,

Marc-Andre Lemburg
eGenix.com

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

OK, I think I see where I went wrong in my perceptions of the file
protocol. I thought that readlines() returned an iterator, not a
list, but I see in the library reference manual on File Objects that
it returns a list. I think I got confused because there is no
equivalent of __iter__ for writing to streams. For input, I'm always
using 'for line in file_object' (in other words,
file_object.__iter__), so I had assumed that writelines was the mirror
image of that, because I never use the readlines method. Then, in my
mind, readlines became the mirror image of writelines, which I had
assumed took an iterator, so I assumed that readlines returned an
iterator. I wonder if this perception problem is common or not.

So, the StreamWriter interface matches the file protocol; readlines()
and writelines() deal with lists. There shouldn't be any change to
it, because it follows the protocol.

Then, the example I wrote would be instead:

rows = (line[:-1].split('\t') for line in in_file)
projected = (keep_fields(row, 0, 3, 7) for row in rows)
filtered = (row for row in projected if row[2]=='1')
formatted = (u'\t'.join(row)+'\n' for row in filtered)
write = out_file.write
for line in formatted:
    write(line)

I think it's correct that the file object write C code only does
1000-line chunks for sequences that have a defined length: if it has a
defined length, then that implies that the data exists now, and can be
concatenated and written now. Something without a defined length may
be a generator with items arriving later.

I find this to be a confusing issue.

1. The arg for .join is any iterable of strings, not just a list. (I think Daniel got that.) Since codecs.py still has "self.write(''.join(list))":
   a) calling the arg 'list' is doubly wrong.
   b) there is no iterable feature to add, at least not to the base class.

2. The codecs.writelines entry says "Writes the concatenated list of strings to the stream (possibly by reusing the write() method)." For the base class, that is overly restrictive, but I gather that Marc-Andre does not want to require that all subclasses be as relaxed. I presume there must be a reason. That means that the feature request of doing so is rejected by him. In that respect, this should be closed unless other developers strongly disagree.

3. The iobase.writelines entry says "Write a list of lines to the stream." For StreamIO, iter(somelist) works just as well. If that applies to .writelines in general, the doc should say 'iterable of lines'. (Antoine, Benjamin?)

If io.writelines does the same (concatenate all together), it should say so, but one comment suggests it does not

4. If an iterable produces millions of lines, it does not matter if it is a list, generator, or any other iterable. So the title does not point to the real problem, if there is one.

On 25.03.2013 23:11, Terry J. Reedy wrote:

2. The codecs.writelines entry says "Writes the concatenated list of strings to the stream (possibly by reusing the write() method)." For the base class, that is overly restrictive, but I gather that Marc-Andre does not want to require that all subclasses be as relaxed. I presume there must be a reason. That means that the feature request of doing so is rejected by him. In that respect, this should be closed unless other developers strongly disagree.

The doc string of the base class merely describes what the
.writelines() method will be doing. It's not meant as a requirement.

In one of the previous messages, I've already outlined a workable
approach to solve the case that the original request was addressing.
There isn't much point in making the common case slower just
to address a theoretical corner case.

Since no patch has since been submitted, I'd suggest to close the
ticket as won't fix or out of date.

Agreed. Modern consumers can use the io module.