str.split returns a list, which is inefficient when you just want to process items one be one. You could emulate this with str.find and tracking indexes manually, but this should really be a builtin behavior.

The bytes (and bytearray?) version of this should generate memoryview's instead of new bytes objects to avoid a copy.

While not required, It'd be useful if the implementation of this pre-scanned the data internally so that the length of the generated sequence was known up front. This could imply an internal bitset of vector of split indices is kept for the life of the generator (implementation detail left up to the implementor) if scanning over the input data more than once is undesirable.

Start with a pure python proof of concept to work everywhere, then write a native version.

While not required, It'd be useful if the implementation of this pre-scanned the data internally so that the length of the generated sequence was known up front. This could imply an internal bitset of vector of split indices is kept for the life of the generator (implementation detail left up to the implementor) if scanning over the input data more than once is undesirable.

bytearray can be modified between iterations.

Indeed, a bytearray version would require the talked about but not implemented due to complexity (in PEP-3118) support for locking a buffer from other mutations. best concentrate on bytes then.

Do we have a memoryview equivalent for PyUnicode? If not, we should... (a separate enhancement request)

There is no string view that I know of. Interesting idea, though, thanks to the immutability of strings. Would much have to be different other than boundary checking and __hash__ (and hoping extension authors are changing things in-place)?

I say go ahead and open the issue for the idea.

Indeed, a bytearray version would require the talked about but not implemented due to complexity (in PEP-3118) support for locking a buffer from other mutations.

I rather think that a bytearray version can't pre-scan the data. Note that an array for pre-scanned result can be larger than input data (if we split into a large number of small items). Also note that iterative split useful when we do not want to process all input, but only several first items.

Actually I think that in most common cases non-iterative split will be faster than iterative one.

There is no string view that I know of. Interesting idea, though, thanks to the immutability of strings. Would much have to be different other than boundary checking and __hash__ (and hoping extension authors are changing things in-place)?

Objects/stringlib/unicode_format.h contains internal structure SubString which can be taken as a basis. But it is unlikely that it will be useful. All API which accept strings will require converting substring views to regular strings. And substring object can consume more memory than full string. This looks like a step backwards from PEP-393.

I personally would have changed both str.split and os.walk to return iterators in 3.0, like many other builtins. The rationale for os.walk continuing to produce a list is that there would be little time saving as the list is not *that* long and most uses look at all the items anyway. (See tracker issue.) This argument might be even stronger for str.split.

When I expressed my view about str.split (after 3.0, I think), Guido said that if people do not look at all the items, they typically need random access, and hence a list anyway, so why make them write list(xxx.split()) all the time?

I will also note that Guido has opposed string views and partial object views in general on the basis that a tiny view can keep a mega object alive, and that it is too much to ask people to keep track of when they should copy the view to release the underlying object.

Iterators also (should) keep the underlying object alive, but they are usually short-lived and not passed around hither and thither.

I personally would prefer that the API for this proposal simply be a new parameter (iter(able)=False/True). But that runs into 'argument values should not change the return type' (but lists and iterators are *both iterables*!). Instead there supposedly should be a new, almost identical function with a new, almost identical name. But that runs into 'we should not add builtins with a really good reason' (which I agree with) and more directly 'we should not repeat the confusing range/xrange mess' (which I also agree with). So the status quo is a Catch 22 situation with conflicting principles that produce paralysis. As I said, I prefer redefining the return as an iterable.

It'd perhaps have been better if things like memoryview were never exposed to the user at all as a distinct type and became an internal implementation detail behind PyBytes and PyUnicode objects (they could hold a reference to something else or collapse that down to their own copy on their own terms, up to the particulars of the Python VM).

Anyways, the above is getting off topic for this issue. I retract my memoryview suggestion; that belongs in its own issue.

An iterating version of str.split is indeed hard to add today IFF we are against a str.itersplit() method name or against an optional keyword only argument that'd cause split(iterator=True)'s return type to potentially be different.

-1 on os.walk returning an iterator. The API is already a bit challenging for some and our experience with itertools.groupby() is that returning an inner iterator can be very confusing.

Raymond: Is that for the wrong ticket, or was the message incorrect? :)

Alex, it was response to Terry's message: http://bugs.python.org/issue17343#msg183782

FWIW, I'm +1 on an iterator version of str.split().

I'm not sure yet that it would be worthwhile to propagate the idea to other string-like objects though.

If someone wants whip-up a patch for str.iter_index(), I would be happy to review it. Be sure to add a test case to make sure that the results are non-overlapping: list('aaaa'.iter_index('aa')) == [0, 2]

I'm guessing Terry wanted to say "os.listdir" instead of "os.walk".

May be str.iter_indices() or even just str.indices()?

I'm guessing Terry wanted to say "os.listdir" instead of "os.walk".
yes, sorry.

No one has submitted a patch for this or has expressed an interest in a long time. Perhaps the use case is already served by re.finditer()

Unassigning. Feel free to push this forward or to close due to lack on interest.

Perhaps the use case is already served by re.finditer()

def split_whitespace_ascii(s: str):
    return (pt.group(0) for pt in re.finditer(r"[A-Za-z']+", s))

solution above does not cover all possible data and is incorrect for bytes-like objects.

writing regular expressions for different separators/data would be a quite overheadish, so the idea of one-case solution doesn't seem to go very far and requires a bigger change in code for different separators.

let's try to revive this one :)

Making string.split iterator sounds like an interesting task. I found this issue because recently we talked in project that string.split returns a list and it can cause increased memory usage footprint for some tasks when there is large response to parse.

Here is small script, created by my friend Juancarlo Anez, with iterator version of string.split. Compared with default string split it uses much less memory. When running with memory-profiler tool: https://pypi.org/project/memory-profiler/

It creates this output
3299999
Filename: main.py

Line # Mem usage Increment Occurences Line Contents
============================================================
24 39.020 MiB 39.020 MiB 1 @Profile
25 def generate_string():
26 39.020 MiB 0.000 MiB 1 n = 100000
27 49.648 MiB 4.281 MiB 100003 long_string = " ".join([uuid.uuid4().hex.upper() for _ in range(n)])
28 43.301 MiB -6.348 MiB 1 print(len(long_string))
29
30 43.301 MiB 0.000 MiB 1 z = isplit(long_string)
31 43.301 MiB 0.000 MiB 100001 for line in z:
32 43.301 MiB 0.000 MiB 100000 continue
33
34 52.281 MiB 0.297 MiB 100001 for line in long_string.split():
35 52.281 MiB 0.000 MiB 100000 continue

You can see that default string.split uses much more memory.