I verified this bug in the Py3.0 and Py3.1. Shall come out with a patch
for it.

Oops. I didn't think it translate the code in addinfobase to the new
style of iterators.

Jeremy

On Tue, Dec 9, 2008 at 7:50 AM, Senthil <report@bugs.python.org> wrote:

Senthil <orsenthil@gmail.com> added the comment:

I verified this bug in the Py3.0 and Py3.1. Shall come out with a patch
for it.

----------
nosy: +orsenthil

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4608\>

---

---

Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/jeremy%40alum.mit.edu

Here is a patch to fix the issue.
Jeremy, is it approach okay? Or do you have any other suggestion?

Regarding Senthil's patch:
__next__() method seems superfluous to me (and the implementation is buggy).

Jakub,

I have attached a revision to the patch.
You are right, when __iter__ returns self.fp (as in previous patch), the
__next__ is superflous.
But, I was thinking of __iter__ returning an instance of addbase,
instead of self.fp and in that case __next__ was required. But I see
that i had not changed self.fp to self.

This is implemented in the similar lines of IOBase class, io.py
w.r.t to your other comment, why do you think __next__ implementation is
incorrect?

Thanks,
Senthil

Oops, __next__ is OK. Sorry for the confusion.

Senthil, do you think you could provide a test case for this?

Thank you!

Test cases attached.

The second one highlights a bug in the current patch, as it fails to
return a line longer than 65475 chars. This behavior doesn't match trunk's.

This issue is already fixed by jeremy at Revision 70815, wherein "The
response from an HTTP request is now an HTTPResponse instance instead of
an addinfourl() wrapper instance."

So the issue won't be present in the py3k code ( confirmed).

However, the test added by Daniel,which tests for urlopen() for a
request which is b"verylong" * 8192 still fails.

It is not just with iteration; but test_200 will fail too if the request
is a large chunk. This is only in py3k branch, test will pass in the
trunk code. I am investigating further.

This looks as though its a short write:

[pid 28343] recvfrom(5, "GET / HTTP/1.1\r\nAccept-Encoding:"..., 8192, 0, NULL, NULL) = 118
[pid 28343] poll([{fd=5, events=POLLOUT, revents=POLLOUT}], 1, 10000) = 1
[pid 28343] sendto(5, "HTTP/1.0 200 OK\r\n", 17, 0, NULL, 0) = 17
[pid 28343] poll([{fd=5, events=POLLOUT, revents=POLLOUT}], 1, 10000) = 1
[pid 28343] sendto(5, "Server: TestHTTP/ Python/3.2a1+\r"..., 33, 0, NULL, 0) = 33
[pid 28343] poll([{fd=5, events=POLLOUT, revents=POLLOUT}], 1, 10000) = 1
[pid 28343] sendto(5, "Date: Sun, 08 Aug 2010 09:41:08 "..., 37, 0, NULL, 0) = 37
[pid 28343] poll([{fd=5, events=POLLOUT, revents=POLLOUT}], 1, 10000) = 1
[pid 28343] sendto(5, "Content-type: text/plain\r\n", 26, 0, NULL, 0) = 26
[pid 28343] poll([{fd=5, events=POLLOUT, revents=POLLOUT}], 1, 10000) = 1
[pid 28343] sendto(5, "\r\n", 2, 0, NULL, 0) = 2
[pid 28343] poll([{fd=5, events=POLLOUT, revents=POLLOUT}], 1, 10000) = 1
[pid 28343] sendto(5, "verylongverylongverylongverylong"..., 56001, 0, NULL, 0) = 49054
[pid 28343] shutdown(5, 1 /* send */) = 0

The attached patch handles short writes, and adds ajaksu2's tests.

Thanks, Brian.
Pushed with revision 83833.

This issue appears to persist when the protocol used is FTP:

root@tp-db $ cat test.py
from urllib.request import urlopen
for line in urlopen('ftp://gentoo.osuosl.org/pub/gentoo/releases/'):
print(line)
break

root@tp-db $ python3.2 test.py
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    for line in urlopen('ftp://gentoo.osuosl.org/pub/gentoo/releases/'):
TypeError: 'addinfourl' object is not iterable

Oops, previous example was a directory, but it's the same if the url points to a ftp file.

The patch that makes addinfourl() iterable was not commited due to the change to HTTP request see: msg86365 (http://bugs.python.org/issue4608#msg86365).

Since urllib is protocol agnostic it should behave the same with FTP, right?

So, where to fix? Change the addinfourl() to become itrable or change the FTPHandler return?

It turned out that although the addinfourl instance had the __iter__ attribute in addbase.__init__ correctly assigned, __iter__ wasn't found by the iter builtin. It seems that iter always tries to use the __iter__ method of the class and doesn't look at the instance.

Riccardo Attilio Galli and I made the attached patch. The patch also fixes a corresponding TypeError for "file://" URLs, not just "ftp://" URLs.

New changeset c0a68b948f5d by Raymond Hettinger in branch '3.2':
Issue bpo-4608: urllib.request.urlopen does not return an iterable object
http://hg.python.org/cpython/rev/c0a68b948f5d

New changeset d4aeeddf72e3 by Raymond Hettinger in branch 'default':
Issue bpo-4608: urllib.request.urlopen does not return an iterable object
http://hg.python.org/cpython/rev/d4aeeddf72e3

Thanks for the patch.