If we call some function f with a generator as star argument and this
generator raises a TypeError, we get the following exception:

>>> def f(x): pass
... 
>>> def broken(): raise TypeError
... 
>>> f(*(broken() for x in (0,)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after * must be a sequence, not generator

This is a misleading error message, as it's usually no problem to use a
generator as a star argument. Even just replacing the TypeError by some
other exception leads to the expected result, i.e. the exception gets
correctly propagated.

The problem seems to be around line 3710 of Python/ceval.c where the
generator is converted to a tuple. If this conversion raises a
TypeError, then the error message is replaced, which will mask any
TypeError raised by the generator.

I'm not sure how to solve this. We probably can't distinguish a "good"
TypeError from a "bad" TypeError at this point, so we might have to make
a special case for the conversion of generators.

The issue1615 has the same kind of problems: an AttributeError is masked
by another one.

I'm getting confused about whether it's actually desired behaviour that
generators can be star arguments.

The error message seems to say it's not: "argument after * must be a
sequence". The docs seem to agree: "If the syntax *expression appears in
the function call, expression must evaluate to a sequence." However
test_extcall specifically tests function calls with (non-sequence)
iterables as star arguments.

I can confirm that (a) this exact behavior is happening and (b) it quite
confused me (most of the time it works!). What would be a "good"
TypeError? I'd vote for generators to be explicitly supported even if it
required a special case. Thanks!

I added a simple check for iterables. This is not very elegant, but
performance is only affected in the case of an exception. Patch and
corresponsing test are attached as "TypeError.patch".

As I pointed out above, the actual error message "must be a sequence" is
also inconsistent with the implementation (and tests) which allows any
kind of iterable. The attached and independent patch
"message_and_docs.patch" changes this to "must be an iterable" and
corrects docs and tests accordingly.

This patch leaks a reference on each call to PyObject_GetIter(). And I'm
not sure it is a good idea to call this function again: it may be very
expensive! I'd prefer a simple check on the tp_iter member.

Sorry, I had meant to use PyIter_Check instead of PyObject_GetIter.
Don't know why I didn't do so... ;-)

I corrected the patch.

I have also hit this error.  I went to report it but found it already entered (good news), but not resolved from nearly a year ago (bad news).

The error masked another bug that I had in my program and it took me quite awhile to figure out what the real problem was.

I use *generator arguments quite a lot, so was surprised to see the error.  So I, for one, can say that if you disable *generator arguments, you will break existing code.

If anybody cares, I have verified that this error also appears in Python2.5 and Python2.4 and am attempting to add python2.5 to the Versions list.  (And yes, *generators were allowed in Python2.4!)

Is this headed for resolution?  Progress on it seems to have stalled nearly a year ago.  Can I vote to revive this?

I verified with 3.1 the two OP cases and that generators work fine as long as they supply the correct number of values.

def f(x): return x
def broken(): return 1
print(f(*(broken() for x in (0,))))
# prints 1

Change (0,) to (0,1) the normal arg num mismatch message appears.

test_extcall tests version of Nothing() that follow both the old and new iteration protocol. It is possible that 'sequence' in meant in the broader sense of finite iterable rather that the narrow sense of
5.6. Sequence Types — str, bytes, bytearray, list, tuple, range

Since that is confusing, I would replace 'sequence' with 'finite iterable'. (Infinite iterables, obviously, are bad, just as in any other uncontrolled situation, such as "a,*b = itertools.count()".)

So, combine the correction and the suggestion above with original and diff against current trunk (py3k branch) if you can or at least 3.1.2.

IIUC, the only change you suggest for my patch is using "finite iterable" instead of "sequence" or "iterable", right?

I've looked at the docs and there seems to be no precedent for "finite iterable". I think it's just as obvious that the iterable has to yield a correct (and finite) number of parameters as the fact that "list(itertools.count())" is a bad idea. So for consistency I would like to keep "iterable".

Ok, leave iterable as is. More important, you have two disjoint patches that I believe should be combined.

Attaching a combined patch against the current py3k.

I think the patch isn't entirely correct.  It uses PyIter_Check for detecting the case when an *iterable* raises TypeError, but that function actually checks for an *iterator*.  The check on the tp_iter member mentioned by Amaury Forgeot d'Arc probably would be better, but even that wouldn't detect every iterable: "Its presence normally signals that the instances of this type are iterable (although sequences may be iterable without this function)." (http://docs.python.org/dev/py3k/c-api/typeobj.html#PyTypeObject.tp_iter)  (Apparently any object with a __getitem__ is iterable.  By the way, collections.abc.Iterable also doesn't detect this case.)

I'm attaching an updated patch. Instead !PyIter_Check() this patch checks for tp_iter == NULL && !PySequence_Check.  If this condition is false, PyObject_GetIter has a chance to succeed (and if it fails, we shouldn't mask the exception).  I also added more tests which show why the previous patch was incorrect.

#11944 is probably a duplicate of this and should be checked when this is fixed

I haven’t tried to understand what the patches do, but Issue 5218 looks like a very similar problem with a patch including a test case.

#13904 is another dup, with patches to test and ceval. I asked that they be reloaded to this issue.

#5218 is a 4th duplicate, also with a patch to the ceval loop and test.

Sorry Martin, I see you already said that. Anyway, I closed other three in favor of this one.

Sounds like we just need someone comfortable with modifying ceval.c to apply this ;)

I am posting a new version of Daniel’s patch as issue4806-star-TypeError.v2.patch:

* Merged with recent “default” (3.5) branch
* Dropped the documentation fix, since revision a8aa918041c2 already fixed that in an identical fashion
* Changed to a “look before you leap” style check before attempting PySequence_Tuple(). This way we get to keep other valid errors such as “iter() returned non-iterator”.

I am comfortable with the code changes previous patch as well as mine. Checking the “tp_iter” field and PySequence_Check() is basically what happens behind the scenes, via PyObject_GetIter() anyway. So I think either patch is valid and a worthwhile improvement.

Up for this bug. I got this bug in my code and loose one hour to understand what happened. Please review Martin's patch !

I think this is ready to push for Python 3. Python 2 might need an extra Py_TPFLAGS_HAVE_ITER check, perhaps reusing some code from the Issue 5218 patch.

New changeset 1a8dc350962b by Martin Panter in branch '3.5':
Issue #4806: Avoid masking original TypeError in call with * unpacking
https://hg.python.org/cpython/rev/1a8dc350962b

New changeset 1261f5f6fc95 by Martin Panter in branch 'default':
Issue #4806: Merge * unpacking fix from 3.5
https://hg.python.org/cpython/rev/1261f5f6fc95

In Python 2, the old-style “instance” objects are making it hard to fix this like in Python 3. And completely removing the custom error message seems a bit drastic for 2.7.

Instead, I propose to just check for a generator type after the TypeError is caught, and don’t bother checking for other kinds of iterables and sequences.

It gets even weirder with coroutines involved:



>>> f = (coro() for coro in l) 
>>> asyncio.gather(*f) # TypeError: asyncio.gather() argument after * must be a sequence, not generator
>>> asyncio.gather(*[*f]) # ok

Because coroutines are executed later, the last gather() doesn't show the error message. So it tricked you into thinking that the second version is ok while the first one is not.

But of course, both are ok, the problem lies in a bug deeper in your coroutine triggering the TypeError.

Michel: I suspect your code doesn’t actually get up to handling any coroutines, and the problem is in your generator expression. What is “l”, and what are the items in it? The bug should already be fixed ready for the 3.5.2 and 3.6 releases, but I can produce this on 3.5.0:

>>> l = [None]
>>> f = (coro() for coro in l) 
>>> asyncio.gather(*f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: gather() argument after * must be a sequence, not generator

In the current 3.6 code the bug is fixed:

>>> asyncio.gather(*f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <genexpr>
TypeError: 'NoneType' object is not callable

The reason why the second call does nothing interesting is because the generator expression has already die because of the exception, and there is nothing left to iterate:

>>> remaining = [*f]
>>> remaining
[]
>>> asyncio.gather(*remaining)
<Future finished result=[]>

We fixed our bug days ago, but I would have expected [*gen] to have triggered an exception before it even got to gather().

The real code was something like:

>>> l = (ensure_awaitable(callable_obj) for callable_obj in callable_list)
>>> gather(*l)

ensure_awaitable() is just using inspect to turn coroutine functions into coroutines, and wraps non coroutine callables with asyncio.coroutine(callable)().

In the end, ensure_awaitable did raise TypeError if the argument passed was not a callable.

New changeset eef8f72ddb00 by Martin Panter in branch '2.7':
Issue #4806: Avoid masking TypeError when *-unpacking a generator
https://hg.python.org/cpython/rev/eef8f72ddb00

I believe I encountered this issue today on Python 2.7.14 (https://ci.appveyor.com/project/jaraco/jaraco-windows/build/31/job/lenh5l4dcmj137b9). In this case, I have an iterable (in itertools.imap) that raises a TypeError when evaluated, not a generator.

The issue doesn't happen on Python 3, where 'map' is used instead of itertools.imap.

Does this patch need to be extended to support any iterable?