The bug is described on Stack Overflow in length:

http://stackoverflow.com/questions/34100732/unexpected-output-using-pythons-ternary-operator-in-combination-with-lambda

For reference, the minimal reproducer is:

>>> f1, f2 = lambda: 1, lambda: 1.0
>>> f2()
1

The cause (according to the answer in the link) is that the two lambda's incorrectly compare as equal and end up sharing the same code object...one of the inequality checks is line number, so this only happens when the lambdas are on the same line.

Aside: it would be helpful if people would post bug summaries instead of just linking to an outside article. The link is of course very valuable information, but it saves us time if the poster summarizes the bug so the right people know to look at it.

Apologies for not posting a summary of the bug. I really had no idea how to describe the problem, as it is over my head. I could only refer to my question on Stack Overflow that triggered the discovery of the bug.

It's definitely also in 3.4 by the way.

Thanks for posting the bug. I added the aside because this is the third or fourth of these reference-only things I've seen in the past couple weeks and I finally figured out what bothered me about them. We should put something about this in the devguide on bug reporting, but of course most people reporting bugs will for good reason not read that. So, I just have to live with it :)

And yes, the reproducer reproduces the problem in all python versions currently under maintenance.

I've now changed the title to reflect the underlying bug.

The equality of code objects is determined by the code_richcompare() logic in Objects/codeobject.c.

Two code objects are equal if all of their attributes compare equal. That includes co_name, co_argcount, co_kwonlyargcount, co_nlocals, co_flags, co_firstlineno, co_code, co_consts, co_names, co_varnames, co_freevars, and co_cellvars.

At the heart of David Murray's minimal example, the reason the two distinct code objects compare equal is that their co_consts compare as equal.

If you wanted to fix this, code objects would need to recursively check for both normal equality and type equality.

>>> f1 = lambda: 1
>>> f2 = lambda: 1.0
>>> f1.__code__.co_consts == f2.__code__.co_consts
True
>>> map(type, f1.__code__.co_consts) == map(type, f2.__code__.co_consts)
False

A type-based check runs into problems with 0.0 vs -0.0. For example, on Python 2.7.11:

>>> x, y = lambda: 0.0, lambda: -0.0
>>> y()
0.0

I wasn't able to reproduce the -0.0 problem with Python 3.4 on Ideone; y.__code__.co_consts seems to have an unused 0.0 in it on 3.4. I don't have access to Python 3.5, so I don't know what the situation is like on that version.

Here's another variant (courtesy of Nick Coghlan):

python3.5 -c "seq1 = [1.0 for x in range(5)]; seq2 = [True for x in range(5)]; print(seq1); print(seq2)"

One possible solution for all these variants is to let code objects track both the co.firstlineno and co.firstrowno.

FWIW, the bug is present in PyPy as well.

From what I can see:

• checking constant types in addition to their values should be a two line change (plus tests)
• checking the column number in addition to the line number would be a more comprehensive fix, but also a constructor API change (since PyCode_New doesn't currently accept a column parameter)

The "values of the same constant type that are distinct but equivalent may still compare equal" case is obscure enough that I think the lower impact change is likely a better option, especially as 3.x currently "handles" the "lambda: -0.0" case by having both the unfolded 0.0 and the folded -0.0 in the constant list.

------------
Additional detail for those interested:

The lowest impact fix from a code change would be to add a type equivalence check for constants as Raymond first suggested, as that only involves adding an extra check to code_richcompare: https://hg.python.org/cpython/file/tip/Objects/codeobject.c#l416

However, the idea of tracking "co_firstcolno" in addition to "co_firstlineno" is a more robust fix, as it means any lexically distinct code objects will necessarily be considered distinct by the interpreter. The downside is that it means touching more code and adding a new public API, since PyCode_New doesn't currently accept a "firstcolno" parameter - code objects are implicitly assumed to be one-per-line.

Out of curiosity, I also went looking for the code in the code generator that collapses the "equivalent" code objects together. The issue is that constants are stored in a dict (mapping them to their co_consts index) during compilation and assembly, so equivalent objects will be merged together (and refer to the index of the first one defined).

Interestingly, co_filename is not used as part of the equivalence criteria, so code object equivalence can be fooled across multiple input files. Fortunately in this case, the false equivalence isn't relied on by the code generator.

  $ cat a.py
  def f():
      return 1

  $ cat b.py
  def f():
      return 1.0

  $ ./python.exe -q
  >>> import a, b
  >>> a.f.__code__ == b.f.__code__  # False equivalence
  True
  >>> a.f()
  1
  >>> b.f()                         # Generated code is correct
  1.0

Besides aliasing int/float/decimal/fraction/complex/bool, codeobj.__eq__() can also alias str/unicode on Python 2.7. Likewise, 0.0 and -0.0 can be conflated. NaNs don't seem to be a problem.

I think we should focus on fixing the spec for code object equivalents. Perhaps the test can be simplified to use (co_firstlineno, co_firstrowno, co_filename).

Other possible minimal fixes:

• compile.c:compiler_addop(): special-case code objects too, and stick their identity in the tuple 't'.

• or, in compile.c:makecode(), append the first row number to co_consts.

The lowest impact fix from a code change would be to add a type equivalence check for constants as Raymond first suggested, as that only involves adding an extra check to code_richcompare: https://hg.python.org/cpython/file/tip/Objects/codeobject.c#l416

It is not so easy. (1,) and (1.0,) are equal and have the same type. To make correct type-sensitive equivalence check, you need to introduce new protocol, new special method on low level and new operator/function on high level.

Note that 3.x does not correctly handle -0.0, you just have to work a bit harder:

>>> (lambda: (-0.0, 0.0), lambda: (0.0, -0.0))[1]()
(-0.0, 0.0)

I think we should focus on fixing the spec for code object equivalents. Perhaps the test can be simplified to use (co_firstlineno, co_firstrowno, co_filename).

This is not enough if the code was compiled from a string.

>>> x = eval('lambda: 1')
>>> y = eval('lambda: 1.0')
>>> x.__code__ == y.__code__
True
>>> x.__code__.co_filename == y.__code__.co_filename
True

Nobody seems to have asked this, so I'll be that guy. In which circumstances does comparing two code objects (at function creation time, what's more) make any sense? I mean, I'm fine with being able to compare two code objects, but I don't think that's something that should be automated.

Is there any particular reason why this is so? The only reason I could think of was that small, identical functions could use the same code object -- but then Raymond proved that different files will not share the code object, and identical functions on different lines will not, either.

As functions grow in size and complexity, having two virtually identical functions is probably bad design to begin with. So, seeing as this causes more harm than good (and I doubt it's of any use nowadays - it might have been back then, I don't know), I suggest we simply drop the implcit code objects compare-and-replace that's happening here.

I suppose this is needed above all for testing. test_compile, test_codeop, test_marshal, and few other tests compare code objects.

I'm not suggesting to get rid of the rich compare ability of code objects, which makes sense to me. What doesn't make sense to me, however, is when a function's code object is replaced by another one because it compares equal. I see no use case for this, and only leads to head-scratching.

I believe code objects should not be touched at all.

That's what I suggested ("compile.c:compiler_addop(): special-case code objects too, and stick their identity in the tuple 't'.") and how I fixed the same bug in PyPy (https://bitbucket.org/pypy/pypy/commits/7ec3e1d02197).

[Emanuel Barry]

In which circumstances does comparing two code objects
(at function creation time, what's more) make any sense?

It makes closures efficient:

    >>> def f(x):
            def g(y):
                    return x + y
            return g

    >>> h = f(1)
    >>> i = f(2)
    >>> h.__code__ is i.__code__
    True

http://code.activestate.com/recipes/578353-code-to-source-and-back/ compares code objects as part of round trip testing.

I see. Is there any to special-case those so that only closures use that? Maybe by checking on the function object itself - the function itself would be quite similar, as well.

@mark - I think you've misunderstood me (others did too, so I'm going to assume I just explained poorly) - I'm not debating the use of comparing code objects, I'm talking about the implicit replacement of code objects in functions. However, it seems to be of use, so outright removing the behaviour isn't an option.

It makes closures efficient:

No, code objects are not compared here. The code object is constant, and f() returns different closure objects using the same code object.

AFAIK the comparison of different code objects is used exclusively in testing.

Serhiy: that can't be entirely true, since we have here a bug where two lambdas on the same line are getting the same code object incorrectly, and that is not a testing context.

Ah, haypo explained the code to me, and now I understand your comment Serhiy.

So, the replacement happens because of (a) an optimizer general rule and (b) the fact that code object *can* be compared.

So it sounds like Armin's suggestion of making an exception for code objects in the optimizer is the correct solution.

The issue with code objects that aren't really equal comparing equal would then be a separate bug that affects, as Serhiy said, only testing (that we know of...who knows what people like PJE might be doing with comparing code objects :)

Indeed. My answer actually is an answer to implicit question: could we make different code objects be always non-equal? Answer: no, because we use the comparison of different code objects to test compiler and marshaller.

May be we can get rid of code objects comparability and provide separate function specially for testing. Of course this likely will break tests in third-party packages that do low-level hacking on code objects.

compiler_add_o() uses an heuristic to compare and merge duplicated constants. It has special cases for float and complex numbers, but it's not designed to handle more types.

Funny, I had the same isue last week why I added support for tuple and frozenset "constants" in AST. I had to explicitly support these types in compiler_add_o().

I see two options:

(1) share code between compiler_add_o() and code_richcompare() to ensure that 1 and 1.0 constants are not seen as equal
(2) modify compiler_add_o() to never merge code objects, always considere them as unequal

For (2), there is a minor technical issue: you have to generate an unique key for the dictionary.

I prefer option (1) for consistency.

Would option (1) work if wrap 1 and 1.0 in a tuple? In a list? In a custom collection?

code_richcompare.patch: fix code_richcompare() to not consider that constants (1,) and (1.0,) are equal.

The patch lacks an unit test.

We may use the new _PyCode_ConstantKey() function to compare other code attributes?

Would option (1) work if wrap 1 and 1.0 in a tuple? In a list? In a custom collection?

Right now, the Python compiler is quite limited. It only produces constants for integers and strings, not complex types. The peephole optimizers is responsible to produce tuple and frozenset constants, and the optimizer is more naive. It doesn't try to merge constants, nor remove items from co_consts to only keep the new container. Example:

>>> def f():
...  return (1,2,3)
... 
>>> f.__code__.co_consts
(None, 1, 2, 3, (1, 2, 3))

1, 2, 3 constants are kept, whereas only (1, 2, 3) constant is used.

Test with my patch:

>>> f1, f2 = lambda x: x in {1,}, lambda x: x in {1.0,}
>>> 
>>> f1.__code__ is f2.__code__
False
>>> f1.__code__.co_consts
(None, 1, frozenset({1}))
>>> f2.__code__.co_consts
(None, 1.0, frozenset({1.0}))

The frozenset are different are expected.

The frozenset are different are expected.

Oh wait, in this example, it works, but not because of the expected reason. frozenset({1}) is equal to frozenset({1.0}) and code_richcompare() doesn't create a special key to make them different.

The example only works because the peephole optimizer is lazy and leave 1 and 1.0 in co_consts which are seen as different by the patched code_richcompare().

code_richcompare-2.patch: Updated patch to handle also frozenset, the other constant type generated by the peephole optimizer.

code_eq.py: code to test that frozenset() are inequal. The code hacks constants, don't run the code with new constants or it will crash!

If go this way, I would add explicit support of all types supported in marshal, and include id(obj) into the key of all other types.

FYI this issue is linked to the issue bpo-26146 which allows to emit constants from an AST optimizer (see also the PEP-511).

Let me try to explain this issue again.

"f1, f2 = lambda: 1, lambda: 1.0" is compiled to two MAKE_FUNCTION instructions, MAKE_FUNCTION takes a code object as parameter (and a name). The Python compiler merges constants which are seen as "equal", with exceptions to not merge values of different types, or float of different sign.

Merging duplicate code objects is a cool micro optimization, I prefer to keep it. My patch keeps this micro optimization, but fix the bug: it ensures that equal constants having different types are not seen as equal. For example, 0 is equal to 0.0, but if when used for code constants, the code objects are seen a different.

Patch version 3:

• as suggested by Armin Rigo & Serhiy Storchaka: use id(obj) in the constant key for unknown constant types -- in practice, this patch is never taken, it's just to be extra safe (I checked manually by running the whole test suite when an assertion, assertion not in the posted patch)
• add a lot of unit tests
• add a documentation to _PyCode_ConstantKey()

@serhiy: does it look good to you now?

Would option (1) work if wrap 1 and 1.0 in a tuple? In a list? In a custom collection?

My patch now uses id(obj) in the "constant key" for unknown types. The compiler only emits simple type (int, str, ...), tuple, frozenset and code objects.

You *can* other types if you patch manually constants with custom objects, since code_richcomp() now uses the "constant key" function to compare constants. For example, my fat project has a replace_consts() function to inject builtin functions in constants:
http://fatoptimizer.readthedocs.org/en/latest/fat.html#replace_consts

There is also @asconstants decorator of codetransformer which allow to inject arbitrary types in function constants:
https://pypi.python.org/pypi/codetransformer

It looks to me that the patch makes set and frozenset constants to be equal, but makes sets with different order (as {2**62, 1} and {1, 2**62}) differ. And looks you had missed my comments to previous patches.

Patch version 4:

Fix _PyCode_ConstantKey:

• always return a tuple
• create a frozenset/set if input is a set/frozenset, items are unordered
• enhance unit tests: compare values using repr() to compare value + type at once
• fix reference leak

Other minor changes to take Serhiy's comments in account.

Thanks Serhiy for good suggestions!

And sorry, I missed your comments, it looks like some emails were seen as spam :-/

I meant that set shouldn't be handled in _PyCode_ConstantKey at all. Only frozenset constants can be considered equal. Sets as well as lists all should be different.

There is yet one issue with string and bytes literals. When run Python with the -b option:

>>> a, b = lambda: 'a', lambda: b'a'
sys:1: BytesWarning: Comparison between bytes and string
sys:1: BytesWarning: Comparison between bytes and string

May be the type should be the first item in the key.

Updated patch version 5.

I meant that set shouldn't be handled in _PyCode_ConstantKey at all. Only frozenset constants can be considered equal. Sets as well as lists all should be different.

Ok, I dropped support for set type to only keep support for frozenset. set are now always considered as different, as list and other "unsupported types".

There is yet one issue with string and bytes literals. When run Python with the -b option:
(...)
May be the type should be the first item in the key.

Oh, good catch. It's an old bug.

Yeah, putting the type as the first parameter fixes the issue. I made this change. I had to update compile.c to exchange the type and the value in the tuple.

New changeset 6c33d4cc9b8f by Victor Stinner in branch 'default':
code_richcompare() now uses the constants types
https://hg.python.org/cpython/rev/6c33d4cc9b8f

I pushed my latest patch with minor changes in comments.

I will wait for buildbots before backporting the change to Python 2.7 and 3.5. For the backport, I will probably remove the optimization on frozenset since it's only useful for AST optimizers (the optimization is a new feature, I considered that it was worth it to add it Python 3.6 as part of my work on the PEP-511).

New changeset 8e0a736b82ff by Victor Stinner in branch '3.5':
code_richcompare() now uses the constants types
https://hg.python.org/cpython/rev/8e0a736b82ff

I will probably remove the optimization on frozenset since it's only useful for AST optimizers (the optimization is a new feature, I considered that it was worth it to add it Python 3.6 as part of my work on the PEP-511).

Hum, it doesn't work: test_compile_ast() of test_compile fails without this code. The test relies (indirectly) on the fact that two code objects using a frozenset constant are equal.

New changeset 9e13d97ceca2 by Victor Stinner in branch '2.7':
code_richcompare() now uses the constants types
https://hg.python.org/cpython/rev/9e13d97ceca2

Ok, the fix is now pushed to Python 2.7, 3.5 and 3.6.

Thanks Tijs Van Oevelen for your bug report ;-)

A workaround look to define the two lambda functions on two different lines. Or maybe cast explicitly to float? I don't think that it's a common bug so you should be able to survive with it until the next bugfix version is released :-)

New changeset 16f60cd918e0 by Victor Stinner in branch 'default':
PEP-511
https://hg.python.org/peps/rev/16f60cd918e0

Someone asked on reddit. The Misc/NEWS entry for this reads:

Issue bpo-25843: When compiling code, don’t merge constants if they are equal but have a different types. For example, f1, f2 = lambda: 1, lambda: 1.0 is now correctly compiled to two different functions: f1() returns 1 (int) and f2() returns 1.0 (int), even if 1 and 1.0 are equal.

Shouldn't that last part read

"and f2() returns 1.0 (float), even if 1 and 1.0 are equal."
^^^^^

As in, f2 returns a float, not an int.

If this is a mistake, let me fix it for 3.5.2 final and I'll merge it back into trunk etc. If you fix it it wouldn't ship in 3.5.2 final.

New changeset a36238de31ae by Victor Stinner in branch '3.5':
Issue bpo-25843: Fix the NEWS entry
https://hg.python.org/cpython/rev/a36238de31ae

Oh, I fixed the typo in the Misc/NEWS in 3.5 and default branches.

Yes, which doesn't help 3.5.2 final as I don't pull revisions by default after rc1. :p