New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Undefined behaviour in Objects/complexobject.c's complex_pow #88864
Comments
Objects/complexobject.c's complex_pow uses undefined behaviour, by casting a float of unknown magnitude to a long: int_exponent = (long)exponent.real; At Google we build with clang and -fsanitize=float-cast-overflow by default, which catches this particular kind of undefined behaviour. We didn't notice, however, because the only code we've come across that exercises this behaviour was a commented-out test in test_complex, which was uncommented in 3.8. Running the test, or just '1e19+1j ** 1e19', is enough to trigger the undefined behaviour. I'll prepare a PR to fix it. |
The fix broke NumPy (see also numpy/numpy#19612) It seems incorrect. After all, it doesn't matter much whether the float can be converted to an integer correctly (or even if it returns an undefined value), so long The old cast cast integers to complex when they were out of range (which is fine), the new code raises an error instead. |
Hmm, sorry, I overshot/misread :(. The thing that the NumPy test-suite trips over is that: c_powi(inf+0j, 3) seems to not raise, but: _Py_c_pow(inf+0j, 3.+0j) (or nan+0.j rather then inf+0j) does seem to raise (returning |
(Sorry for the spam. I think we can/should just hard-code the expected values in the NumPy test-suite. So this is not actually an issue for NumPy and probably just warrants a double-check that the behaviour change is desirable.) |
Thanks, @seberg. I'll take a look and see if we can restore the old behaviour, at least for Python 3.9 and 3.10. I'm not convinced that the new behaviour is objectively wrong, but I agree that the *change* in behaviour is problematic. I do find it a little odd that we're special-casing integer exponents for complex numbers when no such special-casing exists for floats. It's not purely for optimization (despite what the code might say). And all the messing around with trying to figure out exactly _which_ doubles can safely be cast to long (which was the original cause of this issue) seems like a symptom of a deeper problem, which is that it doesn't make a whole lot of sense to be distinguishing in the first place. But if we're going to change the behaviour, we should only be doing that in 3.11. OTOH, there is real value in z**2 being fast and accurate; we definitely want to keep that. (Perhaps the math and cmath modules should grow a |
I'm not sure I'm brave or foolhardy enough to try to figure out what all of the special case results *should* be for complex pow. The special cases seem reasonably clear cut for everything else in the cmath module, but for pow it's pretty much going to be a nightmare. I'm not sure there's much Python can do here in the way of guarantees, other than "not obviously horribly wrong". (And while I have reasonable confidence we can achieve the "not obviously horribly wrong" goal for everything else, I still have doubts for pow.) |
Addendum: even the C standards give up at this point. For everything else the special cases are spelled out in detail, but for cpow, from §G.6.4.1 of C17 (in Annex G), we have:
And that's it. (Well, not quite: there's a footnote, which says:
) |
Thanks for looking into it!
It is almost reassuring that the C standard also hedges out :). Although, it means that there is no reliance on math libraries if we want to treat the special cases more carefully :(. |
@pablo: I'm adding you to the nosy list to get your opinion on whether (the backport of) #71959 can go into 3.10.0.rc2. Summary: #71465 (which I reviewed and approved) and its backports fixed some undefined behaviour in complex exponentiation, but also introduced a behaviour change that affected NumPy: some exponentiation expressions that returned a result in 3.10.0b4 raise an OverflowError in 3.10.0.rc1. #71959 reverts the behaviour change but keeps the fix for the undefined behaviour. Can the backport of #71959 to the 3.10 branch go in before 3.10.0 final? |
Thanks for checking with me! Given that this is fixing a bug that we introduced in rc1, I think it makes sense to backport to fix it. I have reviewed the code in #71959 and is contained enough, so I think is a good idea to go ahead. Thanks Mark for all the work! |
I'm keeping track to the 3.10 backports to the second rc so could you add me as a reviewer for the backport? |
Thanks, Pablo!
Will do! I'll waiting on review for the PR against the main branch, but hope to get the backport PR up by this coming weekend at the latest. |
Re-closing; we should be good now. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: