1. Another potential crash caused by Python/dtoa.c: if the bigcomp functionality is disabled by replacing "#define STRTOD_DIGLIM 40" with "#define STRTOD_DIGLIM 4000", then the following string causes a crash:

>>> s = '5254406533529552661096610603582028195612589849649138922565278497589560452182570597137658742514361936194432482059988700016338656575174473559922258529459120166686600002102838072098506622244175047522649953606315120077538558010753730576321577387528008403025962370502479105305382500086822727836607781816280407336531214924364088126680234780012085291903592543223403975751852488447885154107229587846409265285440430901153525136408849880173424692750069991045196209464308187671479664954854065777039726878381767789934729895619590000470366389383963331466851379030183764964083197053338684769252973171365139701890736933147103189912528110505014483268752328506004517760913030437151571912928276140468769502257147431182910347804663250851413437345649151934269945872064326973371182115272789687312946393533547747886024677951678751174816604738791256853675690543663283782215866825e-1180'
[38199 refs]
>>> float(s)
cmp called with a->x[a->wds-1] == 0

I haven't yet found any examples that cause this crash without bigcomp enabled.

The crash is caused by a combination of two problems. (1) there's a buggy check for the smallest denormal in the "if ((aadj = ratio(delta, bs)) <= 2.) {" block in _Py_dg_strtod: the check ignores the possibility that bc.scale is nonzero. (2) The Bigint functions don't deal well with zero inputs: in particular, d2b gives strange results, and left shifting a zero Bigint can give a non-normalized result.

2. The check:

        if (!(word1(&rv) & LSB))
            break;
   

in _Py_dg_strtod again doesn't take into account the possibility that bc->scale is nonzero. It's supposed to be used in exact halfway cases to determine whether the current rv is already 'even' for the purposes of round-half-to-even. But for subnormal scaled rv, this check will (wrongly) always succeed, potentially giving an incorrectly rounded result.

However, it seems to be difficult to trigger this bug: the input must be a subnormal exact halfway case, which implies it must have many hundred digits. On the first pass through the strtod correction loop, the approximation is likely to be out by a few ulps, so we almost never hit the (i == 0) branch. The adjustment for the second pass then typically gives *exactly* the right result, as a consequence of the floating-point addition using round-half-to-even.

Okay, it's not that difficult to trigger (2). With the bigcomp functionality disabled as above:

AssertionError: Incorrectly rounded str->float conversion for 11651287494059419563861790709256988151903479322938522856916519154189084656466977171489691608488398792047332126810029685763620092606534076968286334920536334924763766067178320990794927368304039797998410780646182269333271282839761794603623958163297658510063352026077076106072540390412314438457161207373275477458821194440646557259102208197382844892733860255628785183174541939743301249188486945446244053689504749943655197464973191717009938776287102040358299419343976193341216682148401588363162253931420379903449798213003874174172790742957567330246138038659650118748200625752770984217933648838167281879845022933912352785884444833681591202045229462491699354638895656152216187535257259042082360747878839946016222830869374205287663441403533948204085390898399055004119873046875e-1075: expected 0x0.0d67b36890cfcp-1022, got 0x0.0d67b36890cfbp-1022

Second bug listed here fixed in r77698 in the trunk.

Out of curiosity, is it possible to write unit tests for any/all of this?

I haven't yet found any examples that cause this crash without bigcomp
enabled.

I presume you meant *with* bigcomp enabled?

Forgive my lack of knowledge, but why would you disable bigcomp?

Out of curiosity, is it possible to write unit tests for any/all of this?

There are some tests in Lib/test/test_strtod.py, that I added around two weeks ago and have been updating since; I don't have mechanism for running tests with the altered STRTOD_DIGLIM value, though.

I presume you meant *with* bigcomp enabled?
I did. Thanks!

Forgive my lack of knowledge, but why would you disable bigcomp?

Just as a stress test, really: a failure that occurs with bigcomp disabled might indicate a problem that could also occur with bigcomp enabled. Or someone might want to alter the value of the STRTOD_DIGLIM threshold at some later point based on performance tests; if the code is correct then it ought to be safe to do so.

I forgot to mention: the other reason one might want to disable the bigcomp stuff is that it seems less reliable than the rest of the code: of the 11-12 independent bugs that I found in dtoa.c recently, around 9 of them were related to the bigcomp code.

First bug fixed (in the trunk) in r77713.