Title: m68k Python 3.3 test results
Type: Stage:
Components: Interpreter Core Versions: Python 3.3
Status: open Resolution:
Dependencies: 17746 18062 18063 Superseder:
Assigned To: Nosy List: doko, mirabilos, pitrou, serhiy.storchaka
Priority: normal Keywords:

Created on 2013-05-25 20:55 by mirabilos, last changed 2014-05-22 21:45 by skrah.

File name Uploaded Description Edit
results.txt mirabilos, 2013-05-25 20:54 testsuite results, run#1
Messages (8)
msg189997 - (view) Author: mirabilos (mirabilos) Date: 2013-05-25 20:54

As a followup to I took the tree of Python 3.3.1 I had compiled already, with the patch that was eventually committed for issue17237, and ran the testsuite. Since I used configure flags to get it compiled quickly, it probably lacks functionality that’s normally there, so some tests got omitted.

The result of normally running the testsuite is attached as file; I can re-run any test verbosely, just ask ;-)

I can immediately say something to some failures:

① On m68k, the 68881 FPU, similar to the i8087 FPU, internally operates on 80-bit precision, which means that the “old” dtoa code is used. Contrarily to i8087, we do not want to disable that mode because that is not fully supported in some environments; especially, at least the most wide-spread emulator had bugs wrt. that. (Do note that, when you disable 80-bit precision on i8087, you get double-rounding as the only the mantisse, not the exponent, is bit-reduced, anyway. So I believe that you should not restrict the i8087 to 64-bit precision either.)

That’s responsible for test_optparse, test_difflib, maybe test_cmath, definitely test_format, test_types, test_math… ouch, quite some.

It would be interesting to see what happens when you take a normal i386 (not amd64 or x32) Linux host and compile py3k with SSE disabled (to force it to use the normal i80387 FPU), and disable HAVE_GCC_ASM_FOR_X87 so that X87_DOUBLE_ROUNDING is defined, because that’s what we have on m68k, and see whether the failure mode is the same. If so, you can then probably relatively easily fix those.

root@ara3:~/P-without-block # fgrep X87 pyconfig.h                                                              
/* #undef HAVE_GCC_ASM_FOR_X87 */

② This one is pretty severe: AssertionError: 42 != 44 : wrong size for <class 'BaseException'>: got 42, expected 44

On m68k, “natural” alignment is *not* used: instead of aligning 4-byte quantities on 4-byte boundaries, for almost all types, only a minimum alignment of 2 bytes is used; this also has effect on struct padding, array padding, and things like that.

As shown in fixing this is a “simple” matter of making these invalid assumptions explicit. To that effect, take Include/pyerrors.h and change
#define PyException_HEAD PyObject_HEAD PyObject *dict;\
             PyObject *args; PyObject *traceback;\
             PyObject *context; PyObject *cause;\
             char suppress_context;
#define PyException_HEAD PyObject_HEAD PyObject *dict;\
             PyObject *args; PyObject *traceback;\
             PyObject *context; PyObject *cause;\
             char suppress_context; char padding1[3];
i.e. add the padding that’s implicit on “natural alignment” platforms (a padding1[1] is implicit on m68k). Do *not* use -malign-int because that changes the ABI, and you probably won’t be able to make libc or syscalls any more…

However, this brings us to a problem with that solution (which I’d still prefer over everything else) in Debian (hence, putting doko on Cc), which I’ll demonstrate on the shortest struct I could find in that file, but applies to e.g. PyImportErrorObject too:

typedef struct {
    PyObject *code;  
} PySystemExitObject;

The “code” member of this struct is aligned with only *one* byte of padding, instead of three. This means that, when we apply this bugfix, that the published ABI changes: currently, sizeof(PySystemExitObject) is (8 (PyObject_HEAD) + 4 (dict) + 4 (args) + 4 (traceback) + 4 (context) + 4 (cause) + 1 (suppress_context) + 1 padding + 4 (code)) = 34; after the fact, it will be 36, and the offsetof(code) will have changed.

I’ll leave the understanding of these implications, and how they can best be handled in the distribution, to doko, because he’s probably the man who understands it best. I guess we need versioned dependencies or ABI virtual dependencies, since python3.3 is already uploaded, but some packages have already been built against it (probably not too difficult to identify them).

Or we can only get this fixed in python3.4 – no idea when that will land in Debian, or whether it’ll become the default python3 provider shortly.

③ As for any other failures, I don’t directly have an idea, but I guess we can work on them. I’d somewhat prefer to do that *after* the FPU precision-related bugs have been fixed, though, as it’ll reduce the noise, and possibly followup-errors. (Maybe I should even, for running the tests, fix the alignment issue locally in the tree.)
msg189998 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-25 20:57
Could you a separate bug entry for each class of issues? It will make things more manageable.
msg190001 - (view) Author: mirabilos (mirabilos) Date: 2013-05-25 21:02
OK sure; I put the two I identified already into issue18062 and issue18063; we can then retry here after those get fixed (I’ll just resend results from a patched tree then).
msg190020 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-25 21:58
About test_shutil failure. What filesystem are used? Do you run tests as root?
msg190026 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-25 22:10
Is m68k big-endian?
msg190028 - (view) Author: mirabilos (mirabilos) Date: 2013-05-25 22:19
Serhiy Storchaka dixit:

>About test_shutil failure. What filesystem are used?

/dev/root on / type ext3 (rw,noatime,errors=remount-ro,user_xattr,acl,barrier=1,nodelalloc,data=ordered)

>Do you run tests as root?


>Is m68k big-endian?

Yes: ILP32 big-endian, 2-byte aligned.

FWIW, I'm quite impressed with mksh interactively. I thought it was much
*much* more bare bones. But it turns out it beats the living hell out of
ksh93 in that respect. I'd even consider it for my daily use if I hadn't
wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh
msg190065 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-26 06:12
> >Do you run tests as root?
> Yes.

Well, this is issue17746.

> >Is m68k big-endian?
> Yes: ILP32 big-endian, 2-byte aligned.

Please open a new issue for str.find bug.
msg190090 - (view) Author: mirabilos (mirabilos) Date: 2013-05-26 14:26
Serhiy Storchaka dixit:

>> >Do you run tests as root?
>> Yes.
>Well, this is issue17746.

OK, I’ll re-run the tests as regular user (need to create one… ☺)
and with all those fixes applied, then we’ll have a look again.

(This will take a while.)

<Natureshadow> Oh, ich hab mim Bauch Mittelklick gemacht, als ich nach dem
    Kaffee gegriffen habe…
<mirabilos> Cool, ich hab ne neue eMail-Signatur
<Natureshadow> Sag doch sowas nich, wenn ich den Kaffee in der Hand habe!
    Gib mir nen Lappen! Schnell! Das kommt aber nicht mit in die Signatur!
Date User Action Args
2014-05-22 21:45:36skrahsetnosy: - skrah
2013-05-26 14:26:34mirabilossetmessages: + msg190090
2013-05-26 06:12:05serhiy.storchakasetdependencies: + test_shutil.TestWhich.test_non_matching_mode fails when running as root, m68k FPU precision issue, m68k struct alignment issue vs. PyException_HEAD
messages: + msg190065
2013-05-25 22:19:27mirabilossetmessages: + msg190028
2013-05-25 22:10:40serhiy.storchakasetmessages: + msg190026
2013-05-25 21:58:01serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg190020
2013-05-25 21:02:48mirabilossetmessages: + msg190001
2013-05-25 20:57:33pitrousetmessages: + msg189998
2013-05-25 20:55:04mirabiloscreate