I have been doing a lot of research on this. Wish I had thought do start the way I finished.

Basically, when math.nextafter() was added all the AIX bots were on systems running AIX earlier than AIX 7.2 TL2.

When AIX 7.2 TL2 was released (roughly Q3 2017) a (major?) change was made to the nextafter() function.

root@gcc119:[/home2/root]instfix -k IV95512 -a
IV95512 Abstract: nextafter(+0.0, -0.0) returns +0.0 instead of -0.0.

IV95512 Symptom Text:
 If(x==y) nextafter returns x instead of y.

At first glance - it appears the CPython code is reversing the arguments:

The lines in test_math.py are currently:
 +2026          # NaN
 +2027          self.assertIsNaN(math.nextafter(NAN, 1.0))
 +2028          self.assertIsNaN(math.nextafter(1.0, NAN))
 +2029          self.assertIsNaN(math.nextafter(NAN, NAN))

Moving line 2027 (which is what is failing) to 2029 - the other two lines pass on an AIX system with IV95512 applied. 

As IEEE754 says (and seems to have always said):

https://pubs.opengroup.org/onlinepubs/9699919799: 

If x or y is NaN, a NaN shall be returned.

The current test in Modules/mathmodule.c might be too simple.

I am working on a PR where I check for presence of APAR IV95512 - with the nextafter() changes.