This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author petr.viktorin
Recipients benjamin.peterson, bmwiedemann, mcepl, methane, petr.viktorin, sascha_silbe, vstinner, zbysz
Date 2020-01-08.14:05:06
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1578492306.81.0.929647322653.issue34033@roundup.psfhosted.org>
In-reply-to
Content
> There is also one aspect where i586 builds end up with different .pyc files than x86_64 builds. And then we randomly chose one of them for our "noarch" python module packages and hope they work everywhere (including on arm and s390 architectures).

They are functionally identical, despite not being bit-by-bit identical.
If they do not work everywhere, it's a very serious bug.

> So is someone working towards a concept that makes it is possible to create the same .pyc files anywhere?

No, it's a known issue no one is working on.

> Can I help something there?

Maybe?
The two main culprits are in the marshal serialization algorithm:  https://github.com/python/cpython/blob/master/Python/marshal.c
Specifically:
- a heuristic depends on refcount (i.e. state of objects in the entire interpreter, rather than just relationships between serialized objects): https://github.com/python/cpython/blob/33b671e72450bf4b5a946ce0dde6b7fe21150108/Python/marshal.c#L304
- (frozen)sets are serialized in iteration order, which is unpredictable (and determinig a predictable order is not trivial): https://github.com/python/cpython/blob/33b671e72450bf4b5a946ce0dde6b7fe21150108/Python/marshal.c#L498

A solution will probably come with an unacceptable performance hit -- it's good to keep generating the .pyc files fast. Two options to overcome that come to mind:
- make reproducibility optional (which would make the testing more cumbersome)
- make an add-on tool to re-serialize an existing .pyc.
History
Date User Action Args
2020-01-08 14:05:06petr.viktorinsetrecipients: + petr.viktorin, vstinner, benjamin.peterson, mcepl, sascha_silbe, zbysz, methane, bmwiedemann
2020-01-08 14:05:06petr.viktorinsetmessageid: <1578492306.81.0.929647322653.issue34033@roundup.psfhosted.org>
2020-01-08 14:05:06petr.viktorinlinkissue34033 messages
2020-01-08 14:05:06petr.viktorincreate