This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Port Python to 3DS: micro kernel, homebrew, newlib (Missing guards for some POSIX functions)
Type: compile error Stage:
Components: Extension Modules Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Link Mauve, ezio.melotti, loewis, pitrou, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2014-10-13 12:44 by Link Mauve, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
guards.diff Link Mauve, 2015-02-02 00:54 Patch against latest default review
getedig.patch Link Mauve, 2015-02-17 18:45 New version of the patch
getedig.patch Link Mauve, 2015-02-17 19:28 Same patch, without the git diff format. review
getedig.patch Link Mauve, 2015-02-18 19:19 Newer version of the patch. review
getedig.patch Link Mauve, 2015-02-18 19:38 Forgot one fix in the previous one. review
Repositories containing patches
http://hg.linkmauve.fr/cpython
Messages (9)
msg229243 - (view) Author: (Link Mauve) Date: 2014-10-13 12:44
Many POSIX functions aren’t available on every system, especially embedded ones.

The first patch introduces guards around some of these functions and add them to AC_CHECK_FUNCS in the configure.ac; the second one recompile every changed generated file, using autoreconf -fi and clinic.
msg235214 - (view) Author: (Link Mauve) Date: 2015-02-02 00:54
This issue is still present in latest 3.5, all the way down to 2.7.
msg236142 - (view) Author: (Link Mauve) Date: 2015-02-17 18:45
Removed the unwanted introduced function, and added a comment signaling the end of the HAVE_TTYNAME #ifdef.

The full patch is attached, the diff with the previous version can be found at http://linkmauve.fr/files/getedig.patch
msg236146 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-17 19:24
3.2 and 3.3 are only for security fixes now.
msg236414 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-22 18:49
HAVE_SYS_RESOURCE_H still is used for #include in several files.

If guards are added to pwd functions, shouldn't it be added to grp and spwd functions? Is there a platform that have pwd.h, but not getpwuid, getpwnam or getpwent. May be instead testing the existence of separate functions adds the test about the existence of pwd.h in configure?
msg239065 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-03-23 20:50
Personally, I'd rather answer that it's not our problem if some systems are not POSIX-compliant. Maintainers of such systems can always maintain a patch to disable the missing functionality. Adding conditionals everywhere has a non-trivial maintenance cost.
msg239067 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-23 21:05
> Many POSIX functions aren’t available on every system, especially embedded ones.

What do you call "embedded systems"? I worked on set top boxes (the box to watch television) between 2011 and 2013 and we had a regular Linux kernel with all POSIX functions. Most of the time, the CPU was a slow MIPS. The tool chain was based on GCC. For some hardware, we used the µlibc but this C library provides all functions required by Python.

> Personally, I'd rather answer that it's not our problem if some systems are not POSIX-compliant. Maintainers of such systems can always maintain a patch to disable the missing functionality. Adding conditionals everywhere has a non-trivial maintenance cost.

I agree. Anyway, in the embedded world, softwares are usually old and heavily patched. For example, in 2013 we still used Python 2.5.2 with a lot of patches. For example, patches for cross compilation. But also backported features like HTTP Keep-Alive or HTTPS checks.

Technically, you can easily fork Python repository and keep your patches up to date using rebase.

We defined better rules to support officially a platform. A requirement is for example to have a buildbot running tests daily on the platform. I'm not sure that you are able to host a public buildbot for each custom embedded platform...

Android looks a more important target and it was already discussed to setup a Android buildbot.

I'm in favor of *dropping* all these annoying #ifdef because it's harder to review, maintain and debug such code. Here is my contribution: issue #23753 "Drop HAVE_FSTAT: require fstat() to compile/use Python". A concrete example: we are working on a Python implementation of io.FileIO and I don't want to see hasattr(os, 'fstat') in the Python code because it has a performance cost *at runtime* (see issue #21859).

See also the new Mobile-SIG mailing list which is maybe more appropriate to discuss such changes:
https://mail.python.org/mailman/listinfo/mobile-sig
msg239096 - (view) Author: (Link Mauve) Date: 2015-03-24 08:11
On Mon, Mar 23, 2015 at 09:05:20PM +0000, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> > Many POSIX functions aren’t available on every system, especially embedded ones.
> 
> What do you call "embedded systems"? I worked on set top boxes (the box to watch television) between 2011 and 2013 and we had a regular Linux kernel with all POSIX functions. Most of the time, the CPU was a slow MIPS. The tool chain was based on GCC. For some hardware, we used the µlibc but this C library provides all functions required by Python.

My target platforms have been the Newlib-based Wii and 3DS homebrew toolchains, which emulate a POSIX system on top of the raw hardware (or in the case of the 3DS, a proprietary micro-kernel.

> 
> > Personally, I'd rather answer that it's not our problem if some systems are not POSIX-compliant. Maintainers of such systems can always maintain a patch to disable the missing functionality. Adding conditionals everywhere has a non-trivial maintenance cost.
> 
> I agree. Anyway, in the embedded world, softwares are usually old and heavily patched. For example, in 2013 we still used Python 2.5.2 with a lot of patches. For example, patches for cross compilation. But also backported features like HTTP Keep-Alive or HTTPS checks.

There is a port of Python 2.5.something for the Wii, terribly outdated, and this is why I wanted to merge the non-specifics parts of my port upstream, as I thought it would help everyone in that case.

> 
> Technically, you can easily fork Python repository and keep your patches up to date using rebase.
> 
> We defined better rules to support officially a platform. A requirement is for example to have a buildbot running tests daily on the platform. I'm not sure that you are able to host a public buildbot for each custom embedded platform...

I can provide such a buildbot on my spare Wii.  Not really for the 3DS yet, as the homebrew scene is much younger there and unrecoverable lockups happen.

> 
> Android looks a more important target and it was already discussed to setup a Android buildbot.
> 
> I'm in favor of *dropping* all these annoying #ifdef because it's harder to review, maintain and debug such code. Here is my contribution: issue #23753 "Drop HAVE_FSTAT: require fstat() to compile/use Python". A concrete example: we are working on a Python implementation of io.FileIO and I don't want to see hasattr(os, 'fstat') in the Python code because it has a performance cost *at runtime* (see issue #21859).

On those two platforms, fstat() is correctly defined and works fine, so it shouldn’t be a problem to drop its #ifdef.

> 
> See also the new Mobile-SIG mailing list which is maybe more appropriate to discuss such changes:
> https://mail.python.org/mailman/listinfo/mobile-sig
> 
> ----------
> nosy: +haypo
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22623>
> _______________________________________
msg239100 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-24 09:15
> My target platforms have been the Newlib-based Wii and 3DS homebrew toolchains, which emulate a POSIX system on top of the raw hardware (or in the case of the 3DS, a proprietary micro-kernel.

What do you think of my idea of maintaining a fork of the Python repository to store your patches, and rebase them sometimes?

Would it be an acceptable solution?
History
Date User Action Args
2022-04-11 14:58:09adminsetstatus: pending -> open
github: 66813
2016-01-03 06:41:09ezio.melottisetstatus: open -> pending
versions: + Python 3.6, - Python 3.4
2015-03-24 09:20:26ezio.melottisetnosy: + ezio.melotti
2015-03-24 09:15:16vstinnersetmessages: + msg239100
2015-03-24 09:14:10vstinnersettitle: Missing guards for some POSIX functions -> Port Python to 3DS: micro kernel, homebrew, newlib (Missing guards for some POSIX functions)
2015-03-24 08:11:50Link Mauvesetmessages: + msg239096
2015-03-23 21:05:20vstinnersetnosy: + vstinner
messages: + msg239067
2015-03-23 20:50:09pitrousetnosy: + loewis, pitrou
messages: + msg239065
2015-02-22 18:49:39serhiy.storchakasetmessages: + msg236414
2015-02-18 19:38:21Link Mauvesetfiles: + getedig.patch
2015-02-18 19:19:07Link Mauvesetfiles: + getedig.patch
2015-02-17 19:28:56Link Mauvesetfiles: + getedig.patch
2015-02-17 19:24:48serhiy.storchakasetnosy: + serhiy.storchaka

messages: + msg236146
versions: - Python 3.2, Python 3.3
2015-02-17 18:45:11Link Mauvesetfiles: + getedig.patch

messages: + msg236142
2015-02-02 00:54:24Link Mauvesetfiles: - f3cf19e38efe.diff
2015-02-02 00:54:04Link Mauvesetfiles: + guards.diff

messages: + msg235214
components: + Extension Modules, - Build
versions: + Python 2.7, Python 3.2, Python 3.3, Python 3.4
2014-10-13 12:46:43Link Mauvesetfiles: + f3cf19e38efe.diff
keywords: + patch
2014-10-13 12:44:03Link Mauvecreate