I suggest that platform-specific modules be automatically regenerated during installation. I'm attaching a patch.

(Autotools/make newbie here) Why during install and not build?

I don't care when they will be regenerated.

is auto-generation wanted?
are you sure that you can't end up with bad syntax?

py_compile.compile() can be used to verify syntax.

The new patch creates platform-specific modules at build time and verifies their syntax.

> is auto-generation wanted?

For example, Lib/plat-linux/IN.py was regenerated at Nov 23 12:09:28 2002 (revision c2604d69aa5d) for the last time, and nobody complained. These modules are inconsistent. For example, the IN module contains constants of 64 bits platforms, whereas it can be used on 32 bits platforms.

In my opinion, we should just drop these modules with h2py.py, I (re)opened a thread on python-dev.

-1 on auto-building. The header needed may not be available on the build platform, if it is not normally needed to build Python.

Regeneration of platform-specific modules fixes concerns about their outdateness and architecture differences (32-bit vs 64-bit, big endian vs little endian).

Regeneration of given module could be performed only when corresponding header is available.

Eventually regeneration of platform-specific modules could be controlled by an option of `configure`.

Related :  #1565071 and #3990  . There is no reason to keep plat-xxx files if cannot be managed properly.

What do you do for platforms like OS X where we support one set of binary files that contain multi-architecture C-files that can run as Intel-64, Intel-32 or PPC-32 on the same machine at user option at run time?  For example, the Apple-suppled system Python on OS X 10.6 has all three and you can use `arch` to specify which to run (PPC is emulated on the Intel machines supported by 10.6):

$ file /usr/bin/python2.6
/usr/bin/python2.6: Mach-O universal binary with 3 architectures
/usr/bin/python2.6 (for architecture x86_64):	Mach-O 64-bit executable x86_64
/usr/bin/python2.6 (for architecture i386):	Mach-O executable i386
/usr/bin/python2.6 (for architecture ppc7400):	Mach-O executable ppc

The static IN.py currently shipped in plat-darwin is misleading at best. And you can't improve the situation by regenerating at installation time.

> Related :  #1565071 and #3990  . There is no reason to keep plat-xxx files if cannot be managed properly.

+1

I don't see why these modules should be auto-generated. The constants
in the modules hardly ever change and are also not affected by architecture
differences (e.g. Mac OS X, Solaris, etc.) AFAICT.

If you think they need to be auto-generated, you should make a case by
example.

Note that we cannot simply drop the modules. Some of the constants are
needed for e.g. socket, ctypes or ldd programming.

> you should make a case by example

Did you read comments of this issue and my email thread on python-dev? There are differents examples:

 - LONG_MAX is 9223372036854775807 even on 32 bits system
 - On Mac OS X, FAT programs contains 32 and 64 binaries, whereas constants are changed for 32 or 64 bits

> we cannot simply drop the modules. Some of the constants
> are needed for e.g. socket, ctypes or ldd programming.

Ah? I removed all plat-* directories and ran the full test suite: no failure. The Python socket modules contain many constants (SOCK_*, AF_*,  SO_*, ...):
http://docs.python.org/library/socket.html#socket.AF_UNIX

Which constants are used by the ctypes modules or can be used by modules using ctypes? Can you give examples? I listed usages of plat-* modules in the first message of my thread on python-dev.

By "ldd", you mean "ld.so" (dlopen)? Yes, I agree that we need to expose  dl constants. But the other constants are not used.

Victor, please accept that this entire infrastructure was originally added because there was a need for it, and the need did not go away. It's probably relevant only for a minority of applications, but you would make this minority's lifes much more complicated by removing the modules. If they don't hurt, why remove them?

> If they don't hurt, why remove them?

Bogus constants don't hurt?

FYI, in Debian we have at least:
one package using the CDROM module,
3 packages using the IN module,
14 packages using the DLFCN module.

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> you should make a case by example
> 
> Did you read comments of this issue and my email thread on python-dev?

No.

> There are differents examples:
> 
>  - LONG_MAX is 9223372036854775807 even on 32 bits system
>  - On Mac OS X, FAT programs contains 32 and 64 binaries, whereas constants are changed for 32 or 64 bits

That's because the h2py.py script doesn't know anything about conditional
definitions, e.g.

PTRDIFF_MIN = (-9223372036854775807-1)
PTRDIFF_MAX = (9223372036854775807)
PTRDIFF_MIN = (-2147483647-1)
PTRDIFF_MAX = (2147483647)

Not all constants are really useful because of this, but some are, e.g.
INT16_MAX, INT32_MAX, etc.

>> we cannot simply drop the modules. Some of the constants
>> are needed for e.g. socket, ctypes or ldd programming.
> 
> Ah? I removed all plat-* directories and ran the full test suite: no failure. 

Right, the modules are not tested at all, AFAIK.

> The Python socket modules contain many constants (SOCK_*, AF_*,  SO_*, ...):
> http://docs.python.org/library/socket.html#socket.AF_UNIX

True, but you probably agree that it's easier to parse a header file
into a Python module than to add each and every socket option on
the planet to the C socket module, don't you ? :-)

> Which constants are used by the ctypes modules or can be used by modules using ctypes? Can you give examples? I listed usages of plat-* modules in the first message of my thread on python-dev.

Not constants used by the ctypes, but constants which can be used
with the ctypes module.

> By "ldd", you mean "ld.so" (dlopen)?

Yes.

> Yes, I agree that we need to expose  dl constants. But the other constants are not used.

Not in the standard lib, that's true.

Also note that the plat-* directories can contain platform specific code,
e.g. the OS2 dirs have replacements for the pwd and grp modules.

Finally, the list of standard files to include in those directories
could be extended to cover more system level constants such as
the ioctl or fcntl constants (not only the few defined in the C
code, but all platform specific ones).

I created the issue #13226 to provide RTLD_* constants on all platforms (not only on Linux and sunos5).

> > There are differents examples:
> >  - LONG_MAX is 9223372036854775807 even on 32 bits system
> >  - On Mac OS X, FAT programs contains 32 and 64 binaries, whereas
> >  constants are changed for 32 or 64 bits
> 
> That's because the h2py.py script doesn't know anything about conditional
> definitions, e.g.
> 
> PTRDIFF_MIN = (-9223372036854775807-1)
> PTRDIFF_MAX = (9223372036854775807)
> PTRDIFF_MIN = (-2147483647-1)
> PTRDIFF_MAX = (2147483647)

Why may fix h2py.py, but regenerating plat-*/*.py files on build is not an 
option according to Martin (msg145659).

And it doesn't solve the Mac OS X issue.

> Not all constants are really useful because of this, but some are, e.g.
> INT16_MAX, INT32_MAX, etc.

INT16_MAX and INT32_MAX are not platform dependent. I'm not sure that Python 
should provide such constants.

> Right, the modules are not tested at all, AFAIK.

One more reason to remove them. By the way, there are not documented.

> > The Python socket modules contain many constants (SOCK_*, AF_*,  SO_*,
> > ...): http://docs.python.org/library/socket.html#socket.AF_UNIX
> 
> True, but you probably agree that it's easier to parse a header file
> into a Python module than to add each and every socket option on
> the planet to the C socket module, don't you ? :-)

It's not exactly the purpose of this issue. We are first trying to get correct 
constants.

I don't think that it's really useful to expose all constants. I prefer to 
expose more constants on demand. When we add new constants, we try to use it, 
test it and document it (e.g. O_CLOEXEC).

> > Which constants are used by the ctypes modules or can be used by modules
> > using ctypes? Can you give examples? I listed usages of plat-* modules
> > in the first message of my thread on python-dev.
> 
> Not constants used by the ctypes, but constants which can be used
> with the ctypes module.

Sorry, I don't see which constants are useful with the ctypes module?

> > By "ldd", you mean "ld.so" (dlopen)?
> 
> Yes.

See my issue #13226 for these constants.

> Also note that the plat-* directories can contain platform specific code,
> e.g. the OS2 dirs have replacements for the pwd and grp modules.

(OS/2 has been deprecated in Python 3.3, see the PEP 11.)

Except plat-os2emx, I don't see other platform specific code.

> FYI, in Debian we have at least:
> one package using the CDROM module,

Is it cdsuite? (http://offog.org/code/cdsuite.html)

> 3 packages using the IN module,

I found policykit (it uses IN.INT_MAX). What are the 2 others? Which constants are used?

> 14 packages using the DLFCN module.

Oh, so many? Which projects? Which constants are used?

* STINNER Victor <report@bugs.python.org>, 2011-10-19, 22:55:
>>FYI, in Debian we have at least:
>>one package using the CDROM module,
>Is it cdsuite? (http://offog.org/code/cdsuite.html)

No, Freevo <http://freevo.sourceforge.net/>.

>>3 packages using the IN module,
>I found policykit (it uses IN.INT_MAX).

Not in Debian, apparently...

>What are the 2 others? Which constants are used?

* Ganeti <http://code.google.com/p/ganeti/>: SO_PEERCRED;
* Mandos <http://www.recompile.se/mandos>: SO_BINDTODEVICE;
* Pydhcplib <http://pydhcplib.tuxfamily.org/>: SO_BINDTODEVICE.

>> 14 packages using the DLFCN module.
>
>Oh, so many? Which projects? Which constants are used?

* Boost <http://www.boost.org/>: RTLD_NOW, RTLD_GLOBAL;
* Ecasound <http://www.eca.cx/ecasound/>: RTLD_LAZY, RTLD_GLOBAL;
* GDCM <http://gdcm.sourceforge.net/>: RTLD_LAZY, RTLD_GLOBAL;
* GNU Radio <http://gnuradio.org/>: RTLD_GLOBAL;
* GStreamer <http://gstreamer.freedesktop.org/>: RTLD_GLOBAL, RTLD_LAZY;
* MPI for Python <http://code.google.com/p/mpi4py/>: RTLD_NOW, RTLD_GLOBAL;
* Open Babel <http://openbabel.org>: RTLD_GLOBAL;
* PyTrilinos <http://trilinos.sandia.gov/>: RTLD_NOW, RTLD_GLOBAL;
* SALOME <http://www.salome-platform.org/>: RTLD_NOW, RTLD_GLOBAL;
* VTK <http://www.vtk.org/>: RTLD_NOW, RTLD_GLOBAL;
* Yade <https://launchpad.net/yade>: RTLD_NOW, RTLD_GLOBAL;
* pyOpenSSL <http://launchpad.net/pyopenssl>: RTLD_NOW, RTLD_GLOBAL.

(That's only 12; we have two versions of Ecasound in the archive, and 
two binary packages with DLFCN-using code are produced from GDCM 
sources.)

For Freevo: yes, it uses the CDROM module. But this module doesn't look to be perfect because Freevo has a fallback for FreeBSD and another for... Linux (because of missing CDROM.CDROM_DRIVE_STATUS?):
-----------------------------
try:
    from CDROM import *
    # test if CDROM_DRIVE_STATUS is there
    # (for some strange reason, this is missing sometimes)
    CDROM_DRIVE_STATUS
except:
    if os.uname()[0] == 'FreeBSD':
        # FreeBSD ioctls - there is no CDROM.py...
        CDIOCEJECT = 0x20006318
        CDIOCCLOSE = 0x2000631c
        CDIOREADTOCENTRYS = 0xc0086305
        CD_LBA_FORMAT = 1
        CD_MSF_FORMAT = 2
        CDS_NO_DISC = 1
        CDS_DISC_OK = 4
    else:
        # see linux/cdrom.h and Documentation/ioctl/cdrom.txt
        CDROMEJECT           = 0x5309
        CDROM_GET_CAPABILITY = 0x5331
        CDROMCLOSETRAY       = 0x5319  # pendant of CDROMEJECT
        CDROM_SET_OPTIONS    = 0x5320  # Set behavior options
        CDROM_CLEAR_OPTIONS  = 0x5321  # Clear behavior options
        CDROM_SELECT_SPEED   = 0x5322  # Set the CD-ROM speed
        CDROM_SELECT_DISC    = 0x5323  # Select disc (for juke-boxes)
        CDROM_MEDIA_CHANGED  = 0x5325  # Check is media changed
        CDROM_DRIVE_STATUS   = 0x5326  # Get tray position, etc.
        CDROM_DISC_STATUS    = 0x5327  # Get disc type, etc.
        CDROM_CHANGER_NSLOTS = 0x5328  # Get number of slots
        CDROM_LOCKDOOR       = 0x5329  # lock or unlock door
        CDROM_DEBUG          = 0x5330  # Turn debug messages on/off
        CDROM_GET_CAPABILITY = 0x5331  # get capabilities
        # CDROM_DRIVE_STATUS
        CDS_NO_INFO = 0
        CDS_NO_DISC = 1
        CDS_TRAY_OPEN = 2
        CDS_DRIVE_NOT_READY = 3
        CDS_DISC_OK = 4
        # capability flags
        CDC_CLOSE_TRAY       = 0x1     # caddy systems _can't_ close
        CDC_OPEN_TRAY        = 0x2     # but _can_ eject.
        CDC_LOCK             = 0x4     # disable manual eject
        CDC_SELECT_SPEED     = 0x8     # programmable speed
        CDC_SELECT_DISC      = 0x10    # select disc from juke-box
CDC_MO_DRIVE         = 0x40000
CDC_MRW              = 0x80000
CDC_MRW_W            = 0x100000
CDC_RAM              = 0x200000
# CDROM_DISC_STATUS
CDS_AUDIO = 100
CDS_DATA_1 = 101
CDS_DATA_2 = 102
CDS_XA_2_1 = 103
CDS_XA_2_2 = 104
CDS_MIXED = 105
-----------------------------

So Freevo does stil work if we remove the CDROM module. I still think that the Python standard library is not the right place for such constants. A third party module providing an API would be a better idea.

socket.SO_PEERCRED has been added to Python 3.3 by c64216addd7f:

"Add support for the send/recvmsg API to the socket module. Patch by David Watson and Heiko Wundram. (Closes #6560)"

socket.SO_PEERCRED is not listed in socket documentation.

socket has no SO_BINDTODEVICE constant yet.

RTLD_NOW, RTLD_GLOBAL and RTLD_LAZY will added to the posix module if my issue #13226 is accepted.

New changeset 6159311f0f44 by Victor Stinner in branch 'default':
Issue #12619: Expose socket.SO_BINDTODEVICE constant
http://hg.python.org/cpython/rev/6159311f0f44

+1 for this.

See also #19554.

The platform-specific modules have been removed.