classification
Title: Automatically regenerate platform-specific modules
Type: Stage: resolved
Components: Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder: Remove Lib/plat-*/* files
View: 28027
Assigned To: Nosy List: Arfrever, Neurogeek, Ramchandra Apte, amaury.forgeotdarc, djc, doko, ezio.melotti, haypo, jwilk, lemburg, loewis, merwok, ned.deily, neologix, pitrou, python-dev, r.david.murray, rosslagerwall, rpetrov, zach.ware
Priority: normal Keywords: patch

Created on 2011-07-23 09:43 by Arfrever, last changed 2016-09-09 16:41 by zach.ware. This issue is now closed.

Files
File name Uploaded Description Edit
python-regenerate_platdir.patch Arfrever, 2011-10-16 23:32
Messages (27)
msg140952 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-07-23 09:43
I suggest that platform-specific modules be automatically regenerated during installation. I'm attaching a patch.
msg140957 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-07-23 09:47
(Autotools/make newbie here) Why during install and not build?
msg140958 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-07-23 09:49
I don't care when they will be regenerated.
msg140962 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2011-07-23 10:33
is auto-generation wanted?
are you sure that you can't end up with bad syntax?
msg140969 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-07-23 10:48
py_compile.compile() can be used to verify syntax.
msg140984 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-07-23 12:55
The new patch creates platform-specific modules at build time and verifies their syntax.
msg145653 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-17 00:08
> is auto-generation wanted?

For example, Lib/plat-linux/IN.py was regenerated at Nov 23 12:09:28 2002 (revision c2604d69aa5d) for the last time, and nobody complained. These modules are inconsistent. For example, the IN module contains constants of 64 bits platforms, whereas it can be used on 32 bits platforms.

In my opinion, we should just drop these modules with h2py.py, I (re)opened a thread on python-dev.
msg145659 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-10-17 05:28
-1 on auto-building. The header needed may not be available on the build platform, if it is not normally needed to build Python.
msg145667 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-10-17 11:40
Regeneration of platform-specific modules fixes concerns about their outdateness and architecture differences (32-bit vs 64-bit, big endian vs little endian).

Regeneration of given module could be performed only when corresponding header is available.

Eventually regeneration of platform-specific modules could be controlled by an option of `configure`.
msg145759 - (view) Author: Roumen Petrov (rpetrov) * Date: 2011-10-17 21:14
Related :  #1565071 and #3990  . There is no reason to keep plat-xxx files if cannot be managed properly.
msg145762 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-10-17 22:02
What do you do for platforms like OS X where we support one set of binary files that contain multi-architecture C-files that can run as Intel-64, Intel-32 or PPC-32 on the same machine at user option at run time?  For example, the Apple-suppled system Python on OS X 10.6 has all three and you can use `arch` to specify which to run (PPC is emulated on the Intel machines supported by 10.6):

$ file /usr/bin/python2.6
/usr/bin/python2.6: Mach-O universal binary with 3 architectures
/usr/bin/python2.6 (for architecture x86_64):	Mach-O 64-bit executable x86_64
/usr/bin/python2.6 (for architecture i386):	Mach-O executable i386
/usr/bin/python2.6 (for architecture ppc7400):	Mach-O executable ppc

The static IN.py currently shipped in plat-darwin is misleading at best. And you can't improve the situation by regenerating at installation time.
msg145948 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-10-19 19:50
> Related :  #1565071 and #3990  . There is no reason to keep plat-xxx files if cannot be managed properly.

+1
msg145953 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2011-10-19 20:17
I don't see why these modules should be auto-generated. The constants
in the modules hardly ever change and are also not affected by architecture
differences (e.g. Mac OS X, Solaris, etc.) AFAICT.

If you think they need to be auto-generated, you should make a case by
example.

Note that we cannot simply drop the modules. Some of the constants are
needed for e.g. socket, ctypes or ldd programming.
msg145956 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-19 20:32
> you should make a case by example

Did you read comments of this issue and my email thread on python-dev? There are differents examples:

 - LONG_MAX is 9223372036854775807 even on 32 bits system
 - On Mac OS X, FAT programs contains 32 and 64 binaries, whereas constants are changed for 32 or 64 bits

> we cannot simply drop the modules. Some of the constants
> are needed for e.g. socket, ctypes or ldd programming.

Ah? I removed all plat-* directories and ran the full test suite: no failure. The Python socket modules contain many constants (SOCK_*, AF_*,  SO_*, ...):
http://docs.python.org/library/socket.html#socket.AF_UNIX

Which constants are used by the ctypes modules or can be used by modules using ctypes? Can you give examples? I listed usages of plat-* modules in the first message of my thread on python-dev.

By "ldd", you mean "ld.so" (dlopen)? Yes, I agree that we need to expose  dl constants. But the other constants are not used.
msg145958 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-10-19 20:40
Victor, please accept that this entire infrastructure was originally added because there was a need for it, and the need did not go away. It's probably relevant only for a minority of applications, but you would make this minority's lifes much more complicated by removing the modules. If they don't hurt, why remove them?
msg145961 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-19 20:41
> If they don't hurt, why remove them?

Bogus constants don't hurt?
msg145970 - (view) Author: Jakub Wilk (jwilk) Date: 2011-10-19 21:34
FYI, in Debian we have at least:
one package using the CDROM module,
3 packages using the IN module,
14 packages using the DLFCN module.
msg145971 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2011-10-19 21:40
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> you should make a case by example
> 
> Did you read comments of this issue and my email thread on python-dev?

No.

> There are differents examples:
> 
>  - LONG_MAX is 9223372036854775807 even on 32 bits system
>  - On Mac OS X, FAT programs contains 32 and 64 binaries, whereas constants are changed for 32 or 64 bits

That's because the h2py.py script doesn't know anything about conditional
definitions, e.g.

PTRDIFF_MIN = (-9223372036854775807-1)
PTRDIFF_MAX = (9223372036854775807)
PTRDIFF_MIN = (-2147483647-1)
PTRDIFF_MAX = (2147483647)

Not all constants are really useful because of this, but some are, e.g.
INT16_MAX, INT32_MAX, etc.

>> we cannot simply drop the modules. Some of the constants
>> are needed for e.g. socket, ctypes or ldd programming.
> 
> Ah? I removed all plat-* directories and ran the full test suite: no failure. 

Right, the modules are not tested at all, AFAIK.

> The Python socket modules contain many constants (SOCK_*, AF_*,  SO_*, ...):
> http://docs.python.org/library/socket.html#socket.AF_UNIX

True, but you probably agree that it's easier to parse a header file
into a Python module than to add each and every socket option on
the planet to the C socket module, don't you ? :-)

> Which constants are used by the ctypes modules or can be used by modules using ctypes? Can you give examples? I listed usages of plat-* modules in the first message of my thread on python-dev.

Not constants used by the ctypes, but constants which can be used
with the ctypes module.

> By "ldd", you mean "ld.so" (dlopen)?

Yes.

> Yes, I agree that we need to expose  dl constants. But the other constants are not used.

Not in the standard lib, that's true.

Also note that the plat-* directories can contain platform specific code,
e.g. the OS2 dirs have replacements for the pwd and grp modules.

Finally, the list of standard files to include in those directories
could be extended to cover more system level constants such as
the ioctl or fcntl constants (not only the few defined in the C
code, but all platform specific ones).
msg145977 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-19 22:06
I created the issue #13226 to provide RTLD_* constants on all platforms (not only on Linux and sunos5).
msg145981 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-19 22:24
> > There are differents examples:
> >  - LONG_MAX is 9223372036854775807 even on 32 bits system
> >  - On Mac OS X, FAT programs contains 32 and 64 binaries, whereas
> >  constants are changed for 32 or 64 bits
> 
> That's because the h2py.py script doesn't know anything about conditional
> definitions, e.g.
> 
> PTRDIFF_MIN = (-9223372036854775807-1)
> PTRDIFF_MAX = (9223372036854775807)
> PTRDIFF_MIN = (-2147483647-1)
> PTRDIFF_MAX = (2147483647)

Why may fix h2py.py, but regenerating plat-*/*.py files on build is not an 
option according to Martin (msg145659).

And it doesn't solve the Mac OS X issue.

> Not all constants are really useful because of this, but some are, e.g.
> INT16_MAX, INT32_MAX, etc.

INT16_MAX and INT32_MAX are not platform dependent. I'm not sure that Python 
should provide such constants.

> Right, the modules are not tested at all, AFAIK.

One more reason to remove them. By the way, there are not documented.

> > The Python socket modules contain many constants (SOCK_*, AF_*,  SO_*,
> > ...): http://docs.python.org/library/socket.html#socket.AF_UNIX
> 
> True, but you probably agree that it's easier to parse a header file
> into a Python module than to add each and every socket option on
> the planet to the C socket module, don't you ? :-)

It's not exactly the purpose of this issue. We are first trying to get correct 
constants.

I don't think that it's really useful to expose all constants. I prefer to 
expose more constants on demand. When we add new constants, we try to use it, 
test it and document it (e.g. O_CLOEXEC).

> > Which constants are used by the ctypes modules or can be used by modules
> > using ctypes? Can you give examples? I listed usages of plat-* modules
> > in the first message of my thread on python-dev.
> 
> Not constants used by the ctypes, but constants which can be used
> with the ctypes module.

Sorry, I don't see which constants are useful with the ctypes module?

> > By "ldd", you mean "ld.so" (dlopen)?
> 
> Yes.

See my issue #13226 for these constants.

> Also note that the plat-* directories can contain platform specific code,
> e.g. the OS2 dirs have replacements for the pwd and grp modules.

(OS/2 has been deprecated in Python 3.3, see the PEP 11.)

Except plat-os2emx, I don't see other platform specific code.
msg145987 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-19 22:55
> FYI, in Debian we have at least:
> one package using the CDROM module,

Is it cdsuite? (http://offog.org/code/cdsuite.html)

> 3 packages using the IN module,

I found policykit (it uses IN.INT_MAX). What are the 2 others? Which constants are used?

> 14 packages using the DLFCN module.

Oh, so many? Which projects? Which constants are used?
msg146021 - (view) Author: Jakub Wilk (jwilk) Date: 2011-10-20 15:55
* STINNER Victor <report@bugs.python.org>, 2011-10-19, 22:55:
>>FYI, in Debian we have at least:
>>one package using the CDROM module,
>Is it cdsuite? (http://offog.org/code/cdsuite.html)

No, Freevo <http://freevo.sourceforge.net/>.

>>3 packages using the IN module,
>I found policykit (it uses IN.INT_MAX).

Not in Debian, apparently...

>What are the 2 others? Which constants are used?

* Ganeti <http://code.google.com/p/ganeti/>: SO_PEERCRED;
* Mandos <http://www.recompile.se/mandos>: SO_BINDTODEVICE;
* Pydhcplib <http://pydhcplib.tuxfamily.org/>: SO_BINDTODEVICE.

>> 14 packages using the DLFCN module.
>
>Oh, so many? Which projects? Which constants are used?

* Boost <http://www.boost.org/>: RTLD_NOW, RTLD_GLOBAL;
* Ecasound <http://www.eca.cx/ecasound/>: RTLD_LAZY, RTLD_GLOBAL;
* GDCM <http://gdcm.sourceforge.net/>: RTLD_LAZY, RTLD_GLOBAL;
* GNU Radio <http://gnuradio.org/>: RTLD_GLOBAL;
* GStreamer <http://gstreamer.freedesktop.org/>: RTLD_GLOBAL, RTLD_LAZY;
* MPI for Python <http://code.google.com/p/mpi4py/>: RTLD_NOW, RTLD_GLOBAL;
* Open Babel <http://openbabel.org>: RTLD_GLOBAL;
* PyTrilinos <http://trilinos.sandia.gov/>: RTLD_NOW, RTLD_GLOBAL;
* SALOME <http://www.salome-platform.org/>: RTLD_NOW, RTLD_GLOBAL;
* VTK <http://www.vtk.org/>: RTLD_NOW, RTLD_GLOBAL;
* Yade <https://launchpad.net/yade>: RTLD_NOW, RTLD_GLOBAL;
* pyOpenSSL <http://launchpad.net/pyopenssl>: RTLD_NOW, RTLD_GLOBAL.

(That's only 12; we have two versions of Ecasound in the archive, and 
two binary packages with DLFCN-using code are produced from GDCM 
sources.)
msg146027 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-20 16:43
For Freevo: yes, it uses the CDROM module. But this module doesn't look to be perfect because Freevo has a fallback for FreeBSD and another for... Linux (because of missing CDROM.CDROM_DRIVE_STATUS?):
-----------------------------
try:
    from CDROM import *
    # test if CDROM_DRIVE_STATUS is there
    # (for some strange reason, this is missing sometimes)
    CDROM_DRIVE_STATUS
except:
    if os.uname()[0] == 'FreeBSD':
        # FreeBSD ioctls - there is no CDROM.py...
        CDIOCEJECT = 0x20006318
        CDIOCCLOSE = 0x2000631c
        CDIOREADTOCENTRYS = 0xc0086305
        CD_LBA_FORMAT = 1
        CD_MSF_FORMAT = 2
        CDS_NO_DISC = 1
        CDS_DISC_OK = 4
    else:
        # see linux/cdrom.h and Documentation/ioctl/cdrom.txt
        CDROMEJECT           = 0x5309
        CDROM_GET_CAPABILITY = 0x5331
        CDROMCLOSETRAY       = 0x5319  # pendant of CDROMEJECT
        CDROM_SET_OPTIONS    = 0x5320  # Set behavior options
        CDROM_CLEAR_OPTIONS  = 0x5321  # Clear behavior options
        CDROM_SELECT_SPEED   = 0x5322  # Set the CD-ROM speed
        CDROM_SELECT_DISC    = 0x5323  # Select disc (for juke-boxes)
        CDROM_MEDIA_CHANGED  = 0x5325  # Check is media changed
        CDROM_DRIVE_STATUS   = 0x5326  # Get tray position, etc.
        CDROM_DISC_STATUS    = 0x5327  # Get disc type, etc.
        CDROM_CHANGER_NSLOTS = 0x5328  # Get number of slots
        CDROM_LOCKDOOR       = 0x5329  # lock or unlock door
        CDROM_DEBUG          = 0x5330  # Turn debug messages on/off
        CDROM_GET_CAPABILITY = 0x5331  # get capabilities
        # CDROM_DRIVE_STATUS
        CDS_NO_INFO = 0
        CDS_NO_DISC = 1
        CDS_TRAY_OPEN = 2
        CDS_DRIVE_NOT_READY = 3
        CDS_DISC_OK = 4
        # capability flags
        CDC_CLOSE_TRAY       = 0x1     # caddy systems _can't_ close
        CDC_OPEN_TRAY        = 0x2     # but _can_ eject.
        CDC_LOCK             = 0x4     # disable manual eject
        CDC_SELECT_SPEED     = 0x8     # programmable speed
        CDC_SELECT_DISC      = 0x10    # select disc from juke-box
CDC_MO_DRIVE         = 0x40000
CDC_MRW              = 0x80000
CDC_MRW_W            = 0x100000
CDC_RAM              = 0x200000
# CDROM_DISC_STATUS
CDS_AUDIO = 100
CDS_DATA_1 = 101
CDS_DATA_2 = 102
CDS_XA_2_1 = 103
CDS_XA_2_2 = 104
CDS_MIXED = 105
-----------------------------

So Freevo does stil work if we remove the CDROM module. I still think that the Python standard library is not the right place for such constants. A third party module providing an API would be a better idea.

socket.SO_PEERCRED has been added to Python 3.3 by c64216addd7f:

"Add support for the send/recvmsg API to the socket module. Patch by David Watson and Heiko Wundram. (Closes #6560)"

socket.SO_PEERCRED is not listed in socket documentation.

socket has no SO_BINDTODEVICE constant yet.

RTLD_NOW, RTLD_GLOBAL and RTLD_LAZY will added to the posix module if my issue #13226 is accepted.
msg146360 - (view) Author: Roundup Robot (python-dev) Date: 2011-10-25 11:45
New changeset 6159311f0f44 by Victor Stinner in branch 'default':
Issue #12619: Expose socket.SO_BINDTODEVICE constant
http://hg.python.org/cpython/rev/6159311f0f44
msg156157 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2012-03-17 13:23
+1 for this.
msg202627 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-11-11 13:37
See also #19554.
msg275335 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-09 16:41
The platform-specific modules have been removed.
History
Date User Action Args
2016-09-09 16:41:42zach.waresetstatus: open -> closed

superseder: Remove Lib/plat-*/* files

nosy: + zach.ware
messages: + msg275335
resolution: out of date
stage: resolved
2014-05-14 18:09:26skrahsetnosy: - skrah
2013-11-11 13:37:25skrahsetnosy: + skrah
messages: + msg202627
2013-11-11 13:32:23skrahlinkissue19554 dependencies
2012-03-17 13:23:35Ramchandra Aptesetmessages: + msg156157
2011-10-25 11:45:55python-devsetmessages: + msg146360
2011-10-20 16:43:39hayposetmessages: + msg146027
2011-10-20 15:55:59jwilksetmessages: + msg146021
2011-10-20 07:10:01petri.lehtinensetnosy: - petri.lehtinen
2011-10-19 22:55:13hayposetmessages: + msg145987
2011-10-19 22:24:56hayposetmessages: + msg145981
2011-10-19 22:06:45hayposetmessages: + msg145977
2011-10-19 21:40:23lemburgsetmessages: + msg145971
2011-10-19 21:34:24jwilksetmessages: + msg145970
2011-10-19 20:41:15pitrousetmessages: + msg145961
2011-10-19 20:40:19loewissetmessages: + msg145958
2011-10-19 20:32:10hayposetmessages: + msg145956
2011-10-19 20:17:42lemburgsetmessages: + msg145953
2011-10-19 19:50:27neologixsetmessages: + msg145948
2011-10-18 16:25:00merwoklinkissue3990 dependencies
2011-10-18 16:20:55merwoklinkissue1565071 dependencies
2011-10-17 22:02:10ned.deilysetnosy: + ned.deily
messages: + msg145762
2011-10-17 21:14:44rpetrovsetnosy: + rpetrov
messages: + msg145759
2011-10-17 11:40:23Arfreversetmessages: + msg145667
2011-10-17 05:28:32loewissetmessages: + msg145659
2011-10-17 00:08:00hayposetmessages: + msg145653
2011-10-16 23:32:51Arfreversetfiles: + python-regenerate_platdir.patch
2011-10-16 23:31:32Arfreversetfiles: - python-regenerate_platdir.patch
2011-07-28 20:11:58Neurogeeksetnosy: + Neurogeek
2011-07-23 12:56:18Arfreversetfiles: - python-regenerate_platdir.patch
2011-07-23 12:56:00Arfreversetfiles: + python-regenerate_platdir.patch

messages: + msg140984
2011-07-23 10:48:15Arfreversetmessages: + msg140969
2011-07-23 10:33:35dokosetmessages: + msg140962
2011-07-23 09:49:49Arfreversetmessages: + msg140958
2011-07-23 09:47:06merwoksetmessages: + msg140957
2011-07-23 09:43:32Arfrevercreate