This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: _multiprocessing.c warnings under 64-bit Windows
Type: compile error Stage: resolved
Components: Extension Modules, Windows Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: davin, janglin, jnoller, loewis, pitrou, serhiy.storchaka, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2010-09-06 12:43 by pitrou, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (4)
msg115699 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-06 12:43
There are various warnings when compiling the _multiprocessing extension in 64-bit mode under Windows. Many seem related to the fact that read() and friends under Windows take "int" size arguments rather than "size_t".

20>------ Build started: Project: _multiprocessing, Configuration: Release x64 ------
20>Compiling...
20>multiprocessing.c
20>win32_functions.c
20>socket_connection.c
20>..\Modules\_multiprocessing\socket_connection.c(32) : warning C4267: 'function' : conversion from 'size_t' to 'int', possible loss of data
20>..\Modules\_multiprocessing\socket_connection.c(54) : warning C4267: 'function' : conversion from 'size_t' to 'int', possible loss of data
20>..\Modules\_multiprocessing\socket_connection.c(126) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>..\Modules\_multiprocessing\socket_connection.c(137) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>..\Modules\_multiprocessing\socket_connection.c(145) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(139) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(183) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(237) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(285) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(323) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>semaphore.c
20>pipe_connection.c
20>..\Modules\_multiprocessing\pipe_connection.c(24) : warning C4267: 'function' : conversion from 'size_t' to 'DWORD', possible loss of data
20>..\Modules\_multiprocessing\pipe_connection.c(50) : warning C4267: 'function' : conversion from 'size_t' to 'DWORD', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(139) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(183) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(237) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(285) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\__svn__\modules\_multiprocessing\connection.h(323) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>Linking...
20>   Creating library Z:\py3k\__svn__\PCbuild\\amd64\\_multiprocessing.lib and object Z:\py3k\__svn__\PCbuild\\amd64\\_multiprocessing.exp
20>Generating code
20>Finished generating code
20>Build log was saved at "file://Z:\py3k\__svn__\PCbuild\x64-temp-Release\_multiprocessing\BuildLog.htm"
20>_multiprocessing - 0 error(s), 17 warning(s)
msg116173 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-09-12 11:25
It seems that multiprocessing is fairly conservative wrt. 64-bit support. For example, conn_send_string has a string limit of 0x7fffffff. Therefore, several of the warnings are harmless; the respective lengths fit into int just fine.

I recommend that the size_t parameters are converted to int in the places where this is actually known to be correct from control flow (e.g. both cases of conn_send_string), along with an assert() that the value being cast is <= INT_MAX. 

IIUC, _conn_recvall can legitimately return int, no need to return Py_ssize_t: it will only return error codes (possibly MP_SUCCESS). Also, it might be sufficient to restrict the length of _conn_recvall to MAX_INT.

The alternate route would be to actually widen socket_connection to support 64 bit transmits (*); this would be Jesse's call. It's probably acceptable to defer this until somebody needs this enough to contribute a patch. Copying 2GiB over a pipe takes 1.5s on my system.

(*) My protocol proposal would be this: a length with the highest bit set (i.e. >= 0x80000000) indicates a huge message, where another four bytes of of length follow, for a total of 63 bits of length (MSB is unavailable as it serves as the length-of-length indicator). This would not waste bytes for small messages, and be backwards compatible.
msg222617 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-09 13:27
I'd assume that these were cleared years ago.
msg238370 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-03-18 01:00
Where do we currently stand with all compiler warnings, I'm still seeing some but I recall that we've other open issues about this problem?
History
Date User Action Args
2022-04-11 14:57:06adminsetgithub: 53991
2019-09-11 13:10:12steve.dowersetstatus: open -> closed
resolution: out of date
stage: needs patch -> resolved
2019-03-15 22:07:37BreamoreBoysetnosy: - BreamoreBoy
2015-04-28 02:48:00davinsetnosy: + davin
2015-03-18 01:00:45BreamoreBoysetnosy: + zach.ware, serhiy.storchaka, steve.dower

messages: + msg238370
versions: + Python 3.4, Python 3.5, - Python 3.2
2014-07-09 13:49:19brian.curtinsetnosy: - brian.curtin
2014-07-09 13:27:13BreamoreBoysetnosy: + BreamoreBoy
messages: + msg222617
2010-09-13 14:02:08janglinsetnosy: + janglin
2010-09-12 11:25:31loewissetnosy: + loewis
messages: + msg116173
2010-09-06 12:43:03pitroucreate