Issue26162
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2016-01-20 11:28 by Ali Razmjoo, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (2) | |||
---|---|---|---|
msg258667 - (view) | Author: Ali Razmjoo (Ali Razmjoo) * | Date: 2016-01-20 11:28 | |
Hello, I've got a problem while using threading in python 2.7.10 windows. I copied errors in here https://gist.github.com/Ali-Razmjoo/d503171d338c6381f94f with 845 threads,870 and 1000. there isn't any problem or error with 840-830 threads! |
|||
msg258710 - (view) | Author: Eryk Sun (eryksun) * ![]() |
Date: 2016-01-20 19:57 | |
It's no surprise to see a memory error at 845 threads if you're using 32-bit Python, which is limited to 2 GiB of address space. python.exe is built with a default stack reservation of 2000000 bytes, so 845 threads reserve a total of about 1.58 GiB. Consider also that address space is used by mapped DLLs and files, page tables, private data, the process heap(s), and additional worker threads. Plus if it's 64-bit Windows there's a 64-bit stack for each 32-bit thread. The 64-bit stack reserves 256 KiB, which totals an additional 0.2 GiB for 845 threads. Available space is also lost to fragmentation that leaves unusable blocks. Why do you need so many threads instead of using a thread pool? If you *really* need that many threads and can't switch to 64-bit Python, you'll have to step outside what Python's built-in threading support offers. The following demonstrates calling Windows CreateThread via ctypes. I recommend against doing something like this. import ctypes from ctypes import wintypes kernel32 = ctypes.WinDLL('kernel32', use_last_error=True) CREATE_SUSPENDED = 0x00000004 STACK_SIZE_PARAM_IS_A_RESERVATION = 0x00010000 def check_handle(result, func, args): if result is None: raise ctypes.WinError(ctypes.get_last_error()) return args def check_bool(result, func, args): if not result: raise ctypes.WinError(ctypes.get_last_error()) return None wintypes.LPDWORD = ctypes.POINTER(wintypes.DWORD) wintypes.SIZE_T = ctypes.c_size_t LPSECURITY_ATTRIBUTES = ctypes.c_void_p LPTHREAD_START_ROUTINE = ctypes.WINFUNCTYPE(wintypes.DWORD, wintypes.LPVOID) kernel32.CreateThread.errcheck = check_handle kernel32.CreateThread.restype = wintypes.HANDLE kernel32.CreateThread.argtypes = ( LPSECURITY_ATTRIBUTES, # _In_opt_ lpThreadAttributes wintypes.SIZE_T, # _In_ dwStackSize LPTHREAD_START_ROUTINE, # _In_ lpStartAddress wintypes.LPVOID, # _In_opt_ lpParameter wintypes.DWORD, # _In_ dwCreationFlags wintypes.LPDWORD, # _Out_opt_ lpThreadId ) kernel32.CloseHandle.errcheck = check_bool kernel32.CloseHandle.argtypes = (wintypes.HANDLE,) def CreateThread(lpStartAddress, lpParameter=None, dwStackSize=0, dwCreationFlags=STACK_SIZE_PARAM_IS_A_RESERVATION, lpThreadAttributes=None, close_handle=True): tid = (wintypes.DWORD * 1)() h = kernel32.CreateThread(lpThreadAttributes, dwStackSize, lpStartAddress, lpParameter, dwCreationFlags, tid) if close_handle: kernel32.CloseHandle(h) return tid[0] return h, tid[0] For example, the following creates 2000 worker threads that each reserves 256 KiB for the stack. Since I'm running 32-bit Python on 64-bit Windows, each thread has a corresponding 64-bit thread stack that also reserves 256 KiB. In total it reserves 1 GiB of stack space, which I confirmed using Sysinternals VMMap. import time @LPTHREAD_START_ROUTINE def worker(param): if param is None: param = 0 # do some work time.sleep(60) return 0 tids = [] for i in range(2000): tid = CreateThread(worker, i, 256*1024) tids.append(tid) |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:26 | admin | set | github: 70350 |
2016-01-20 19:57:43 | eryksun | set | status: open -> closed nosy: + eryksun messages: + msg258710 resolution: not a bug stage: resolved |
2016-01-20 11:28:44 | Ali Razmjoo | create |