msg272557 - (view) |
Author: wevsty (wevsty) |
Date: 2016-08-12 18:59 |
When I use Python 3.5.2 some of the time will be a crash.displayed crash module is python35.dll.
I run Python environment is Windows Server 2012R2.I think crash dump maybe will be helpful.Report.wer and carsh dump can be downloaded via the following link.
http://o8z0k9748.bkt.clouddn.com/Python_crash_Dump_20160813_AF1A29AF.7z
I have another question when i using managers lib multiprocessing Maybe have exception.
my code is:
Session_lock.G_Process_Lock = multiprocessing.Manager().Lock()
def execute_select_command(command):
try:
with Session_lock.G_Process_Lock:
values = []
conn = sqlite3.connect(SQLite_DB_NAME)
cursor = conn.cursor()
try:
cursor.execute(command)
values = cursor.fetchall()
except Exception as e:
pass
finally:
cursor.close()
conn.close()
return values
except Exception as e:
print(e)
traceback.print_exc()
finally:
pass
This code is possible to print out the information.
Traceback (most recent call last):
File "C:/Users/Administrator/Desktop/Monitor_URL\Monitor_SQL.py", line 44, in execute_select_command
return values
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 949, in __exit__
return self._callmethod('release')
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 716, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 280, in _send_bytes
ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
TypeError: WriteFile() argument 1 must be int, not None
This maybe seem like a lib problem.
|
msg272577 - (view) |
Author: Eryk Sun (eryksun) * |
Date: 2016-08-13 05:34 |
To diagnose the access violation, it may help if you install the debug binaries and try to reproduce the crash using python_d.exe. Attach the dump file to this issue.
Regarding the multiprocessing question, if its a separate issue you need to file it on its own and nosy davin. At first glance, I don't see why the handle is None in the lock's __exit__ method. After you acquire the lock, try logging the connection information. For example:
import multiprocessing as mp
import multiprocessing.util
def test(lock):
with lock:
mp.util.info('test:lock:token: %r' % (lock._token,))
mp.util.info('test:lock:handle: %r' % (lock._tls.connection._handle,))
if __name__ == '__main__':
mp.util.log_to_stderr(mp.util.INFO)
lock = mp.Manager().Lock()
p = mp.Process(target=test, args=(lock,))
p.start()
p.join()
|
msg272581 - (view) |
Author: wevsty (wevsty) |
Date: 2016-08-13 10:13 |
About the crash,python_d.exe crash dump download address is
http://o8z0k9748.bkt.clouddn.com/python_d_from_vs_20160813_930A2640.7z
Crash dump display,at the
> python35_d.dll!_Py_ForgetReference(_object * op) Line 1758 C
python35_d.dll!_Py_Dealloc(_object * op) Line 1786 C
python35_d.dll!tupledealloc(PyTupleObject * op) Line 236 C
I hope this can be helpful to you.
About the multiprocessing question
I modified my code becomes
def execute_select_command(command):
values = []
try:
with Session_lock.G_Process_Lock:
mp.util.info('test:lock:token: %r' % (Session_lock.G_Process_Lock._token,))
mp.util.info('test:lock:handle: %r' % (Session_lock.G_Process_Lock._tls.connection._handle,))
conn = sqlite3.connect(SQLite_DB_NAME,check_same_thread = False)
cursor = conn.cursor()
try:
cursor.execute(command)
values = cursor.fetchall()
except Exception as e:
pass
finally:
cursor.close()
conn.close()
return values
except Exception as e:
print(e)
traceback.print_exc()
mp.util.info('Exception:lock:token: %r' % (Session_lock.G_Process_Lock._token,))
mp.util.info('Exception:lock:handle: %r' % (Session_lock.G_Process_Lock._tls.connection._handle,))
pass
return values
finally:
pass
After run,Console output
[INFO/SpawnPoolWorker-18] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-18] test:lock:handle: 720
[INFO/SpawnPoolWorker-17] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-17] test:lock:handle: 908
[INFO/SpawnPoolWorker-16] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-16] test:lock:handle: 744
WriteFile() argument 1 must be int, not None
[INFO/SpawnPoolWorker-16] Exception:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-16] Exception:lock:handle: 744
Could not find thread pid_2720_id_389212239784
this problem and crash both the use of multiprocessing lib after appears.Maybe these two issues are related.
If you need my full code or need to open a new issue, please let me know.Thinks
|
msg272584 - (view) |
Author: Decorater (Decorater) * |
Date: 2016-08-13 12:08 |
Crashes also happen from Recursion if not careful. Idk why but it happens. Although some of it is probably by not using try / catch probably anywhere in the python source code as a safeguard. I always like try / catch even in C++ as you never know when crap might happen to make things crash. A Perfect Example is a Access violation.
(Yes I have this is a Standard for me and it gets me to great quality code)
(this makes me wonder if Windows does not use try / catch anywhere. It would be nice if Microsoft Required try / catch in every function for drivers as well to hopefully make them more stable.)
All I know is I like having a lot of Checks (even if not required) that can actually prevent any and all exceptions to begin with.
And yes from what I learned from experiance myself. Always use try and catch anything before it is to late will help you in the long run.
(not to mention I blame Microsoft for not replacing their current Windows Update for when Updates Fail with a Detailed list of the names of the update(s) that failed and a description of what the reason is without Error code(s) as the Codes does not tell crap about why they failed. So, yeah I point my fingers at whoever wrote Windows Update, to fix their crap. As this is a issue in windows 7 SP1, 8, 8.1, and 10.)
|
msg272589 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2016-08-13 14:19 |
Decorator: In python code you generally want to let the exception bubble up so that you can see what the error was. Sometimes you do want to catch general errors, but mostly you don't.
As for C++ try/catch, CPython is written in C, not C++.
So, your comment doesn't really have any relevance to this issue, I'm afraid.
|
msg272615 - (view) |
Author: Decorater (Decorater) * |
Date: 2016-08-13 20:39 |
I can confirm that I also get the same crash.
|
msg272616 - (view) |
Author: Decorater (Decorater) * |
Date: 2016-08-13 20:39 |
I do know for sure the crash happens in python.exe and blames ntdll.dll for it.
|
msg272647 - (view) |
Author: wevsty (wevsty) |
Date: 2016-08-14 05:41 |
To Decorater
If you are found to have crashes, I'm sorry, you can follow the previous suggestion to use python_d.exe then open a new issue uploading crash dump.
It would be better.
About the multiprocessing question,I would like to add some information.
My code uses a multiprocessing Pool,I used multiprocessing.util.debug opened the debug information.debug information is as follows.
[DEBUG/SpawnPoolWorker-7] thread 'MainThread' does not own a connection
[DEBUG/SpawnPoolWorker-7] making connection to manager
[DEBUG/SyncManager-2] starting server thread to service 'SpawnPoolWorker-7'
[INFO/SpawnPoolWorker-4] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-4] test:lock:handle: 712
[INFO/SpawnPoolWorker-7] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-7] test:lock:handle: 768
[DEBUG/SpawnPoolWorker-7] DECREF 'e44c9bc720'
[INFO/SpawnPoolWorker-4] leave:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[DEBUG/SpawnPoolWorker-7] thread 'MainThread' has no more proxies so closing conn
[DEBUG/SpawnPoolWorker-7] DECREF 'b0182059f8'
[DEBUG/SyncManager-2] got EOF -- exiting thread serving 'SpawnPoolWorker-7'
[DEBUG/SpawnPoolWorker-4] INCREF 'e44c9bc720'
handle is closed
'ForkAwareLocal' object has no attribute 'connection'
[DEBUG/SpawnPoolWorker-7] thread 'MainThread' has no more proxies so closing conn
Traceback (most recent call last):
File "C:/Users/Administrator/Desktop/Monitor_URL\Monitor_SQL.py", line 51, in execute_select_command
conn.close()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 949, in __exit__
return self._callmethod('release')
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 716, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 204, in send
self._check_closed()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 136, in _check_closed
raise OSError("handle is closed")
OSError: handle is closed
[INFO/SpawnPoolWorker-7] Exception:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-7] Exception process:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[DEBUG/SyncManager-1] got EOF -- exiting thread serving 'SpawnPoolWorker-7'
[DEBUG/SpawnPoolWorker-4] INCREF 'b0182059f8'
[DEBUG/SpawnPoolWorker-4] DECREF 'b0182059f8'
[DEBUG/SpawnPoolWorker-7] INCREF 'e44c9bc720'
This time the message and not the same,From the information looks like two processes simultaneously locking the same lock object, and then release happened the problems.I do not know if this is correct,
But i will continue to follow the issue.
|
msg272653 - (view) |
Author: Decorater (Decorater) * |
Date: 2016-08-14 07:29 |
I use code that does multithreading to from a library that uses ffmpeg for exact it makes daemon threads so it could be from that.
|
msg272743 - (view) |
Author: wevsty (wevsty) |
Date: 2016-08-15 10:01 |
I try to change the multiprocessing lib codes,upload the file is debugger display.I try to used while loop make sure connection hanle value is not none,but connection hanle value will still becomes none.
I think there may be other threads to modify the handle,But it not my code.
|
msg272861 - (view) |
Author: wevsty (wevsty) |
Date: 2016-08-16 15:51 |
today,Find a new crash
python_d_crash_dump_20160816_A901A6F3.7z,is full crash dump
Download address
http://o8z0k9748.bkt.clouddn.com/python_d_crash_dump_20160816_A901A6F3.7z
dump display crash in the
> python35_d.dll!visit_decref(_object * op, void * data) Line 379 C
python35_d.dll!dict_traverse(_object * op, int(*)(_object *, void *) visit, void * arg) Line 2534 C
python35_d.dll!subtract_refs(_gc_head * containers) Line 401 C
python35_d.dll!collect(int generation, __int64 * n_collected, __int64 * n_uncollectable, int nofail) Line 959 C
|
msg273007 - (view) |
Author: wevsty (wevsty) |
Date: 2016-08-18 00:54 |
A similar situation also occurs in the following Linux,A similar situation on stackoverflow
http://stackoverflow.com/questions/29277150/python-3-4-multiprocessing-bug-on-lock-acquire-typeerror-integer-required
|
msg334464 - (view) |
Author: Michael Jacob (Michael Jacob2) |
Date: 2019-01-28 13:14 |
Does this ticket track the issue in its title or the native crash? If it's the latter, is there a new ticket for the None _handle or shall I create one?
|
msg334628 - (view) |
Author: Michael Jacob (Michael Jacob2) |
Date: 2019-01-31 13:36 |
So, I'm experiencing the issue in the title, too. The pipe handle stays valid for between 5 and 60 minutes, then it goes None when written to.
I'm far from understanding that code, but this crude re-connect code seems to solve the issue for me:
In BaseProxy._callmethod, I changed:
conn.send((self._id, methodname, args, kwds))
to:
try:
conn.send((self._id, methodname, args, kwds))
except TypeError:
self._connect()
conn = self._tls.connection
conn.send((self._id, methodname, args, kwds))
|
|
Date |
User |
Action |
Args |
2022-04-11 14:58:34 | admin | set | github: 71936 |
2021-03-13 22:29:18 | eryksun | set | title: multprocessing errors on Windows: WriteFile() argument 1 must be int, not None; OSError: handle is closed -> multiprocessing Manager error: send() called for a closed connection type: crash -> behavior versions:
+ Python 3.8, Python 3.9, Python 3.10, - Python 3.5, Python 3.7 |
2019-01-31 13:36:04 | Michael Jacob2 | set | messages:
+ msg334628 |
2019-01-28 13:14:32 | Michael Jacob2 | set | nosy:
+ Michael Jacob2
messages:
+ msg334464 versions:
+ Python 3.7 |
2016-08-18 06:30:51 | xiang.zhang | set | nosy:
+ davin
|
2016-08-18 00:54:24 | wevsty | set | messages:
+ msg273007 |
2016-08-17 10:57:43 | vstinner | set | title: multprocessing errors on WindowsWriteFile() argument 1 must be int, not None; OSError: handle is closed -> multprocessing errors on Windows: WriteFile() argument 1 must be int, not None; OSError: handle is closed |
2016-08-17 10:57:35 | vstinner | set | title: python 3.5.2 maybe crash in windows -> multprocessing errors on WindowsWriteFile() argument 1 must be int, not None; OSError: handle is closed |
2016-08-17 00:37:25 | wevsty | set | title: python 3.5.2 maybe crash -> python 3.5.2 maybe crash in windows |
2016-08-16 15:51:05 | wevsty | set | messages:
+ msg272861 |
2016-08-15 10:01:09 | wevsty | set | files:
+ debug_AA1F93C2.JPG
messages:
+ msg272743 |
2016-08-14 07:29:11 | Decorater | set | messages:
+ msg272653 |
2016-08-14 05:41:31 | wevsty | set | messages:
+ msg272647 |
2016-08-13 20:39:57 | Decorater | set | messages:
+ msg272616 |
2016-08-13 20:39:06 | Decorater | set | files:
+ python.exe.8228.dmp
messages:
+ msg272615 |
2016-08-13 14:19:57 | r.david.murray | set | nosy:
+ r.david.murray messages:
+ msg272589
|
2016-08-13 12:08:33 | Decorater | set | nosy:
+ Decorater messages:
+ msg272584
|
2016-08-13 10:13:42 | wevsty | set | messages:
+ msg272581 |
2016-08-13 05:34:46 | eryksun | set | nosy:
+ paul.moore, tim.golden, steve.dower, zach.ware components:
+ Windows
|
2016-08-13 05:34:24 | eryksun | set | nosy:
+ eryksun messages:
+ msg272577
|
2016-08-12 18:59:30 | wevsty | create | |