This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Cloudpickle.py file is crashing due to data type incompatibility.
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.8
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: kafeel.ansari, paul.moore, steve.dower, tim.golden, xtreak, zach.ware
Priority: normal Keywords:

Created on 2019-11-12 09:49 by kafeel.ansari, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
Spark_cloudpickle_error.JPG kafeel.ansari, 2019-11-14 06:58
Messages (4)
msg356430 - (view) Author: Kafeel Ansari (kafeel.ansari) Date: 2019-11-12 09:49
I tried python3.8 for my project. But it is crashing in the beginning . When debugged , found out that "Line 145 in cloudpickle.py" is returning the value in bytes which is not expected.

Code Snippet: 
 
 _cell_set_template_code = _make_cell_set_template_code()

Error: 

return types.CodeType(
TypeError: an integer is required (got type bytes)
msg356437 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-11-12 11:00
Is this on Python 3.8.0 ? There was a similar report in the past with cloudpickle at https://bugs.python.org/issue36886#msg342182 due to PEP 570 with which CodeType.replace was introduced. cloudpickle also added support for this with https://github.com/cloudpipe/cloudpickle/pull/269 and has Python 3.8 in their CI matrix. Can you please add the version of cloudpickle and python version with which this error is caused? I guess it's more of an issue to be fixed with cloudpickle.
msg356582 - (view) Author: Kafeel Ansari (kafeel.ansari) Date: 2019-11-14 06:58
Hi @xtreak ,
Thank you for the reply . Please find below detailed description. 

I was using python 3.8 and installed pyspark 2.4.4 . I encountered error related to cloudpickle file. Please find the attached snapshot. 

I tried to convert the value of co.co_code in cloudpickle.py file (line 132) from bytes to int to check if it is really a issue with cloudpickle file. 

Please let me know if you need more information .
msg356646 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-11-15 05:10
I guess this was fixed upstream with https://github.com/apache/spark/commit/811d563fbf60203377e8462e4fad271c1140b4fa . Please try the latest version as per the commit details. I am closing this as third party since it's not a CPython issue. In future reports please attach the text of the traceback since it's more accessible. Thanks.
History
Date User Action Args
2022-04-11 14:59:23adminsetgithub: 82956
2019-11-15 05:10:11xtreaksetstatus: open -> closed
type: crash -> behavior
messages: + msg356646

resolution: third party
stage: resolved
2019-11-14 06:58:05kafeel.ansarisetfiles: + Spark_cloudpickle_error.JPG

messages: + msg356582
2019-11-12 11:00:17xtreaksetnosy: + xtreak
messages: + msg356437
2019-11-12 09:49:13kafeel.ansaricreate