classification
Title: Cython doesn't work anymore on Python 3.6
Type: Stage:
Components: Versions: Python 3.6
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: scoder, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2016-03-09 16:01 by vstinner, last changed 2016-03-17 16:31 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
mtrand.tar.gz vstinner, 2016-03-09 16:01
Messages (9)
msg261437 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-09 16:01
I tried to run numpy test suite on Python 3.6 compiled in debug mode. I got an assertion error. Then I wanted to try the Git version of numpy, but the "Cythonizing sources" step of the numpy installer fails with:
---
Traceback (most recent call last):
  File "bin/cython", line 9, in <module>
    load_entry_point('Cython==0.23.4', 'console_scripts', 'cython')()
  (...)
  File "lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 6585, in generate_execution_code
    self.body.generate_execution_code(code)
  File "lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 7026, in generate_execution_code
    fresh_finally_clause().generate_execution_code(code)
  File "lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 7013, in fresh_finally_clause
    node_copy = copy.deepcopy(node)
  File "/home/haypo/prog/python/default/Lib/copy.py", line 182, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  (...)
  File "/home/haypo/prog/python/default/Lib/copy.py", line 314, in _reconstruct
    item = deepcopy(item, memo)
  File "/home/haypo/prog/python/default/Lib/copy.py", line 174, in deepcopy
    rv = reductor(4)
TypeError: can't pickle Argument objects
---
Argument class indirectly comes from Cython.Compiler.Nodes.ExprStatNode.

Command to reproduce the bug:
---
tar -xf mtrand.tar.gz  # download mtrand.tar.gz attached to this issue
python -m venv ENV
ENV/bin/python -m pip install cython
ENV/bin/python ENV/bin/cython --fast-fail -o mtrand.c mtrand.pyx
---

It looks like cython behaviour failed after this change:
---
changeset:   99677:b8d108a2a38e
parent:      99675:80d1faa9735d
parent:      99676:0cd2de69fb66
user:        Serhiy Storchaka <storchaka@gmail.com>
date:        Fri Dec 25 21:05:35 2015 +0200
files:       Lib/test/test_csv.py Misc/NEWS Objects/typeobject.c
description:
Issue #22995: Instances of extension types with a state that aren't
subclasses of list or dict and haven't implemented any pickle-related
methods (__reduce__, __reduce_ex__, __getnewargs__, __getnewargs_ex__,
or __getstate__), can no longer be pickled.  Including memoryview.
---

Sorry, I don't understand yet if it's a regression in Python 3.6 or a real bug in Cython.
msg261438 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-09 16:06
Oops, my commands to reproduce the bug were wrong.

Commands to reproduce the bug:
---
<start in Python source code directory>

tar -xf mtrand.tar.gz  # download mtrand.tar.gz attached to this issue

./python -m venv ENV
ENV/bin/python -m pip install cython
ENV/bin/python ENV/bin/cython --fast-fail -o mtrand/mtrand.c mtrand/mtrand.pyx
---
msg261441 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-03-09 16:31
May be this is the same issue as reported in msg257646. This was a bug in Cython and it is fixed now.
msg261442 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-09 16:39
"May be this is the same issue as reported in msg257646. This was a bug in Cython and it is fixed now."

Oh, I wasn't aware of the issue #22995. I will read it.

The bug isn't fixed yet in the latest release of Cython (0.23.4, released at 2015-10-10). I guess that it's fixed in the development version.
msg261444 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-09 16:51
> This was a bug in Cython and it is fixed now.

If the change is deliberate, can you please mention it in "What's New in Python 3.6" doc? Especially in the "Porting to Python 3.6" section:
https://docs.python.org/dev/whatsnew/3.6.html#porting-to-python-3-6

Can you suggest a fix in the doc? I guess that a method should be added: __reduce__(), __reduce_ex__() or __getstate__()?
msg261460 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-03-09 17:54
Cython 0.23.4 was released at 2015-10-10. The fix was applied 2016-01-16. Try to use Cython from trunk. May be this is not exactly the same bug and Cython needs other fix.

The change is deliberate. It helps to find existing bugs.

Except trivial case when your extension class has no any fields in additional to pickleable parent class, the class needs to provide some combinations of __getstate__, __setstate__, __getnewargs__/__getnewargs_ex__ and __reduce__/__reduce_ex__ to be pickleable. This was said in PEP 307. Previously you silently produced invalid pickle and got an exception or invalid object when unpickle it. Now you more likely get an exception at pickling time.
msg261636 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-03-12 06:28
Stefan, could you please check that this issue is fixed and there are no other cases of pickling or copying unpickleable classes?
msg261910 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2016-03-17 12:29
Our CI build server says it's all fine. The fix will eventually be released, certainly before Py3.6 comes out.
msg261915 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-17 16:31
> Our CI build server says it's all fine. The fix will eventually be released, certainly before Py3.6 comes out.

Ok, fine :-)
History
Date User Action Args
2016-03-17 16:31:34vstinnersetstatus: open -> closed
resolution: third party
messages: + msg261915
2016-03-17 12:29:28scodersetmessages: + msg261910
2016-03-12 06:28:24serhiy.storchakasetmessages: + msg261636
2016-03-09 17:54:51serhiy.storchakasetmessages: + msg261460
2016-03-09 16:51:05vstinnersetmessages: + msg261444
2016-03-09 16:39:59vstinnersetnosy: scoder, vstinner, serhiy.storchaka
messages: + msg261442
2016-03-09 16:31:50serhiy.storchakasetnosy: + scoder
messages: + msg261441
2016-03-09 16:06:56vstinnersetmessages: + msg261438
2016-03-09 16:01:57vstinnercreate