classification
Title: CPython doesn't work when you disable refcounting
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, larry, serhiy.storchaka, vstinner
Priority: low Keywords:

Created on 2016-04-30 04:36 by larry, last changed 2019-05-15 02:55 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
larry.turn.off.refcounts.1.diff.txt larry, 2016-04-30 04:36 review
Messages (6)
msg264541 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-04-30 04:36
So here's a strange one.

I want to do some mysterious experiments with CPython.  So I disabled refcount changes in CPython.

I changed Py_INCR and Py_DECR so they expand to nothing.  I had to change some other macros to match (SETREF, XSETREF, and the Py_RETURN_* ones) to fix some compiler errors and warnings.  Also, to prevent the str object from making in-place edits, I changed _Py_NewReference so that the initial reference count for all objects is 2.

CPython builds, then gets to the "generate-posix-vars" step and fails with this output:

./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
        echo "generate-posix-vars failed" ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
fi
Fatal Python error: Py_Initialize: Unable to get the locale encoding
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 1078, in _path_importer_cache
KeyError: '/usr/local/lib/python36.zip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 979, in _find_and_load
  File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 903, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1137, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1108, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1080, in _path_importer_cache
  File "<frozen importlib._bootstrap_external>", line 1056, in _path_hooks
  File "<frozen importlib._bootstrap_external>", line 1302, in path_hook_for_FileFinder
  File "<frozen importlib._bootstrap_external>", line 96, in _path_isdir
  File "<frozen importlib._bootstrap_external>", line 81, in _path_is_mode_type
  File "<frozen importlib._bootstrap_external>", line 75, in _path_stat
AttributeError: module 'posix' has no attribute 'stat'
Aborted (core dumped)
generate-posix-vars failed
Makefile:598: recipe for target 'pybuilddir.txt' failed
make: *** [pybuilddir.txt] Error 1

I'm stumped.  Why should CPython be dependent on reference counts actually changing?  I figured I'd just leak memory like crazy, not change behavior.

Attached is my patch against current trunk (1ceb91974dc4) in case you want to try it yourself.  Testing was done on Ubuntu 15.10 64-bit, gcc 5.2.1.
msg264542 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-30 05:06
There is other code that sets refcount.

Include/object.h:764:    Py_REFCNT(op) = 1)
Objects/moduleobject.c:38:        Py_REFCNT(def) = 1;
Objects/longobject.c:5450:            Py_REFCNT(op) = refcnt + 1;
Objects/unicodeobject.c:1762:        Py_REFCNT(unicode) = 3;
Objects/unicodeobject.c:15044:    Py_REFCNT(s) -= 2;
Objects/unicodeobject.c:15103:            Py_REFCNT(s) += 1;
Objects/unicodeobject.c:15107:            Py_REFCNT(s) += 2;

I would suggest to set initial refcount to 1000000000, define Py_REFCNT(s) as returning constant 1000000000, and fix subsequent compiler errors.
msg264546 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-04-30 09:47
I did as you suggested.  I also discovered that _Py_NewReference is usually a macro, and I'd only fixed the version in Objects/object.c (the one not being used).  When I fixed both those things, and switched the makefile so it uses a different Python interpreter in the build itself, it builds successfully.  The interpreter doesn't run though.

The next problem: there are a couple of spots where code asserts the refcount of an object must == 1.  This is so they can modify the object in-place.  This is usually a sanity-check, so I assume the code knows what it's doing, so I removed the refcount assertion.  I did this two or three times: set, unicode, and (maybe) bytes.

Now it's segfaulting in lookdict_unicode_nodummy().  mp->ma_keys looks like garbage, the keys it refers to are invalid addresses.  I'd be very happy to post my patch if someone else wanted to try debugging it.
msg264550 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-30 10:40
Just now I'm reading the code of the peephole optimizer, and it contains such assertion since it modifies the lnotab bytes array in-place. May be there are other similar cases.
msg264553 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-04-30 12:29
Here is my theory: if the code asserts that it's true, and the code normally runs fine, then it's normally true, and therefore I can remove the assertion and the code will run correctly.  I haven't hit that assertion in the peephole optimizer.  But I have assertions turned off.

The checks in unicode / set / bytes (maybe) were not regulated by _NDEBUG, they were unconditional "if (refcnt != 1) BadInternalCall();" code.
msg342541 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-15 02:55
Well, CPython requires reference counting. This issue looks highly experimental with not activity for 3 years, I close it.
History
Date User Action Args
2019-05-15 02:55:35vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg342541

resolution: not a bug
stage: needs patch -> resolved
2016-04-30 12:29:52larrysetmessages: + msg264553
2016-04-30 10:40:18serhiy.storchakasetmessages: + msg264550
2016-04-30 09:47:55larrysetmessages: + msg264546
2016-04-30 05:06:30serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg264542
2016-04-30 04:36:50larrysetversions: + Python 3.6
2016-04-30 04:36:37larrycreate