This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Assertion with very long lists
Type: Stage:
Components: Interpreter Core Versions:
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: nnorwitz Nosy List: gvanrossum, jalet, nnorwitz, tim.peters
Priority: normal Keywords:

Created on 2002-05-29 10:12 by jalet, last changed 2022-04-10 16:05 by admin. This issue is now closed.

Messages (14)
msg10952 - (view) Author: Jerome Alet (jalet) Date: 2002-05-29 10:12
I've tried to import a module which only define a list
of 170000 elements (a word list) which defines an
element per line, so the module also has 170000 lines
in it. Here's the error :

--- CUT ---
Python 2.1.3 (#1, Apr 20 2002, 10:14:34) 
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2
Type "copyright", "credits" or "license" for more
information.
>>> import dictionnaire
python: ../Python/ceval.c:695: eval_code2: Assertion
`(stack_pointer - f->f_valuestack) <= f->f_stacksize'
failed.
Abandon
--- CUT ---

this seems to be the same problem as bug id 210647 but
there's no segfault.

the module is attached to this bug report.
msg10953 - (view) Author: Jerome Alet (jalet) Date: 2002-05-29 10:14
Logged In: YES 
user_id=97214

module too long to be attached. ask it by email at
alet@librelogiciel.com if needed.
msg10954 - (view) Author: Jerome Alet (jalet) Date: 2002-05-29 12:42
Logged In: YES 
user_id=97214

I've done more tests, the bug only occurs if every line has
a backslash at its end :

liste = [ \
            "item0", \
            "item1", \
            # ... many many lines
            "itemN" ]

and it works if I remove the \ from each end of line.

hth.
msg10955 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-05-31 22:32
Logged In: YES 
user_id=33168

I think there are several problems here.
I was able to narrow down at least one problem to a long list.
It doesn't seem to have anything to do w/backslashes.

There seems to be an interaction with importing the module.
If the long list is in __main__, the problem does not occur.
The problem only occurs when the list is in an imported module
and only after importing it the second time (after the .pyc
is created).

If a list has 32768+ items there seems to be a memory problem.
There is an exception, then a crash.  Here's the exception:

TypeError: unsupported operand type(s) for &: 'str' and 'str'

Here's a stack trace against CVS version:

#0  chunk_free (ar_ptr=0x401b1620, p=0x8156a10) at malloc.c:3180
#1  0x400fdbf4 in __libc_free (mem=0x8156a18) at malloc.c:3154
#2  0x0805c0db in list_dealloc (op=0x401c4eac) at
Objects/listobject.c:214
#3  0x080c7618 in PyDict_SetItem (op=0x401bd79c,
key=0x401c4ec0, 
    value=0x80e185c) at Objects/dictobject.c:373
#4  0x080c93f9 in PyDict_SetItemString (v=0x401bd79c,
key=0x80d9070 "path", 
    item=0x80e185c) at Objects/dictobject.c:1895
#5  0x08099ad8 in PyImport_Cleanup () at Python/import.c:286
#6  0x080a266d in Py_Finalize () at Python/pythonrun.c:228
#7  0x08053766 in Py_Main (argc=2, argv=0xbffff904) at
Modules/main.c:379
msg10956 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-06-01 00:06
Logged In: YES 
user_id=33168

I found at least part of the problem, perhaps everything.
The co_stacksize > 32767 and only a short value is marshalled
to the .pyc.

See Python/marshal.c line 246.

Not sure how to fix yet.
msg10957 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-06-01 00:17
Logged In: YES 
user_id=33168

Got a fix.  Patch attached.  
It will probably have to be changed,
since I don't remember what else needs to be
done when the MAGIC is changed.
msg10958 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-06-01 02:04
Logged In: YES 
user_id=33168

Final patch attached.
msg10959 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-06-01 14:56
Logged In: YES 
user_id=33168

Added a patch for 2.2 (2.1) is basically the same.
No test for 2.2/2.1 though.
Could someone please review before checkin?
msg10960 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2002-06-01 18:14
Logged In: YES 
user_id=31435

Assuming the patch you're talking about is the one named 
long-list-2.2.diff, looks good to me, so Accepted and 
assigned back to you.  Thanks!
msg10961 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-06-01 18:28
Logged In: YES 
user_id=33168

Thanks Tim.
Checked in for 2.1/2.2 as 2.175.2.3 and 2.192.6.2. 

Could you or someone also review the other patch for current
CVS?
That patch has a test and is different ([rw]_short ->
[rw]_long).
msg10962 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-06-04 17:20
Logged In: YES 
user_id=33168

Tim, could you also review long-list.diff.  This is the
patch against CVS version.
It changes what is marshalled to disk.  Thanks.
msg10963 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-06-13 22:08
Logged In: YES 
user_id=33168

Guido, could you review this patch before checking in?
Thanks.
msg10964 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-06-14 00:14
Logged In: YES 
user_id=6380

This looks good to me. You might want to increase the size
of the list in the testcase so that it actually crashes for
non-debug Python without the fix; I could only get it to
fail an assertion in debug mode.
msg10965 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-06-14 01:08
Logged In: YES 
user_id=33168

I increased the list size in the test to 65000 which crashed
on my box.

Checked in as:
  Python/import.c  2.206
  Python/marshal.c  1.72  Lib/test/test_import.py  1.12
 
History
Date User Action Args
2022-04-10 16:05:21adminsetgithub: 36661
2002-05-29 10:12:56jaletcreate