classification
Title: Malloc errors in test_io
Type: behavior Stage: needs patch
Components: macOS Versions: Python 3.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: William.Schwartz, ned.deily, pitrou, ronaldoussoren
Priority: normal Keywords:

Created on 2009-03-30 22:02 by ronaldoussoren, last changed 2018-10-10 07:07 by ronaldoussoren. This issue is now closed.

Messages (12)
msg84672 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-03-30 22:02
The malloc warnings happen on OSX, with a fresh checkout of the python3 
branch. Python was build using --enable-universalsdk=/ --with-universal-
archs=all, on a x86_64 capable laptop running Leopard.

The issue goes away when running from the commandline, the "-E -bb" 
flags that 'make test' use seem to be involved in the issue.


test_io

Testing large file ops skipped on darwin.
It requires 2147483648 bytes and a long time.
Use 'regrtest.py -u largefile test_io' to run it.

Testing large file ops skipped on darwin.
It requires 2147483648 bytes and a long time.
Use 'regrtest.py -u largefile test_io' to run it.
python.exe(35535,0x7fff701d1720) malloc: *** mmap(size=-
9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(35535,0x7fff701d1720) malloc: *** mmap(size=-
9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(35535,0x7fff701d1720) malloc: *** mmap(size=-
9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
test_ioctl
msg84674 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-03-30 22:08
What's more annoying: the error goes away while running in a debugger.

The issue happens at least in tes_constructor:

test_constructor (__main__.CBufferedWriterTest) ... 
python.exe(35957,0x7fff701d1720) malloc: *** mmap(size=-
9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
ok
msg84695 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-03-30 22:37
Buffer_size is set to 0x7fffffffffffffff in BufferedWriter_init at the end 
of test_constructor.

I have no idea why this happens, but this definitly seems wrong to me. 
Debugging is rather hard at the moment because the issue goes away in the 
debugger.

Setting this as a release blocker because arguments of a wrong magnitude 
to memory allocation functions are scary.
msg84706 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-30 23:09
>malloc: *** mmap(size=-9223372036854775808) failed

Does this error occur even if you revert Modules/mmapmodule.c to r69214?
I have touched this module recently (I believe I fixed only SEGV bugs,
not changed behavior) so I want to know it.
msg84710 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-03-30 23:15
I don't think this is related to Python's mmap module, the message says 
that malloc(3) cannot mmap some extra memory space.

I'll test with the older version of mmap just in case.
msg84712 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-30 23:17
Sorry, this is not related to mmap module... Forgot my comments.
msg84723 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-30 23:56
If you look at test_io, calling malloc() with sys.maxsize is precisely
the purpose of the test, to check that allocation failures are detected
and reported properly. So the warnings can be safely ignored, if they
only happen in test_constructor.
msg84724 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-30 23:56
PS: if the warnings are annoying, the test can be disabled selectively
under Mac OS X.
msg84726 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-03-31 00:05
I can't believe that I completely missed that this is the purpose of the 
tests, I honestly believed that I had checked everything there was to 
check :-(

Disabling the test won't be necessary, I'm keeping this issue open for a 
while longer to check if we can avoid getting a message on stderr during 
the test. It's rather annoying that malloc logs error messages during what 
turns out to be perfectly fine behaviour.
msg95386 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-11-17 16:04
Closing the issue as this is not a bug in python (as noted in the 
comments) and there doesn't seem to be a workaround for avoiding the 
spurious messages on stderr.
msg327446 - (view) Author: William Schwartz (William.Schwartz) * Date: 2018-10-10 00:01
In Jupyter Notebook, I tried to pass a large amount of data (about 2.3 GB) to Statsmodels's KDEUnivariate.fit (https://www.statsmodels.org/dev/generated/statsmodels.nonparametric.kde.KDEUnivariate.fit.html#statsmodels.nonparametric.kde.KDEUnivariate.fit) with kernel='cos'. In the Notebook, I got a MemoryError exception and traceback (which unfortunately I have since lost).

In the terminal where I was running the Jupyter server, the Python 3.7 kernel spat out:

Python(86244,0x7fff91104380) malloc: *** mach_vm_map(size=465845322670080) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

The MemoryError coming up in the Notebook interface is the correct behavior, but I wanted to bring to your attention that the malloc junk on stderr happens even outside of Python's tests.

I'm on macOS 10.13.5.
msg327457 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-10-10 07:07
The error is printed by libc, there's nothing we can do about that.
History
Date User Action Args
2021-06-21 17:37:18serhiy.storchakalinkissue27547 superseder
2018-10-10 07:07:36ronaldoussorensetmessages: + msg327457
2018-10-10 00:01:01William.Schwartzsetversions: + Python 3.7, - Python 3.1
nosy: + William.Schwartz

messages: + msg327446

components: + macOS
2014-06-12 17:22:06ned.deilylinkissue21733 superseder
2014-04-16 18:21:10ned.deilylinkissue21260 superseder
2009-11-17 16:04:26ronaldoussorensetstatus: open -> closed
type: crash -> behavior
resolution: wont fix
messages: + msg95386
2009-03-31 00:06:07ronaldoussorensetmessages: + msg84726
2009-03-30 23:56:34pitrousetmessages: + msg84724
2009-03-30 23:56:00pitrousetpriority: release blocker -> normal
versions: + Python 3.1
nosy: + pitrou

messages: + msg84723
2009-03-30 23:37:22ocean-citysetnosy: - ocean-city
2009-03-30 23:17:35ocean-citysetmessages: + msg84712
2009-03-30 23:15:06ronaldoussorensetmessages: + msg84710
2009-03-30 23:09:13ocean-citysetnosy: + ocean-city
messages: + msg84706
2009-03-30 22:37:49ronaldoussorensetpriority: release blocker
type: crash
messages: + msg84695

stage: needs patch
2009-03-30 22:08:18ronaldoussorensetmessages: + msg84674
2009-03-30 22:02:55ronaldoussorencreate