This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: MemoryError While Executing Python Code
Type: resource usage Stage: test needed
Components: Versions: Python 2.6
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: brian.curtin, freakcycle, mark.dickinson, p_noblebose, pitrou
Priority: normal Keywords:

Created on 2010-02-10 14:58 by p_noblebose, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
Aura_Test_Scripts_FCT_ACTUSER_EVENT_L.tsv p_noblebose, 2010-02-11 08:13 Test cases document
Messages (8)
msg99164 - (view) Author: Prashanth noble (p_noblebose) Date: 2010-02-10 14:58
we are executing two queries and storing the result sets  and matching the data.

the table has got around 8 million records and while executing the same i am getting the Memory Issue.

TRACE Arguments: [ <DatabaseLibrary.resultset.Resultset instance at 0x01750828> | <DatabaseLibrary.resultset.Resultset instance at 0x01750E68> ] 
FAIL MemoryError 

DEBUG Traceback (most recent call last):

 File "c:\python26\lib\site-packages\DatabaseLibrary\__init__.py", line 52, in resultsets_should_match
   resultset1.matches_with(resultset2)
 File "c:\python26\lib\site-packages\DatabaseLibrary\resultset.py", line 37, in matches_with
   for row_a, row_b in zip(self, other): 


Please let me know how to resolve this issue
msg99197 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-02-11 04:14
Can you provide a test case which can reproduce the issue? Specifically one which doesn't require the use of your DatabaseLibrary package?
msg99201 - (view) Author: Prashanth noble (p_noblebose) Date: 2010-02-11 08:13
please find the testcase i am using
msg99210 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-02-11 14:46
Prashanth, I'm not even sure what to do with that file.

Can you provide a Python script which demonstrates your issue? It should be something that myself or others could run on their computer, free of any third party packages. This will isolate the issue to Python and show us how to reproduce what you are seeing.
msg99213 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-02-11 15:10
If these 8 million records are loaded into memory, it can be normal to get a MemoryError. It depends on the complexity of the records (number of fields) and the way they are kept in memory (tuples? dicts?); and of course of the amount of available RAM in your system.

In any case your description of the problem is not sufficiently detailed for us to now whether it's a problem with Python, a problem with your code, or simply some mismatching expectations on your part.
msg109366 - (view) Author: Peter Wolf (freakcycle) Date: 2010-07-06 04:44
I am new to this so apologies if I posting this in the wrong spot.

I am also getting a memory error on an 86MB python data file.It is one long list.As far as I can tell the data file is valid python code.My computer has 3.4 GB of ram.

When I type the following in a terminal 
> python mydatafile.py

On the following line I get 

MemoryError

That is all.
msg109375 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-07-06 09:32
Peter, this looks like a different problem;  please could you open a new issue?

When/if you do, please could you also report:

 - what version of Python you're using
 - your system details (what OS?  32-bit or 64-bit?)

If you can give us a better idea of what's in your 86Mb file, that would also help:  perhaps post the first few and last few lines.  What's in this list?  Integers?  Floats?  Strings?

It's not totally implausible that Python really *does* need gigabytes of memory to process a list in an 86Mb file.
msg109381 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-07-06 10:01
Closing the original issue: there's insufficient information to reproduce or analyze the problem.
History
Date User Action Args
2022-04-11 14:56:57adminsetgithub: 52147
2010-07-06 10:01:47mark.dickinsonsetstatus: open -> closed
resolution: works for me
messages: + msg109381
2010-07-06 09:32:03mark.dickinsonsetnosy: + mark.dickinson
messages: + msg109375
2010-07-06 04:44:36freakcyclesetnosy: + freakcycle
messages: + msg109366
components: - Windows
2010-02-11 15:10:27pitrousetnosy: + pitrou
messages: + msg99213
2010-02-11 14:46:32brian.curtinsetmessages: + msg99210
2010-02-11 08:13:38p_noblebosesetfiles: + Aura_Test_Scripts_FCT_ACTUSER_EVENT_L.tsv

messages: + msg99201
2010-02-11 04:14:53brian.curtinsetpriority: normal

nosy: + brian.curtin
messages: + msg99197

type: performance -> resource usage
stage: test needed
2010-02-10 14:58:28p_noblebosecreate