This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author stadt
Recipients
Date 2001-02-18.02:37:37
SpamBayes Score
Marked as misclassified
Message-id
In-reply-to
Content
I do *not* mean file upload fields. I stumbled upon this with a webform that
contains 141 'simple' input fields like the form you can see here (which 'only'
contains 31 of those input fields):

http://www.cyberchair.org/cgi-cyb/genAssignPageReviewerPapers.py

(use chair/chair to login)

When the maximum number of file descriptors used per process was increased
to 160 (by the sysadmins), the problem did not occur anymore, and the webform
could be processed.

This was the error message I got: 

Traceback (most recent call last):
  File "/usr/local/etc/httpd/DocumentRoot/ICML2001/cgi-bin/submitAssignRP.py", line 144, in main
  File "/opt/python/2.0/sparc-sunos5.6/lib/python2.0/cgi.py", line 504, in __init__
  File "/opt/python/2.0/sparc-sunos5.6/lib/python2.0/cgi.py", line 593, in read_multi
  File "/opt/python/2.0/sparc-sunos5.6/lib/python2.0/cgi.py", line 506, in __init__
  File "/opt/python/2.0/sparc-sunos5.6/lib/python2.0/cgi.py", line 603, in read_single
  File "/opt/python/2.0/sparc-sunos5.6/lib/python2.0/cgi.py", line 623, in read_lines
  File "/opt/python/2.0/sparc-sunos5.6/lib/python2.0/cgi.py", line 713, in make_file
  File "/opt/python/2.0/sparc-sunos5.6/lib/python2.0/tempfile.py", line 144, in TemporaryFile
OSError: [Errno 24] Too many open files: '/home/yara/brodley/icml2001/tmp/@26048.61'

I understand why you assume that it would concern *file* uploads, but this is
not the case. As I reported before, it turns out that for each 'simple' <input> field
a temporary file is used in to transfer the contents to the script that uses the
cgi.FieldStorage() method, even if no files are being uploaded. The problem is not
that too many files are open at the same time (which is 1 at most). It is the *amount*
of files that is causing the troubles. If the same temporary file would be used, this
problem would probably not have happened. 

My colleague Fred Gansevles wrote a possible solution, but mentioned that this
might introduce the need for protection against a 'symlink attack' (whatever
that may be). This solution(?) concentrates on the open file descriptor's problem,
while Fred suggests a redesign of FieldStorage() would probably be better.

import os, tempfile
AANTAL = 50

class TemporaryFile:
    def __init__(self):
        self.name = tempfile.mktemp("")
        open(self.name, 'w').close()
        self.offset = 0
    
    def seek(self, offset):
        self.offset = offset
    
    def read(self):
        fd = open(self.name, 'w+b', -1)
        fd.seek(self.offset)
        data = fd.read()
        self.offset = fd.tell()
        fd.close()
        return data
    
    def write(self, data):
        fd = open(self.name, 'w+b', -1)
        fd.seek(self.offset)
        fd.write(data)
        self.offset = fd.tell()
        fd.close()

    def __del__(self):
        os.unlink(self.name)

def add_fd(l, n) :
    map(lambda x,l=l: l.append(open('/dev/null')), range(n))

def add_tmp(l, n) :
    map(lambda x,l=l: l.append(TemporaryFile()), range(n))

def main ():
    import getopt, sys
    try:
        import resource
        soft, hard = resource.getrlimit (resource.RLIMIT_NOFILE)
        resource.setrlimit (resource.RLIMIT_NOFILE, (hard, hard))
    except ImportError:
        soft, hard = 64, 1024
    opts, args = getopt.getopt(sys.argv[1:], 'n:t')
    aantal = AANTAL
    tmp = add_fd
    for o, a in opts:
        if o == '-n':
            aantal = int(a)
        elif o == '-t':
            tmp = add_tmp
    print "aantal te gebruiken fd's:", aantal   #dutch; English: 'number of fds to be used'
    print 'tmp:', tmp.func_name
    tmp_files = []
    files=[]
    tmp(tmp_files, aantal)
    try:
        add_fd(files,hard)
    except IOError:
        pass
    print "aantal vrije gebruiken fd's:", len(files)  #enlish: 'number of free fds'

main()

Running the above code:

    python ulimit.py [-n number] [-t]
    default number = 50, while using 'real' fd-s for temporary files.
    When using the '-t' flag 'smart' temporary files are used.

    Output:

        $ python ulimit.py
        aantal te gebruiken fd's: 50
        tmp: add_fd
        aantal vrije gebruiken fd's: 970

        $ python ulimit.py -t
        aantal te gebruiken fd's: 50
        tmp: add_tmp
        aantal vrije gebruiken fd's: 1020

        $ python ulimit.py -n 1000
        aantal te gebruiken fd's: 1000
        tmp: add_fd
        aantal vrije gebruiken fd's: 20

        $ python ulimit.py -n 1000 -t
        aantal te gebruiken fd's: 1000
        tmp: add_tmp
        aantal vrije gebruiken fd's: 1020
History
Date User Action Args
2007-08-23 13:53:03adminlinkissue231249 messages
2007-08-23 13:53:03admincreate