I am using Python as a hosted scripting runtime for a product. All the user scripts are stored in a database. I use "compile" and "exec" to run the scripts. I'd like to use PDB to debug scripts. Unfortunately, PDB makes a call to linecache, which calls tokenize.open assuming the code is in a file. I'd like to pass in a function that takes a filename and returns the source code in a string. 

At the very least, moving the call to "linecache.getline" into a separate method in PDB ("self.get_lines") would allow me to override that method with my own.

The lazycache() function of the linecache module meets your request, I think. See the following debugging session using the attached script:

$ python -m pdb debug_script.py
> /path/to/cwd/debug_script.py(1)<module>()
-> def debug_script(script):
(Pdb) n
> /path/to/cwd/debug_script.py(19)<module>()
-> """
(Pdb) n
> /path/to/cwd/debug_script.py(20)<module>()
-> code, globs = debug_script(s)
(Pdb) n
> /path/to/cwd/debug_script.py(21)<module>()
-> exec(code, globs)
(Pdb) s
--Call--
> /path/to/cwd/script_name(2)<module>()
-> x = 1
(Pdb) s
> /path/to/cwd/script_name(2)<module>()
-> x = 1
(Pdb) s
> /path/to/cwd/script_name(3)<module>()
-> y = 'blabla'
(Pdb) s
--Return--
> /path/to/cwd/script_name(3)<module>()->None
-> y = 'blabla'
(Pdb) s
--Return--
> /path/to/cwd/debug_script.py(21)<module>()->None
-> exec(code, globs)
(Pdb)

This patch is an attempt at allowing the source debugging of scripts executed by the Python exec() function. It misses tests and documentation.

You may use it using the idiom given in the following example to avoid stepping into the pdb code on the first invocation of pdb.exec_script() (see the exec_script() doc string):

import sys

def main():
    foo = 123
    s = """if 1:
        x = foo
        x = 555
    """
    exec_script(s)

if __name__ == '__main__':
    if '--debug' in sys.argv[1:]:
        import pdb
        exec_script = pdb.exec_script
        pdb.Pdb(skip=['pdb']).set_trace()
    else:
        exec_script = exec

    main()

Thanks. This is clever. I've tried it out and it works. Would it be more appropriate to use "importlib" and "importlib.abc" to implement a custom loader for a string script? It looks like importlib.abc.InspectLoader does the right thing.

This is a simple code object compiled from a source (the string), a module is quite different and more complex. The patch uses a fake module Loader to use linecache, there is no gain in going any further and pulling from the importlib machinery, I think.