I checked with a couple grep programs and they use depth first. Which makes sense, since you'd want the return order to be something like:

/a/spam.txt
/a/aa/spam.txt
/a/bb/spam.txt
/x/spam.txt
/y/spam.txt
/z/spam.txt

...instead of the bread-first:

/a/spam.txt
/x/spam.txt
/y/spam.txt
/z/spam.txt
/a/aa/spam.txt
/a/bb/spam.txt

However, it turns out this is moot since the first thing GrepDialog.py does with the returned results is sort them.