The difference in behavior you are seeing does indeed have to do with the PATH environment variable.  When you start IDLE via a terminal shell, it inherits the PATH value from your shell environment.  path_helper(8) sets the default value for a login shell PATH by consulting the entries in /etc/paths.d/. On OS X 10.6, that includes /usr/X11/bin, which is where xterm is located.  However, when you launch IDLE from the Finder, either by double-clicking on the IDLE app icon or by opening a file using IDLE as the default app (as you are doing in test 1), a shell is not involved and the PATH inherited by the app environment is slightly different.  In paticular, /etc/paths.d is not consulted and so /usr/X11/bin is not on the path.  As Amaury suggested, you should be able to see that by looking at PATH in both cases.  For IDLE.app launched from the Finder, you'll probably see something like:

   >>> os.environ['PATH']
   '/usr/bin:/bin:/usr/sbin:/sbin'

While it is possible to change the default environment variables for processes launched (see http://developer.apple.com/library/mac/#qa/qa1067/_index.html), it is rarely necessary or desirable to do that.  For this case, the simplest solution is to supply the absolute path to 'xterm':

  import subprocess
  subprocess.Popen(['/usr/X11/bin/xterm'])

Or you could get fancier by modifying PATH yourself.