A subprocess looks simpler (and safer?) than a C thread with a pipe.

+ f = open(self.procfile, 'r')

'rb' mode is enough here, no need of Unicode ;-)

+ self.mem_watchdog = subprocess.Popen(..., stdin=f)

Can't you open the /proc/pid/stat file in the child process? It might be an issue with SELinux or Grsecurity, but I don't expect that our buildbot use such security patch.

+ statm = sys.stdin.read()

file_watchdog() did read only 1024 bytes. I don't know if it would be better or not to use an arbitrary limit (to avoid filling the memory?).

You should catch OSError here.

You may only display the memory usage if it changed and start by a sleep.

--

You may also write the watchdog script in .py file instead of passing it on the command line, so it will be easier to maintain it later.