Coming back to this, I noticed a number of things wrong with the logic that I'm not sure how I missed before.  I had to rewrite pretty much all of the logic.  The good news is the patch is now smaller :)

The new tests are failing, because they are wrong: the msg argument to sendmail/mail *must* be a byte string if smtputf8 is in effect, since an email message may contain multiple encodings as well as other binary data.  (Well, the msg argument could be an ascii string, but what would be the point?)

I'm uploading this without fixing the tests because I'm out of time for now and may not get back to it before the weekend.  Perhaps someone else will have time to work on the tests before then.