After discussing this with Guido and Larry, we're going to go with the "implicit format mode", as outlined by Serhiy, and drop the !f feature.

So the rules are:
{x=} -> "x="+repr(x)
{x=:.2f} -> "x="+format(x, ".2f")
{x=:} -> "x="+format(x, "")
{x=:!s:20} -> "x="+format(str(x), "20")
{x=:!r:20} -> "x="+format(repr(x), "20")

I think the 95% case will be {x=}, the 99%+ case will be {x=:2f} case. So I'm happy with this outcome. All functionality you had available with !f is still available, but with slightly different spellings. The most common cases now have the shortest spellings.