This happens because the role is pretty much ignored by Sphinx for linking purposes.  I remember Georg talking about the reason for this, but I forget what it was.  Perhaps it would be possible to, as you suggest, have the roll affect the priority...but it may be the case that the references don't currently retain the needed information.  Regardless, that would be an issue for the Sphinx tracker.