From "man font":

FONT OPTIONS

       -size size
              The desired size of the font.  If the size argument is  a  posi‐
              tive  number, it is interpreted as a size in points.  If size is
              a negative number, its absolute value is interpreted as  a  size
              in pixels.  If a font cannot be displayed at the specified size,
              a nearby size will be chosen.  If size is unspecified or zero, a
              platform-dependent default size will be chosen.

              Sizes  should normally be specified in points so the application
              will remain the same ruler size on the screen, even when  chang‐
              ing screen resolutions or moving scripts across platforms.  How‐
              ever, specifying pixels is useful in certain circumstances  such
              as  when  a  piece of text must line up with respect to a fixed-
              size bitmap.  The mapping between points and pixels is set  when
              the  application  starts,  based  on properties of the installed
              monitor, but it can be overridden by calling the tk scaling com‐
              mand.

If the font size is specified in absolute pixels the scaling factor does not affect it.

See fix_scaling() in Lib/idlelib/run.py for example how to handle this problem.