The scaling is relative the current viewport, so we need to make sure it is correct before changing the scale setting.