Levels can exagerate tonal differences, which can appear to make mask edges expand and contract (I actually use this deliberately for this purpose, I posted a demonstration with a feather
here), but with a well-made mask there should be no difference in appearance from painting directly on the image or on a color layer.
Try this: in QM mode, doubleclick on the QM icon and change the opacity to make the mask completely opaque. You can even change the color. I'll normally save this for a final fine-tuning pass, but it is a great help. Once you've made your best mask, leave QM mode and save the selection.