I'm retouching a picture, but something weird is happening here

I'm a bit confused! right this moment my marquee tool won't cut smooth edges, the layer mask will not give me smooth edges if I use rectangular marquee tool filled with black
...even gradien tool is not working!
I know someone here knows the answer...please help!
Gerry