posterized edges of a layer mask

[captain_j_hook]

Hello to everybody (: I am brand new and I've found this website through Scott Kelby's book on retouch!!! It's great, I look forward to becoming an active member of this community *_*
Anyway. I have a huge issue that from time to time occurs to me driving me mad and put me in complete discouragement...

Sometimes when I am working on the edges of a mask it gets sort of posterized. It doesn't matter whether I am using gradient tool or brush with low flow and zero hardness... basically the edges don't go smooth and there is a harsh transition that reminds me of posterization and it seems I am using a hard edge brush. It can't be blurred, it can't be feathered via mask panel, whatever I do the posterization stays in place or slightly changes position, no dent at all.

I know posterization is due to information gaps and I know about 16 bit theory (but Dan Margulis affirms 16 bit or 8 bit doesn't matter in reality... anyway! interesting topic for a future thread!), smooth transitions in the file reflected in the histogram... I know these things. But we're talking about a mask! A mask for a layer that has to be blended with the one underneath. A simple mask made from scratch, not even obtained via a channel which has been destroyed with some crazy curve to force black and white... a pure simple mask where I simply start painting white or black but still- it is posterized.

I don't get what I am doing wrong! And sometimes this occurs even to a 50% gray soft light layer and I want to paint big brushes and then I put a work curve on top to make sure I am not doing any silly stuff... even those big brushes are posterized sometimes.

I've read a thread on Adobe forum and people were talking about 16 bit workflow and the importance of a good monitor... I have a brand new Eizo GC 247 properly calibrated and I don't think it can be that... And of course it can't even be the 16 bit thing neither because we're not talking about actual informations in the file but the "informations" contained on a layer mask!!!!

It's probably something stupid and I am probably doing something ridicolous but.... I really would like to finally find out!!!! If you could give me your opinion about it that would be great.

Sorry for the endless post.

[Doug Nelson]

Re: posterized edges of a layer mask

First of all, welcome!

Can you determine if this "posterizing" is actually happening, or is it just a visual artifact? If you apply an adjustment or filter to the mask does the effect remain? Can you post a screencap? (zoom in to keep the filesize under the limit)

[captain_j_hook]

Re: posterized edges of a layer mask

Hello and thanks for your reply!
The thing is: I'd like to know if it is an artifact or not, if it is affecting the image or not... There is no big telltale difference in the actual image but this doesn't reassure me, since most of the times things seem fine to the unaided eye but then you go to info panel and check numbers or enable a work curve or an invert adjustment layer and you realise things were not fine at all... so I don't really trust what I see.

Here I've painted a random mask two times , white and black.

[Flashtones]

Re: posterized edges of a layer mask

looks normal enough on my laptop.

maybe a problem with your video card? if you're on windows are your drivers up to date?

edit: normal enough means it's not uncommon to get a little banding like that in 8-bits and pure gradients. you can add some noise to it, but that can sometimes show thru to the image. kinda depends on the nature of the image.

going up to 16-bits can help on the masks and gradients. i think even dan concedes that.

[Doug Nelson]

Re: posterized edges of a layer mask

I see virtually no banding in your attachments, which would indicate it's your monitor. You can objectively measure this using the info palette. Run your cursor over the "band" and see if the numbers jump right at that point. I suspect they won't.

That being said, too smooth of any gradient can cause display and printing problems. It likely wouldn't cause a problem on a mask, but it is a trap that many of us have fallen into by trying for perfection only to have our hardware rebel against the concept.

[captain_j_hook]

Re: posterized edges of a layer mask

Oh YESSS of course!! The info palette could have told me! And yes Doug, you were right: no jumps. [RGB] 45-46-47-48-49... there's no interruption!!! :_: I've tried to do a mask with a circular gradient as well. Shot a printscreen, opened it on PS like I did with the one I've posted here and no jumps there as well. So it's me period. -_- Me and my screen. I could have understood my macbookpro screen but what's with the EIZO? It must be a video card thing as Fleshtones points out... I set PS like this in preferences: enable openGL, mode: base, vertical synchronization and anti-alias checked. My video card is the one that came with my good old macbookpro: ATA radeon HD 6750M with 1 GB dedicated (but I am not sure this can turn out useful for what we need in PS actually... I think it is good for 3d and renderings but I may be wrong).

[Doug Nelson]

Re: posterized edges of a layer mask

Do you calibrate?

[captain_j_hook]

Re: posterized edges of a layer mask

Yes! SpyderPro3.