Masking: the edge

[skydog]

When masking or cutting out an object (like a person) to place in a new background, I need some pointers to improve the blend of the edge of the cutout with the background. I'm not getting a smooth transition. I tried Layer>matting...and not getting the results I'd like using this. I know I've seen a tutorial on this in the past but no longer know the tips that were provided. thanks

[pixelzombie]

try the blur tool in conjunction with the burn tool on the layer mask, i sometimes use a curve move to get the edges to come in and then selective blurring from there...

[TommyO]

I've seen many methods over the years, all work well. A common one is similar to pixelzombie's suggestion. It involves using Modify>Border. First, Ctrl-click the layer icon of your subject. This will load the transparency of the layer. Then, click Select>Modify>Border and use a value of roughly 5 pixels (depending on the density of your image of course). Hit Ctrl-J to copy this to a new layer. Then, apply a low value Gaussian Blur to that layer and adjust the opacity to your liking.

This should soften just the edge, as is often the case in real image edges of people, etc. Other objects, like piano's, don't usually need this.

And of course, don't forget the shadow. That is usually what throws our eyes off.

[edgework]

The problem is always with the transition pixels between masked object and old background. Leave them in, you have a dark or light halo. Pull in your mask edge, you have a scissors cut-out. And a global solution, such as blurring and tightening with levels ignores the fact that the edge for something like jeans is going to require different treatment than hair.

Leave the halo. Mask as best you can but don't try to cut so deeply into the silhouette that you destroy the natural edge..

Cut out enough of the new background to cover the masked object and dupe it to a new layer on top.

If the halo is lighter than the background, set this new layer to multiply. if the halo is darker, set it to screen.

Group to the masked object and hide with a layer mask.

Use a small brush at low opacity, peferably a tablet, and paint white into the mask along the halo.

It's pretty easy to "creep up" on the pixels with gradual strokes, bringing them to the same level as the background. And, considering that it actually works, it's not that time consuming.

Usually it's the best way to capture flyaway hair, fuzzy sweaters and any pixels that would naturally blend into the background.

[nevie]

Whenever I make a selection I feather the edges, that usually helps to keep transitions pretty smooth. Or maybe try refine edges and work with your selection to get the best transition.

[jagerman]

Once you have your base mask. Command>click on the mask to make it into a selection. Then head to the "select" pull down menu and start playing with commands like contract, expand, smooth and inverse. Brushing areas back in and out till it is nice and clean. No feathering. Once I drop the image into my base image, blur the mask a small amount (.5-1) until the edges blend naturally.

Extra tip - keep your unfeathered (no blur) mask and add a second mask and blur that one. Turn your original mask off (shift>click). That way if you over do it you can always go back.

Hair, smoke, bubbles etc.. is a different process.

Good Luck, -W

[cricket1961]

You can try loading the selection for the existing mask, inverse the selection, and then loading the original selection back in using intersect. Sometimes a very slight feathering, say .2 or.3 done once at the time of your choosing, can help.

This should give you a good mask for any overlapping area to use a curves adjustment layer to lighten or darken the area for the halo.

Its also a good way to use the resulting selection to do a copy all and paste and then doing a slight blur of your choice.

Hope it helps Sky.

Chris

[Markzebra]

Yes, you need to burn or dodge the edge pixels - no mask is every really good enough

[skydog]

thanks for the pointers

[Karlington]

I recommend watching Deke McClellands Lynda-tuts on Channels & Masks. His way of cutting out an object is extremley effective and timesaving (depending on how precise you want your cut out to be). He uses a channel (that as you know works as an alpha channel) and tweeks it by using levels, painting with overlay etc to get a high quality mask. Perfect for jeans as well as hair – all at once! Use it all the time!