Smoothing Selection Edges

[TonyW]

Ran into a clever method for refining selection edges in Katrin Eismann’s book Photoshop Masking and Composition. Although her method uses quick masks and curves it can easily be adapted to work in Elements – even without add-ins.

1)Make your initial selection using whatever method you want (for the example of a bow on a white background I used the magic wand)
2)Add a levels adjustment layer, group it and click OK (to give you the mask)
3)Alt-click on the mask to show the mask
4)Apply a low Gaussian Blur (0.5 to 1) to soften the mask edges
5)Ctrl-L to bring up levels and move the middle slider to the right – which contracts the mask slightly (click preview off and on to see how much you moved it)
6)Use the magic wand at a high tolerance to select the white part of the mask.
7)Click back on your image and you’ll now have a smoother selection.

If you have Grant’s tools you can skip steps 2 and 3 and use Quick Mask instead and for steps 6 and 7 use Masks Standard. Doing it this way has the advantage that you can see the image while you’re applying the blur and mask contraction with levels.

Bow on the left is selected (from a white background) with the magic wand, one on the right is after refining the selection with a Gaussian blur and level adjustment of the mask.

http://www.pixentral.com/hosted/15wY6nsYnhRXo7xCS4VGEHmoVZfhQ_thumb.jpg (http://www.pixentral.com/show.php?picture=15wY6nsYnhRXo7xCS4VGEHmoVZfhQ)

Tony

[AngelicKim]

Tony, thanks for the tip. I keep meaning to ask for a way to smooth the edges as I hadn't figured it out yet.

[kimi_boo]

Tony you really have to look at the big image to see the difference. I am not sure if I understand it. This is one of those things I will need to print and actually do to get it. Thanks for the info. :D

[Byron Gale]

Very cool, Tony... thanks!!

[margtvl]

Tony,
Should I be copying my selection to a new layer after step 1?

[bayhli]

Definitely smoother... thanks for the useful tip, Tony!

[TonyW]

Tony,
Should I be copying my selection to a new layer after step 1?

You don't need to - the levels adjustment layer has to be grouped with the layer that has the selection on otherwise you won't get the mask.

Tony

[Wendy]

Hi Tony ...

This really is a neat way to do edges ... I like it :)

Wendy

[Russinator]

Tony, thanks for the tip. I will give it a try soon. Russ :)

[w7vp]

Tony
My biggest problem is defringing selections of sky through trees. So far I have not seen any techniques that do not leave a lot of artifacts.:(

[TonyW]

I know the problem - but I haven't finished the book yet :) . It's excellent and covers just about everything you'd ever want to know about selecting, masking and merging images. It's written for Photoshop so many of the techniques need to be adapted for Elements and some you just can't do.

One thing I have found with sky replacement with trees is it helps to get rid of fringing first if you can. Some is lens chromatic aberration and I use PTLens to minimize that. Most is purple fringing and I've used the color replacement brush for that. If I come up with a drop dead sky replacement method I'll be sure to let everyone know :D

Tony

[Benny Pedersen]

Tony
My biggest problem is defringing selections of sky through trees. So far I have not seen any techniques that do not leave a lot of artifacts.:(

Ctrl+A,Ctrl+C,Alt Click,Ctrl+V,Ctrl+D,
http://hjem.get2net.dk/b_pedersen/gallery/fld/a/how.jpg
Benny

[kabe83]

I'm not understanding. I have my selected image on the top layer, grouped with a levels adjustment layer. If I Alt click on the mask, all I see is white. If I activate the selection, I can fill the mask layer with black where my selection is and apply blur, but levels doesn't do anything. Where am I going wrong?:confused:

[Richard Lynch]

My biggest problem is defringing selections of sky through trees. So far I have not seen any techniques that do not leave a lot of artifacts.:(
Do you mean artifacts like you get from JPEG compression? Why not post a (difficult) sample image and we can work up a method right here. It may require a little fancy dancing ;-)

[Richard Lynch]

Alt Click
Benny, I thought I knew them all (and then some)...nice trick!
I'm not understanding. I have my selected image on the top layer, grouped with a levels adjustment layer. If I Alt click on the mask, all I see is white.
Kabe, that IS the point. You can turn anything you have collected in your image by copying and paste it into the Levels mask because of Alt-click (option-click on Mac). You are not using Levels so much as you are borrowing the mask -- though you can mask the levels with the mask you created. Benny may have been a little minimalist in his explanation...

You want to have a layer with content (e.g., Background) active before selecting All and Copying. Alt-click on the mask to view it and make it active. When you paste, a grayscale version of the layer you copied will fill the mask.

Does that help?

[GaryK]

I just tried the above.. had some trouble getting it to work but finally.
I used a coloured bottom layer and a different coloured clipped layer, did a similar gradiant as Benny showed on the bottom layer.

I then pretended that the fuzzines around the gradiants was the fringing.. not exactly the same but close.
I ran a levels adjustment on the mask this got rid of the "background" (I had a grey throughout because of the coloured bottom layer)
I then ran a threshold adjustment on the mask to get rid of the fuzzines around the gradiants.

Think this would work with "tree" problem?

One thing.. I couldn't do an adjustment layer on the gradiant. I had to use the menus for the level and the threshold.


PS... I'm still struggling with how I got Benny's thing to work??

[GaryK]

http://www.pixentral.com/hosted/1WWSOe9jbHoyX4kkHstOKpqOYjlVb_thumb.jpg (http://www.pixentral.com/show.php?picture=1WWSOe9jbHoyX4kkHstOKpqOYjlVb)

Sorry I forgot to resize :rolleyes::rolleyes:

Notice the size of the mask gradiants compared to the original (trees)
I could have made it smaller by more threshold tweaking.

[TonyW]

Richard, Benny: Interesting discussion - I've been experimenting along much the same lines but with a black/white gradient map above the image so I can tune the mask to the image (usually with a white sky you can mask everything except the sky by sliding the black slider way over to the right. I got that idea by reading Richard's book of course :) . On the mask I've been using a Gaussian Blur (1 plus or minus a bit) and a Filter>Other>Maximum 1 to smooth the edges. Not quite perfected but it's coming and does do a fairly good job on trees other than the purple fringing. Have a couple of ideas I'm trying to fix those but if you have any other suggestions to try :)

So my layers would look something like:

359

And here's my test case - a particularly bad case of fringing - I figure if I can turn the sky blue on this one I'm doing well :D

http://www.pixentral.com/show.php?picture=1pXZjyOM0bprqt41qyYc2Opmk7mRO1

Tony

PS - Sure your ideas Gary - going to have to take a look. If anyone wants to play with my test case feel free to play :D

[Benny Pedersen]

Benny, I thought I knew them all (and then some)...nice trick!

SNIP... Benny may have been a little minimalist in his explanation...



http://hjem.get2net.dk/b_pedersen/gallery/fld/a/how.jpg

In this case I ignored the G.blur along with Levels on the mask (as Ricard saw; not the build in levels on the mask but Ctrl+L on it). But it does work fine to get a better selection. Now said but infact I also ignored the selection task (somehow impossible with trees) and instead did focus on a task/result.

Ex.
Bird: http://hjem.get2net.dk/b_pedersen/gallery/fld/040627_050656_01.jpg
Sky: http://hjem.get2net.dk/b_pedersen/gallery/fld/040701_091826.jpg
Both: http://hjem.get2net.dk/b_pedersen/gallery/fld/040627_050656_.jpg

I did not make any adjustments on the mask, so after "Merge visible" just repeat a mask to get the rest but this time paint on the mask. Since the new "source" after merged has less contrast on edges, the result would be very fine. Im sure also for trees. That was what I tried to explain in my lesson 2
http://www.photoshopelementsuser.com/forum/showthread.php?t=12411&highlight=lesson2
but my english isn't so good.

Never did a selection (only the trick),
http://hjem.get2net.dk/b_pedersen/gallery/fld/050710_180319.jpghttp://hjem.get2net.dk/b_pedersen/gallery/fld/050710_175826.jpghttp://hjem.get2net.dk/b_pedersen/gallery/fld/050710_180319_.jpg

And my trick (I never read books) also solve other problems such as to much contrast (sun in front), http://hjem.get2net.dk/b_pedersen/gallery/gallery.htm?37

Benny

[Benny Pedersen]

Richard, Benny: Interesting discussion - I've been experimenting along much the same lines but with a black/white gradient map above the image so I can tune the mask to the image (usually with a white sky you can mask everything except the sky by sliding the black slider way over to the right. I got that idea by reading Richard's book of course. On the mask I've been using a Gaussian Blur (1 plus or minus a bit) and a Filter>Other>Maximum 1 to smooth the edges. Not quite perfected but it's coming and does do a fairly good job on trees other than the purple fringing. Have a couple of ideas I'm trying to fix those but if you have any other suggestions to try :)

So my layers would look something like:

359

And here's my test case - a particularly bad case of fringing - I figure if I can turn the sky blue on this one I'm doing well :D

http://www.pixentral.com/show.php?picture=1pXZjyOM0bprqt41qyYc2Opmk7mRO1

Tony

PS - Sure your ideas Gary - going to have to take a look. If anyone wants to play with my test case feel free to play :D

Whole process in cardboard.jpg
http://hjem.get2net.dk/b_pedersen/gallery/fld/a/cardboard.jpg

Benny

[TonyW]

That's good Benny. The nice thing about having the masks in place is that you can just drop in any sky you want into the sky layer. I'm trying to do it without using any painting or selection which is why I used the gradient map layer to mask everything except the highlights. That way I can automate the process with an action. I have one that works fine on most images but it still has trouble with filling in the small holes in trees.

Tony

[Benny Pedersen]

That's good Benny. The nice thing about having the masks in place is that you can just drop in any sky you want into the sky layer. I'm trying to do it without using any painting or selection which is why I used the gradient map layer to mask everything except the highlights. That way I can automate the process with an action. I have one that works fine on most images but it still has trouble with filling in the small holes in trees.

Tony

Solved
http://www.photoshopelementsuser.com/forum/showthread.php?t=13616

Benny