Photoshop Tip: Masking 101
A mask is a selection that lets you isolate a subject or specific area of your photo. It's analogous to using masking tape to cover your baseboards before you paint; once the mask is in place, you can make edits (paint the wall) without affecting the areas that are masked off (the baseboards). Adobe Photoshop allows you to do much more than paint walls, of course. Here, I'll focus on isolating an object and knocking out the background so that the object can be placed on other backgrounds.
There are many different approaches to masking, but because this is an introduction, we'll explore three core techniques that you can combine and refine for the best results with either Photoshop CS3 or CS4. (I'll skip the discussion of CS4's Masks panel, which collects a number of mask controls into a convenient interface, but strictly speaking, does not introduce anything that cannot be accomplished with CS3).
The concept of masking in Photoshop can be difficult to grasp, so let's first examine how it works. Open a photo and then, from the toolbar, choose the Lasso Tool. Using the mouse, draw a crude selection around something in the photo. Now press Q on your keyboard to enter Quick Mask Mode. The photo will be bathed in red, except the part you selected.
This red area represents the image mask and it's fully editable. A Photoshop mask is actually a grayscale image, shown in transparent red so that you can see both the mask and your photo at the same time. Now, choose Window -> Channels and you will notice five items: RBG, Red, Blue, Green, and Quick Mask. Clicking the eye icon next to RGB lets you see the mask in its basic grayscale form; the selection you made earlier is represented by white, while the masked area is represented by black. As an experiment, use the Brush Tool to paint a white line across the mask. Click the eye icon next to RGB to turn the color channels back on and then press Q to leave Quick Mask Mode. Notice that the selection has changed and now includes the line you painted across the photo.
Now that you understand the basics of a mask, it's time to move on. Choose File -> Revert to toss out your edits and start over.
Technique 1: Magic Wand Tool
Photoshop's Magic Wand Tool can help you make selections of areas with a limited color range. It's the tool to use when the object you want to isolate is cleanly defined against an unvarying background, such as the surface of a table or a cloudless sky.
In this example, I'm using a picture of a cookie that has been photographed on white poster board (see the full image in the middle of this page). Select the Magic Wand tool to start. In the Tool Options bar (Window -> options), you'll need to configure three important items: Tolerance (which should be set to 32 by default), as well as Anti-alias and Contiguous (both of which should be checked). Now take the Magic Wand tool and click the white area around the cookie. A significant portion of the white area will be selected, but perhaps not all of it. Hold down the Shift key and you will see the icon change to include a plus sign, indicating that you want to add to your current selection. Move the cursor over an unselected area and click it; this area will then join the selection. Continue to do this until the entire area around the cookie appears to be selected.
Depending upon where you click, the selection may leak into the cookie itself. Zoom into these areas and then reselect the Magic Wand. Change the tolerance of the wand to 12, and this time, hold down the option key. The cursor will now include a minus sign, indicating that you want to subtract areas from the selection. Now, with the Option key depressed, click all areas within the cookie that shouldn't be selected. (The lower tolerance should prevent the subtractions from spilling outside of the cookie.)
Once you're happy with the selection, you'll need to invert it by choosing Select -> Inverse. (It's easier to isolate the cookie by selecting the background, but ultimately, it's the cookie you want to see.) Further adjustments to the selection can be made with Technique 3.
Technique 2: Lasso Tool
The Lasso Tool lets you create a selection by drawing a line around the object. You'll need to use the Lasso when it's effectively impossible to select chunks of the background using the Magic Wand; for example, if you want to isolate a picture of a bird that's set against a forest background.
We'll use our cookie image again in this example just to keep things simple. Zoom in to where you can comfortably fit the image in the window on your screen (unlock the layer if you have to by double-clicking it) and then choose the Lasso Tool. In the tool options, you'll want to ensure that Anti-alias is checked and that Feather is set to 0. (Feathering can be applied to the selection later.)
Unless you're highly skilled and extraordinarily precise with a mouse or track pad, attempting to accurately encircle the cookie by clicking and dragging will be absolutely futile. Instead, hold down the option key as you click the edge of the cookie (which gives you the Polygonal Lasso Tool). A line then appears between the point you clicked and the cursor. Move the cursor along the contours of the cookie and click again and again (never letting go of the option key) until you arrive back at your starting point. Make sure to join the start and end points. You will now have made a full selection of the cookie using very short, straight-line segments. If you've made a few small errors, you can always edit your mask using Technique 3.
Technique 3: Brush Tool
In some cases, it's easier to make an accurate selection within the Quick Mask Mode. Once again, we'll use our cookie image as a starting point. Press Q to enter Quick Mask Mode; assuming you haven't made a selection already, you will not see any red overlay. Press D to return to the default foreground (black) and background (white). Select the Brush Tool and control-click (or right-click) on your photo to call up the Brushes panel. You'll want to configure the brush so that it accurately reproduces the softness of the cookie edge; in the case of the cookie, we've chosen a diameter of 15 and a hardness of 85. Make sure the Mode is Normal and the Opacity is 100 percent. Now simply paint the inside of the cookie, matching the contours as closely as possible and using a larger brush to mask the cookie's middle section. Zoom in and use smaller brushes to paint the edge areas in finer detail, and if necessary, use the Eraser Tool to clean up mistakes.
To finish up, you'll need to invert the mask by choosing Image -> Adjustments -> Invert. You'll often find that you need to make small edits to your Magic Wand and Lasso selections, and those too can be made with the Brush Tool. Enter Quick Mask Mode by pressing Q, and you'll see the masked area represented in red. (Feel free to invert the mask for editing, as indicated above.) Now simply use the Brush and Erase tools to fine-tune the mask.
Saving the mask
Once the mask is complete, it's important to save the selection. Press Q to leave Quick Mask Mode and then choose Select -> Save Selection. Enter a name for the new channel and click OK. This channel functions as backup of your mask, allowing you to refine it (see below) without worry. In the event that you ruin your mask during the refinement process--which can happen--just reload the original mask and start over. Choose Select -> Load Selection; when the dialog box appears, simply select the mask channel from the pop-up menu and click OK.
Refining the mask
Creating a mask without any degree feathering will almost certainly leave sharp lines that do not match the soft contours of the photo. If you were to proceed with the mask as it is, the object you want to isolate will look like you've used scissors to cut it out of a magazine. Most often, you'll want the edges of the object to look as natural as possible, fading softly into whatever color background you choose.
Hit Q to leave Quick Mask Mode (if you haven't already) and choose Select -> Refine Edge to call up Photoshop's edge-refining controls. At the bottom of the window, there are five icons that let you see the mask in different ways. Click either the third icon (to view on black) or the fourth icon (to view on white), whichever makes the edges more obvious. Tug the Contract/Expand slider to the left to contract the selection until the edge auras disappear. Now increase the feathering slightly so that it matches the softness of the edges in your photo. (If you bump up feathering significantly, you may need to further contract the selection.) Click OK to continue.
Knocking out the background
Now it's time to apply your mask. Go to the Layers panel by choosing Window -> Layers. If your current layer is called Background, double-click the layer. Give the layer a name and click OK. At the very bottom of the Layers panel you will see a row of icons. Click the third icon from left to add your current selection as a layer mask. This instantly knocks out the background, making it fully transparent. Save your document as a Photoshop file (if you haven't already) to preserve both your layer masks and channels. Now you can choose File -> Save As and save a copy of your file as a PNG for use online or in other projects!
[Chris McVeigh is an author, illustrator, and toy photographer based in Halifax, Nova Scotia.]