Photoshop masking techniques
There are numerous reasons for making a mask
in Photoshop: to isolate an area of an image for
color correction, to separate an element from the
background for compositing or for localized special
effects. And there are a number of ways to make
those masks.
In Ye Olden Days—before pixels—we twirled
delicate knives to cut out silhouettes with Rubylith
masking material. We hunched over light tables,
squinting at details as we traced the edge of a flower,
a hand or—ack!—bicycle spokes. Then we gingerly
peeled out little segments of the Rubylith lacquer,
leaving clear base. You kids don’t know how good
you have it. It’s so much easier to play with pixels.
Safety bonus: You can’t cut yourself with a mouse.
Hate the Pen tool?
Don’t feel bad; you’re not alone. It takes time to
master the DPT (Dreaded Pen Tool). Simple geometric
corners and curves aren’t difficult (figure 1),
and a skilled wielder of the nib can draw quite complex
shapes. But attempting to outline subjects such
as soft hair or busy foliage would drive even the
most fervent Bézier devotee crazy—and the results
would be harsh and unrealistic.
Unless you want a crisp, hard edge on silhouettes,
put down the Pen tool: There are far better
ways to select irregular shapes. And you can make
Photoshop do most of the work.
Selection techniques
While you can use the Lasso tool to select most
shapes, it’s tedious and—like the Pen tool—doesn’t
lend itself to isolating complex organic shapes such
as fur or hair. Instead, you should consider using
selection tools that follow pixel edges based on density
or color, so the image itself helps you generate
the mask.
Magic Wand
The Magic Wand tool selects color ranges based on
your initial click. The Tolerance setting (figure 2)
governs how wide a range is selected—the higher
the tolerance, the wider the range of colors selected.
To add to the selection, hold down Shift and click
in additional areas. To subtract, hold down Option/
Alt and click to remove an area. For isolated objects
with fairly uniform color, the Magic Wand works
well, but it can be tedious to select multiple small,
isolated areas—such as individual flowers in a field.
In such a case, turning off the Contiguous option allows separate, similar color areas to be selected.
And it’s tough to select soft components such as hair
without a harsh edge.
New in Photoshop CS3
After all these years, the Wand gets an upgrade.
And we get the ability to finesse a selection before
it’s finalized. How did we live without the Quick
Selection tool and the Refine Edge option?
Quick Selection tool (or Son of Magic Wand):
This new selection tool is now the default tool in
the old Magic Wand slot in the new Photoshop
CS3 toolbox. Like the Wand, the Quick Selection
tool allows you to select areas based on color. But
instead of shift-clicking to add to the selection, just
draw across the object, and a selection is created. You
can shift-click to add to the selection. But you’ll be
amazed by how adept the new tool is at selecting just
what you want to capture—if you drag it across a
representative range of colors, you’ll often get exactly
the selection you want with the first pass.
Refine Edge: Whatever method you’ve used to
make a selection, the new Refine Edge option can
improve it. Use the sliders to expand or contract a
selection, or add a feather. Your favorite slider, however,
will probably be the Smooth control. Use it to
subdue pixelation without deforming the outline.
You can judge the results before finalizing the settings
by toggling the mask appearance—use the five buttons
at the bottom of the Refine Edge dialog to see
how it will turn out (figure 3).
Extracting an image
Choose Filter > Extract to isolate irregular edges and
erase the background pixels. Use the Highlighter
tool to indicate the edge of the object, and use the
Fill bucket to tell Photoshop where the opaque
interior is. If there’s fairly decent contrast between
the element and background, the results can be very
good. However, the Extract operation deletes all
background pixels, so if you accidentally erase part
of the subject, it’s gone.
Saving selections
Once you have a satisfactory selection, what should
you do with it? If it’s for a one-time use, just deselect
and go on with your life. But if you plan to use the
selection again in the future, you’ll need to store it
in an alpha channel—an active selection evaporates
when an image is saved (the ants march off ...).
In a CMYK image, the first four channels represent
the four printing plates (inks). In an RGB
image, the first three channels represent the three
primary colors of light—red, green and blue. Alpha
channels are extra channels—beyond the color channels—used to store masks in a tangible form. Perhaps
it’s unfortunate that the Channels palette looks so
much like the Layers palette—they represent such
different concepts.
To save a mask as an alpha channel, start with
an active selection and choose Select > Save Selection.
You can accept the default name “Alpha 1” or give
it a custom name. To activate the mask later, choose
Select > Load Selection and select the name of the
alpha channel you wish to activate.
At its simplest, an alpha channel mask consists
of a white “hole” in a black background. These masks
function like stencils: The “opening” in the mask
allows pixels to be visible, or allows you to paint or
apply an effect in a localized area. The black area
acts like the cardboard of a stencil, hiding pixels. But
alpha channel masks aren’t limited to just black and
white, and that’s where the fun begins.
To create a mask based on a channel, start by
examining the individual channels of the image.
Open the Channels palette (Window > Channels),
and click on the individual channel names to display
them one by one. Determine which channel would
give you the best head start on creating a mask
(figure 5).
When you find the best channel, duplicate it by
choosing Duplicate Channel from the palette menu.
You can give it a meaningful name, or just accept the
default name and click OK (figure 6).
Photoshop will display the new alpha channel.
You’ll need to modify it in order to create a nice,
clean white opening for the subject (the dandelion)
and a good, solid black background. One way to do
this is to use Levels (Image > Adjustments > Levels).
To darken the background, move the black triangle
to the right until you see the background become
solid. To lighten the interior of the mask, drag the
white triangle to the left (figure 7). You want a clean
black-and-white mask, but you don’t want to go too
far or you’ll erode detail. So push the black triangle a
bit to the right, then move the white triangle a bit to
the left. Toggle back and forth between the controls
until you have a decent mask. If necessary, use painting
and drawing tools to refine the mask, and then return to the composite image by clicking on the
CMYK name (not just the eyeball) at the top of the
Channels palette. In an RGB image, you’d click on
the RGB name.
At this point, the mask is “sleeping”—it’s not yet
functioning as a mask. To activate the mask, choose
Select > Load Selection and choose the name of your
mask channel from the Channel pop-up menu. You’ll
see the “marching ants” marquee around the mask
shape, indicating an active selection. At this moment,
you could paint with a brush or perform a color correction,
and the effect would be limited to just the
area of the image that’s within the active selection.
But to use the mask as a silhouette, you need to convert
it to a Layer Mask. First, you’ll need to unlock
the Background layer.
The Background layer in Photoshop behaves
like a photo print glued down to a piece of foam
board: You can’t put anything under it, and you can’t
have transparency in the layer. To allow the mask to
silhouette the subject from its background, you must
first “unglue” the Background layer. In the Layers palette,
double-click the name of the Background layer;
you’ll then see the New Layer dialog. You can give
it a fancy name, or you can just accept the default
name: Layer 0. Click OK. There’s no change to the
appearance of the layer, but notice that the name of
the layer has been changed in the Layers palette.
To convert the “marching ants” to a Layer Mask,
click the mask icon at the bottom of the Layers palette
(figure 8). Now the mask can separate the subject
from its background without destroying pixels.
Once you create the layer mask, you’ll see the mask
icon next to the image icon in the Layers palette.
This is a hint that you have two occupants in this
layer: the image and its mask. You can work separately
on mask and image—just click the appropriate thumbnail. To view the mask alone, Option/Alt-click
on the mask thumbnail. And you can Shift-click the
mask thumbnail to disable the mask.
Note that, however you create it, any active
selection can be converted to a layer mask.
Preserving transparency: saving for other
applications
Layers and transparency in .psd files will be honored
by Adobe Illustrator and InDesign, as well
as QuarkXPress 7.0. Thus soft edges are retained
when you place an unflattened native Photoshop
(.psd) file into those applications. Note that,
although the TIFF format does support layers
and transparency, a .psd is smaller on disk than
an equivalent TIFF (not that you care, with that
big, honkin’ 120 GB drive). And, while InDesign
does recognize transparency in a layered TIFF, it
does not allow you to use Object Layer options to
manipulate the visibility of layers in a TIFF. Also
keep in mind that the old-fashioned EPS format
does not support transparency or layers, and is
useful currently only when saving duotone images
(InDesign supports duotone-native Photoshop files,
but QuarkXPress does not).
