Why the obvious method is broken
The instinctive way to build a saturation mask is to read saturation the way Photoshop's Hue/Saturation panel does: through an HSB-style model, where saturation is defined as a ratio relative to brightness. That definition has a built-in bias. As brightness drops toward black, even a tiny amount of color registers as a proportionally large saturation value. A dark shadow with a faint color cast, or plain sensor noise with a slight tint, ends up reading as highly saturated, sometimes more "saturated" than a genuinely vivid bright color sitting right next to it. Build a mask on that basis and you'll be isolating shadow noise along with your actual saturated highlights.
A more accurate read: Selective Color instead of HSB
Selective Color doesn't work in HSB space at all. It adjusts each pixel based on a CMYK-style ink read (how much of each printing color is present), computed directly from the RGB values, with no brightness-relative saturation ratio involved. That makes it possible to push fully saturated colors toward one extreme and fully neutral, gray tones toward the other, in a single deterministic pass, without the dark-tone bias that HSB-based methods carry.
Building the mask, step by step
This is the technique developed by Tony Kuyper and documented on his site goodlight.us, adapted here as a walkthrough:
1. Add a Selective Color adjustment layer, set to Absolute
Absolute mode applies a fixed shift regardless of how much of that ink is already in the pixel, which is what makes the next steps push every pixel to a clean extreme instead of a partial one. Relative mode scales the adjustment by the existing value, which won't get you there.
2. Push the color channels to one extreme
Go through each of the six color channels in the Colors dropdown, Reds, Yellows, Greens, Cyans, Blues and Magentas, and drag the Blacks slider all the way to -100% on each one.
3. Push the non-color channels to the other extreme
Go through the three non-color channels, Whites, Neutrals and Blacks, and drag the Blacks slider all the way to +100% on each one.
After both passes, every pixel in the image has been pushed toward black if it was carrying real color, or toward white if it was neutral or gray. All three RGB channels now hold effectively the same grayscale data, since the adjustment was applied uniformly across the composite.
4. Duplicate a channel and discard the adjustment layer
Open the Channels panel and duplicate any one of the three color channels. The Green channel is the common choice, mostly by convention rather than by necessity: since all three channels are now identical, Red or Blue works exactly the same. Delete the Selective Color adjustment layer once you've made the duplicate. What's left in that duplicated channel is your finished saturation mask.
Read it as: white where the image is neutral or desaturated, black where the image is strongly saturated. If you want to target the saturated areas directly instead, invert the channel with Ctrl/Cmd+I.
What it's actually good for
Ctrl-click the duplicated channel's thumbnail to load it as a selection, then use it to mask a Hue/Saturation or Vibrance adjustment layer. That lets you push saturation into the parts of an image that are still fairly neutral, a hazy sky, a flat foreground, without also amplifying areas that are already vivid into clipping. It works just as well in reverse, protecting skin tones (which read as relatively neutral in this mask) from a saturation boost meant for the rest of the frame.
Quick answers to a few related questions
Does this need to be done in 16-bit mode?
The technique itself avoids the HSB-related distortion regardless of bit depth, but working in 16-bit still matters once you refine the resulting mask further with Levels or Curves, for the same rounding-accumulation reason that applies to any composite-channel mask.
Why Absolute mode specifically, not Relative?
Relative mode scales the slider's effect by how much of that ink the pixel already has, so a pixel with only a little color left would only get pushed a little. Absolute mode applies the full -100%/+100% shift regardless, which is what drives every pixel cleanly to one extreme or the other in a single pass.
Is this worth turning into a Photoshop Action?
Given how repetitive the nine slider drags are, yes, it's a good candidate for an Action if you'll be building these masks more than a couple of times.
EN
IT