Using Vector Mask for best quality

You must have noticed that when you use layer mask, it creates a soft edged mask around the selected area of the object. Later when you transform such masked area, you will get jagged edged object. The quality of the image degrades if such image is transformed or converted into higher resolution later on.

We can avoid this by using vector mask. Make a selection using a path or take an object created using shape tool. Then convert the path or shape into vector mask (Layer > Vector Mask > Current Path). Like layer mask, vector mask is only applied to layers not to backgrounds. So, now if you transform the image to higher sizes, you won’t get jagged edge. It will maintain sharped and fine edge of the selected area of image. Main advantage of using vector mask is: it prints at full resolution of the printer and is resolution–independent like all vector objects.

Here is an example showing comparison between layer mask and vector mask output:

Image cutout with layer mask and vector mask in actual size:

Image transformed to 300%, output of the image of selected part: