How can I achieve this maskin effect?

Unity Engine
1

Suppose I have this UI image of a yellow frame:


Now I want to mask it (masked area is the red rectangle)

So it end up like this:

2

did you mean inverted mask?
unfortunately, Unity UI mask currently can’t do that, cmiiw

instead here is the result if you use Unity UI mask

there’s a feedback for this feature
http://feedback.unity3d.com/suggestions/ui-beta-add-ability-to-mask-with-smooth-alpha-and-invert-the-mask

3

Yeah, basically an inverted mask.