What you have achieved via masking is working correctly. Masking works on alpha channel, transparent area vs visible area and images you posted in your question show exactly that.
Masking won't be enough for the effect you want to achieve. You need to experiment with CIFilter
api.
Check out cifilter.io that shows a few examples of how these are supposed to look like. You might want to check out the source code for the same.
If you run this iOS project in Xcode, it provides a very easy way to browse and choose which effect is the closest to what you want to achieve.
Good luck!