First of all, check out the demo projects. especially demo No3_glow and No4_doha are of interest.
First Masks: See also No3_glow.
Masks need to be inside the subfolder called "masks" and need to be transparent PNG's. Idealy masks have the same resolution and ratio as your input source has, ie. If you use FullHD Movies the PNG should have a resolution of 1920 x 1080. If this is not the case the masking will be of lesser quality.
Second Softedges: See also No4_doha
Softedges are a vast subject in themself and I plan to make another video tutorial on it, but I havent gotten much feedback with this software, so I will wait until some of you whish to have more indepth infos.
Here a short version:
There are two possibilities. Either you use a standard SoftEdge (SE), for this you got to the tab "softedges" and switch on the first SE_modul. Inside the input-source window you should have now a red square, which you can postion with the help of x-pos, y-pos, x-scale and y-scale.
Now you switch over to the first texturetab and select your input source. here switch on the SE-module and select the first SE. Now select a Gradient and a direction. If you have already a mesh on the outputwindow, you should have a softedge somewhere on the texture. you can ajust the position inside the SE-tab until it fits. Then you select the second texture tab and repeat the process, but select a different direction.
The nice thing about this approach is, you only need to set the SE once.
If you wanna use customized SE, you need to import them as obj-wavefront files. Have a look at the one used for doha, its a simple flat obj file which you should be able to import into your favorite 3d modelling package. the file needs to be inside the subfolder called "softedges".
once you put your file into this folder, you can switch on the subsection of the SE-Moodule under the SE-Tab. selected your file and press "editor" right under the SE-tab. Now select the softdege you wanna edit (probably sofedge01) and the reference mesh to help you position the SE.
Currently there is a usability bug in this part of the App, you might need to close the SEdgeEditor and repeat above steps.
Now you have a similar editor as the MeshWarpEditor, except that you apply the mesh onto the right position on the input texture. the texture-reference is actually the uv-map of the projected mesh and should help you to position the SE-vertices as precise as possible.