Attempt at making a Displacement Filter

Question

So, basically, I'm trying to make a OBS Filter that displaces the pixels based on a lightmap/luminance map. I decided to learn how to make a filter by following this tutorial. But, in this tutorial, they don't explain much in terms of pixel displacement. So, I made a function that basically gets the brightness value of a texture I input and tested it by changing the pixel's alpha value with the red value of the texture:

float4 get_displacement(float2 position)
{
    float2 pattern_uv = position / pattern_size;
    float4 pattern_sample = pattern_texture.Sample(linear_wrap, pattern_uv / scale);
    return pattern_sample;
}

float4 pixel_shader(pixel_data pixel) : TARGET
{
    float4 source_sample = image.Sample(linear_wrap, pixel.uv);
    if (pattern_size.x <= 0){
        return source_sample;
    }
    float2 position = pixel.uv * float2(width, height);
    float4 lightmap = get_displacement(position);
    return float4(source_sample.rgb, lightmap.r);
    return source_sample;
}

Which results to this (Note: The green is from a colour source that's behind the image to show the alpha value)

But, for some reason, when I try it with the vertex_shader, the function that decides where the pixel is rendered at, it seems to not work:

pixel_data vertex_shader(vertex_data vertex)
{
    pixel_data pixel;
    pixel.uv  = vertex.uv;
    if (pattern_size.x <= 0){
        pixel.pos = mul(float4(vertex.pos.xyz, 1.0), ViewProj);
        return pixel;
    }
    float2 position = vertex.uv * float2(width, height);
    float4 lightmap = get_displacement(position);
    pixel.pos = mul(float4(vertex.pos.x + (lightmap.r * testRamp1), vertex.pos.yz, 1.0), ViewProj);
    return pixel;
}

(Note: testRamp1 is used as a value that I can change from a slider inside of OBS via some filter Properties)

The result that I'm expecting is something similar to this To see if the issue was from me changing the XY position, I tested it using this function:

pixel_data vertex_shader(vertex_data vertex)
{
    pixel_data pixel;
    pixel.uv  = vertex.uv;
    pixel.pos = mul(float4(vertex.pos.x + 100, vertex.pos.yz, 1.0), ViewProj);
    return pixel;
}

And it gave me an expected result. I also changed the 100 with the testRamp1 value, and it works just the same based on the value of the slider.

So, I then tested if it was from the pixels needing to all move the same distance as each other. So, I change the function to this:

pixel_data vertex_shader(vertex_data vertex)
{
    pixel_data pixel;
    pixel.uv  = vertex.uv;
    pixel.pos = mul(float4(vertex.pos.x + (vertex.uv.x * testRamp1), vertex.pos.yz, 1.0), ViewProj);
    return pixel;
}

Which then gives me either a squashed image when testRamp1 is set to a negative value, and it gives me a stretched image when it's set it to a positive value.

But as soon as I try to get the value of an image, may it be the pattern or from the source image, it no longer works(not even the filter parameters appear). For example, I used, this function to use the values of the source image:

pixel_data vertex_shader(vertex_data vertex)
{
    pixel_data pixel;
    float4 source_sample = image.Sample(linear_wrap, vertex.uv);
    pixel.uv  = vertex.uv;
    pixel.pos = mul(float4(vertex.pos.x + (source_sample.r * testRamp1), vertex.pos.yz, 1.0), ViewProj);
    return pixel;
}

At this point, I'm at a loss of words as to what could be causing this issue

Answer 1

First of all, the vertex shader is not what you want to use for this kind of effect. What you actually want to do, is to sample the image in the pixel shader, but offset the UV values slightly by your displacement before you pass them to the Sample function.

The primary reason, why you don't want to do this in the vertex shader, is, that the number of vertices is usually much smaller than the number of pixels - in the worst case, you only have 4 vertices in total (one for each corner of your screen), so the granuality of things you can do in the vertex shader is rather coarse. (Note: I'm not too familiar with OBS filters, and don't know how many vertices OBS dispatches, but certainly much less than the number of pixels you have on your screen).

Now, the reason why your vertex shader didn't work at all, is a bit more technical. In short, you can't use Sample in a vertex shader, you'd have to use SampleLevel or SampleGrad instead (note that these functions require more parameters). This is because Sample automatically calculates a UV gradient between adjacent pixels, to figure out the level of detail that is needed for your texture (whether or not it actually has multiple levels of detail). But the vertex shader operates on vertices, not on pixels, so the concept of an "adjacent pixel" doesn't make sense in a vertex shader - thus, the Sample method doesn't work.