1

There are dozens of image filters written for the Android version of our app in GLSL (ES). As of iOS 12, OpenGL is deprecated, and CIFilter kernels have to be written in Metal.

I had some previous background in OpenGL, however writing CIFilter kernels in Metal is new to me.

Here is one of the filters. Could you help me in translating it to Metal as a CIFilter kernel? That would provide a good example for me so I could translate others.

#extension GL_OES_EGL_image_external : require
precision mediump float;
varying vec2 vTextureCoord;
uniform samplerExternalOES sTexture;
uniform float texelWidth;
uniform float texelHeight;
uniform float intensivity;
void main() {
    float SIZE = 1.25 + (intensivity / 100.0)*2.0;
    vec4 color;
    float min = 1.0;
    float max = 0.0;
    float val = 0.0;
    for (float x = -SIZE; x < SIZE; x++) {
        for (float y = -SIZE; y < SIZE; y++) {
            color = texture2D(sTexture, vTextureCoord + vec2(x * texelWidth, y * texelHeight));
            val = (color.r + color.g + color.b) / 3.;
            if (val > max) { max = val; } else if (val < min) { min = val; }
        }
    }
    float range = 5. * (max - min);
    gl_FragColor = vec4(pow(1. - range, SIZE * 1.5));
    gl_FragColor = vec4((gl_FragColor.r + gl_FragColor.g + gl_FragColor.b) / 3. > 0.75 ? vec3(1.) : gl_FragColor.rgb, 1.);
}
David
  • 125
  • 2
  • 15
  • Here are some resources on the Metal shading language: https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf hope that helps – Ian Rehwinkel Jan 28 '19 at 14:24
  • Can you provide an example input and output image and the expected range of the intensity parameter? It's not immediately obvious from inspection what this kernel is supposed to do. – warrenm Jan 31 '19 at 19:50
  • @warrenm Hello, Warren! This kernel produces a sketched version of the input image. The intensity parameter range is (0,100). Here is an example: original - https://imgur.com/SlARGMy, p=100 -> https://imgur.com/IqRw4A1, p=50 -> https://imgur.com/Lygjaqs, p=0 -> https://imgur.com/oypgOo0 – David Jan 31 '19 at 22:20

1 Answers1

4

Here's the Metal source for a kernel that attempts to replicate your described filter:

#include <metal_stdlib>
#include <CoreImage/CoreImage.h>

using namespace metal;

extern "C" {
namespace coreimage {

float4 sketch(sampler src, float texelWidth, float texelHeight, float intensity40) {
    float size = 1.25f + (intensity40 / 100.0f) * 2.0f;

    float minVal = 1.0f;
    float maxVal = 0.0f;
    for (float x = -size; x < size; ++x) {
        for (float y = -size; y < size; ++y) {
            float4 color = src.sample(src.coord() + float2(x * texelWidth, y * texelHeight));
            float val = (color.r + color.g + color.b) / 3.0f;
            if (val > maxVal) {
                maxVal = val;
            } else if (val < minVal) {
                minVal = val;
            }
        }
    }

    float range = 5.0f * (maxVal - minVal);

    float4 outColor(pow(1.0f - range, size * 1.5f));
    outColor = float4((outColor.r + outColor.g + outColor.b) / 3.0f > 0.75f ? float3(1.0f) : outColor.rgb, 1.0f);
    return outColor;
}

}
}

I assume you're already familiar with the basics of how to correctly build Metal shaders into a library that can be loaded by Core Image.

You can instantiate your kernel at runtime by loading the default Metal library and requesting the "sketch" function (the name is arbitrary, so long as it matches the kernel source):

NSURL *libraryURL = [NSBundle.mainBundle URLForResource:@"default" withExtension:@"metallib"];
NSData *libraryData = [NSData dataWithContentsOfURL:libraryURL];

NSError *error;
CIKernel *kernel = [CIKernel kernelWithFunctionName:@"sketch" fromMetalLibraryData:libraryData error:&error];

You can then apply this kernel to an image by wrapping it in your own CIFilter subclass, or just invoke it directly:

CIImage *outputImage = [kernel applyWithExtent:CGRectMake(0, 0, width, height)
                                   roiCallback:^CGRect(int index, CGRect destRect)
                        { return destRect; }
                                     arguments:@[inputImage, @(1.0f/width), @(1.0f/height), @(60.0f)]];

I've tried to select sensible defaults for each of the arguments (the first of which should be an instance of CIImage), but of course these can be adjusted to taste.

warrenm
  • 31,094
  • 6
  • 92
  • 116
  • Thank you a lot, Warren – David Feb 01 '19 at 13:37
  • Happy to help! Thanks for the bounty. – warrenm Feb 01 '19 at 17:29
  • Is it possible to wrap this inside CIFilter (backed by Metal)? Or it will cause a performance issues? – Roi Mulia Jul 18 '19 at 10:18
  • @RoiMulia I would expect that to work just fine. Feel free to post another question in the [metal] tag if you try it and run into difficulty. – warrenm Jul 18 '19 at 16:21
  • Hey @warrenm ! Thank you for your reply. I've created a new question regards my issue: https://stackoverflow.com/questions/57121127/convert-opengl-shader-to-metal-swift-to-be-used-in-cifilter . Would love if you could take a look, I tried to make the question as informative as possible. Thank you! – Roi Mulia Jul 20 '19 at 01:33
  • Hey Waren, just encountered new issues regards memory consumption on our advanced CIFilter backed by Metal. Seems like when we are using 4K videos - it's super hard for the device, and it's crashing. The thing is that the final output our app produce is not more than 1080p than we are doing extra work for nothing. If you have a spare min, I'd love if you could take a look and guide me in the right direction (as always ). Here's a link: https://stackoverflow.com/questions/57153640/metal-resize-video-buffer-before-passing-to-custom-kernel-filter . Thank you so much, no words! – Roi Mulia Jul 22 '19 at 21:09