4

I need to implement salt & pepper layer in keras like Gaussian noise, I tried to use the following code but it produces several errors. could you please tell me what is the problem? do you have any other suggestion for implementing S&P layer? Thank you.

from keras.engine.topology import Layer

class SaltAndPepper(Layer):

    def __init__(self, ratio, **kwargs):
        super(SaltAndPepper, self).__init__(**kwargs)
        self.supports_masking = True
        self.ratio = ratio

    def call(self, inputs, training=None):
        def noised():
            r = self.ratio*10
            s = inputs.shape[1]
            n = int( s * r/10 )
            perm = np.random.permutation(r)[:n]
            inputs[perm] = (np.random.rand(n) > 0.5)
            return inputs

        return K.in_train_phase(noised(), inputs, training=training)

    def get_config(self):
        config = {'ratio': self.ratio}
        base_config = super(SaltAndPepper, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

Traceback (most recent call last):

File "", line 125, in decoded_noise=SaltAndPepper(0.5)(decoded)

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\engine\base_layer.py", line 457, in call output = self.call(inputs, **kwargs)

File "", line 57, in call return K.in_train_phase(noised(), inputs, training=training)

File "", line 52, in noised n = int( s * r/10 )

TypeError: unsupported operand type(s) for /: 'Dimension' and 'int'

Update:

I used @today's solution and wrote the following code:

decoded_noise=call(0.05,bncv11)#16

which bncv11 is the output of batch normalization layer before it.

but it produces this error, why does it happen?

Traceback (most recent call last):

File "", line 59, in decoded_noise=call(0.05,bncv11)#16

File "", line 34, in call return K.in_train_phase(noised(), inputs, training=training)

File "", line 29, in noised mask_select = K.random_binomial(shape=shp, p=self.ratio)

AttributeError: 'float' object has no attribute 'ratio'

after saving the model and used it produces this error:

Traceback (most recent call last):

File "", line 1, in b=load_model('Desktop/los4x4_con_tile_convolw_FBN_SigAct_SandPAttack05.h5',custom_objects={'tf':tf})

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\engine\saving.py", line 419, in load_model model = _deserialize_model(f, custom_objects, compile)

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\engine\saving.py", line 225, in _deserialize_model model = model_from_config(model_config, custom_objects=custom_objects)

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\engine\saving.py", line 458, in model_from_config return deserialize(config, custom_objects=custom_objects)

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\layers__init__.py", line 55, in deserialize printable_module_name='layer')

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\utils\generic_utils.py", line 145, in deserialize_keras_object list(custom_objects.items())))

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\engine\network.py", line 1022, in from_config process_layer(layer_data)

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\engine\network.py", line 1008, in process_layer custom_objects=custom_objects)

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\layers__init__.py", line 55, in deserialize printable_module_name='layer')

File "D:\software\Anaconda3\envs\py36\lib\site-packages\keras\utils\generic_utils.py", line 138, in deserialize_keras_object ': ' + class_name)

ValueError: Unknown layer: SaltAndPepper

I put this code in my program where I define my network structure:

from keras.engine.topology import Layer

class SaltAndPepper(Layer):

    def __init__(self, ratio, **kwargs):
        super(SaltAndPepper, self).__init__(**kwargs)
        self.supports_masking = True
        self.ratio = ratio

    # the definition of the call method of custom layer
    def call(self, inputs, training=True):
        def noised():
            shp = K.shape(inputs)[1:]
            mask_select = K.random_binomial(shape=shp, p=self.ratio)
            mask_noise = K.random_binomial(shape=shp, p=0.5) # salt and pepper have the same chance
            out = inputs * (1-mask_select) + mask_noise * mask_select
            return out

        return K.in_train_phase(noised(), inputs, training=training)

    def get_config(self):
        config = {'ratio': self.ratio}
        base_config = super(SaltAndPepper, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))
david
  • 1,255
  • 4
  • 13
  • 26
  • The stacktrace say that your ratio is a Dimension object and not a number variable. – Nakeuh Apr 12 '19 at 14:35
  • what does it mean? because I am a beginner and I do not know what this code does. what should I solve it? – david Apr 12 '19 at 14:38
  • is it possible to implement s&p noise with lambda layer or we should produce a layer like this? – david Apr 12 '19 at 14:39
  • To be honest I don't know much about s&p layers. But the error you have is a pretty basic python error. In your code, you try to do the following division `r/10`, and the error message tells you that it's not possible since the `r` variable is not a number. The `r` value come from the ratio (the second parameter you use when creating your Layer. It should be a number, and I suppose that it is not. Can you provide the code you use to create your Layer ? – Nakeuh Apr 12 '19 at 14:46

1 Answers1

2

In image processing, salt and pepper noise basically changes the value of a ratio of pixels, which are selected randomly, to either salt (i.e. white, which is usually 1 or 255 depending on the range of image values) or pepper (i.e. black which is usually 0). Although, we can use the same idea in other domains besides image processing. So you must specify three things first:

  1. How many of the pixels should be changed? (noise ratio)
  2. Which pixels should be selected and changed?
  3. Which of those selected pixels should be salted (and the other peppered)?

Since there is a function in Keras backend to generate random values from a binomial distribution (i.e. 0 or 1) with a given probability, we can easily do all the above steps by generating two masks: one for selecting the pixels with the given ratio, and the other one for applying either of salt or pepper to those selected pixels. Here is how to do it:

from keras import backend as K

# NOTE: this is the definition of the call method of custom layer class (i.e. SaltAndPepper)
def call(self, inputs, training=None):
    def noised():
        shp = K.shape(inputs)[1:]
        mask_select = K.random_binomial(shape=shp, p=self.ratio)
        mask_noise = K.random_binomial(shape=shp, p=0.5) # salt and pepper have the same chance
        out = inputs * (1-mask_select) + mask_noise * mask_select
        return out

    return K.in_train_phase(noised(), inputs, training=training)

Note in the code above I have assumed a few things:

  • The value of the given noise ratio is in the range [0,1].
  • If you are using this layer to directly apply it on images, then you must note that it assumes that the images are grayscale (i.e. single color channel). To use it for RGB images (i.e. three color channels) you may need to modify it a little such that a pixel with all of its channels are selected for adding the noise (although, this depends on how you define and use salt and pepper noise).

  • It assumes the value of salt is 1 and the value of pepper is 0. Though, you can easily change the value of salt to x and pepper to y by changing the definition of mask_noise as follow:

    mask_noise = K.random_binomial(shape=shp, p=0.5) * (x-y) + y

  • The same noise pattern is applied on the all the samples in a batch (however, it would be different from batch to batch).

today
  • 32,602
  • 8
  • 95
  • 115
  • Thank you. how do I use it as a layer in my code? I used this decoded_noise=call(0.05,bncv11) but it produces this error Traceback (most recent call last): File "", line 59, in decoded_noise=call(0.05,bncv11)#16 File "", line 34, in call return K.in_train_phase(noised(), inputs, training=training) File "", line 29, in noised mask_select = K.random_binomial(shape=shp, p=self.ratio) AttributeError: 'float' object has no attribute 'ratio' – david Apr 12 '19 at 18:57
  • @david Please don't edit someone else's answer to add more information about your question. Either use comment section, or if your update is large and contains codes then **edit your own question** and add the updated information at the end of it. Then **leave a comment** under that person's answer to notify him/her of your update. – today Apr 12 '19 at 19:05
  • @david No, that's the `call` method of the layer you have defined. I did not include the full code because the rest was the same. So don't use `call` directly. Put it in your custom layer, and then use your custom layer (as you original did). – today Apr 12 '19 at 19:06
  • sorry, I find out my mistake. Thank you very much. this layer applies S&P during training and during the test, it does not work, right? is it possible to work when I test too or not? – david Apr 12 '19 at 19:09
  • @david I thought since you had originally included `K.in_train_phase` you did not want to apply the layer in test phase. However. if you want it to be applied in both training and test, then simply use `return noised()` instead. – today Apr 12 '19 at 19:13
  • sorry, this means I should return noised() instead of K.in_train_phase(noised(), inputs, training=training), right? sorry for my questions, because I am really a beginner in keras:( – david Apr 12 '19 at 19:19
  • @david Yes, that's right. Or alternatively don't change your code, and set `training` argument to `True` when creating the layer: `sp = SaltAndPepper(0.5, training=True)(inputs)`. This way it would be applied in both training and test phases. – today Apr 12 '19 at 19:22
  • I deeply appreciate your help;) – david Apr 12 '19 at 19:23
  • if I want to produce Gaussian blurring, what should I do? I can use the same code and just change the call part? – david Apr 12 '19 at 19:51
  • @david Yeah, the `call` method is actually where the logic (i.e. computations) of the layer resides. – today Apr 12 '19 at 20:00
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/191764/discussion-between-david-and-today). – david Apr 12 '19 at 20:02
  • Thank you. I know to do Gaussian blurring in Matlab, but in Keras I do not know what should I do? could you please guide me? I asked this question here: https://stackoverflow.com/questions/55643675/how-do-i-implement-gaussian-blurring-layer-in-keras – david Apr 12 '19 at 20:06
  • I put the SaltAndPepper code in above of my network and train it. but when I save the model and then I want to use the trained model with test data it produces this error Unknown layer: SaltAndPepper. I put the complete error and code above. why has this error happened? – david Apr 17 '19 at 13:40
  • 1
    @david See [this answer](https://stackoverflow.com/a/52845882/2099607). You need to pass `{'SaltAndPepper': SaltAndPepper}` as `custom_objects` in `load_model`. Further, I want to correct something I previously mentioned: in order to make a layer (which is not normally applied in test phase) to be applied in both training and test phases, you need to pass `training=True` argument to the **call** of that layer (I previously mentioned it should be passed to the layer itself; that's wrong). See [this answer](https://stackoverflow.com/a/55714910/2099607) for an example. – today Apr 17 '19 at 17:21
  • yes, I did sp = SaltAndPepper(0.5, training=True)(inputs) before and it produced an error. Hence, I chose your first solution and change my code, but this one is better (sp = SaltAndPepper(0.5 )(inputs,training=True)). I appreciate your complete explanations. – david Apr 18 '19 at 01:26