Implementing Custom Min_MAX_Pooling Layer in Tensorflow

Question

Hi i am trying to implement coustom min max plooing layer in tensorflow using lambda layers to reduce noise in time series data. Here is the function that dose the min max pooling

def min_max_pooling(sequence, window=5):

    output = tf.constant([],dtype='float64')
    max_ = tf.Variable(0,dtype = 'float64')
    min_ = tf.Variable(0,dtype = 'float64')

    # loop over sequence in chunks, get the min max values and concat all of them into single 
    tensor and return as output.

    for i in range(window, len(sequence) + window, window):
        chunk = sequence[i - window:i]
        print(i)
        
        # get the max and min values from chunk

        max_.assign(chunk[tf.argmax(chunk)])
        min_.assign(chunk[tf.argmin(chunk)])

        # get the index of max and min values from chunk

        max_index = tf.argmax(chunk)
        min_index = tf.argmin(chunk)

        
        # append values to output tensor according to the original sequence
        # if min was first in sequence than max i,e.  tf.greater(max_index , min_index) == True,  
        # append min first and then max else vice versa

        if tf.greater(max_index , min_index):
            output = tf.concat([output, [min_]],-1)
            output = tf.concat([output, [max_]],-1) 

        else:
            output = tf.concat([output, [max_]],-1)
            output = tf.concat([output, [min_]],-1)

    return tf.convert_to_tensor(output)

# print(tf.autograph.to_code(min_max_pooling))

# min_max_pooling = tf.autograph.to_graph(min_max_pooling)

The function accepts two arguments a 1d tensor of time series data(scaled between 0 to 1) and window size. It than calculates output sequence w.r.t to windows size and returns a tensor. Basicly its a function that works just like maxpooling1d which helps to reduce noise(downsample's data) but also accounts for min values which is the reason why i want to implement it. Here is test output of this function.

tf.Tensor( [0.99941323 0.98313041 0.97799619 0.98533079 0.98635764 0.99457239 0.99413232 0.99105178 0.99193193 0.98753117 0.98489071 0.98371718 0.98459733 0.98445064 0.98386387 0.98547748 0.99163855 0.99061171 0.99735954 1. ], shape=(20,), dtype=float64)

[![enter image description here][1]][1]

after minmaxpooling

[![enter image description here][2]][2]

Now the problem arises when i use it as a lambda layer in tensorflow models i get all kinds of error tried to solve all of them but still cant figure out the problem. I cant get it working with tensorflow.here is the code for that.\

input_layer = tf.keras.layers.Input(shape=(1000,), name="input_layer")

output_layer = tf.keras.layers.Lambda(min_max_pooling, name="lambda_layer")(input_layer)

model = tf.keras.models.Model(input_layer, output_layer, name="model")

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005), loss="categorical_crossentropy", run_eagerly=True)

All i want to build is a layer that accepts a sequence of time series data and removes the noise from it just like maxpooling1d except also taking into account the significance of min values in sequence.

The end result should be like passing a tensor to the layer

tf.Tensor(
[0.99941323 0.98313041 0.97799619 0.98533079 0.98635764 0.99457239
 0.99413232 0.99105178 0.99193193 0.98753117 0.98489071 0.98371718
 0.98459733 0.98445064 0.98386387 0.98547748 0.99163855 0.99061171
 0.99735954 1.        ], shape=(20,), dtype=float64)

and getting the downsampled output as

tf.Tensor(
[0.99941323 0.97799619 0.99457239 0.98753117 0.98489071 0.98371718
 0.98547748 1.        ], shape=(8,), dtype=float64)

Now i know that i dont know a lot about tensorflow. But please can anyone help me implement the full working code for the problem.? And also dont know how to pass the window paramater to lambda layer? want help with that to. Any kind of help is appreciated.

ValueError: Exception encountered when calling layer "lambda_layer" (type Lambda).

New error

The following Variables were created within a Lambda layer (lambda_layer)
but are not tracked by said layer:
  <tf.Variable 'lambda_layer/map/while/Variable:0' shape=() dtype=float32>
  <tf.Variable 'lambda_layer/map/while/Variable:0' shape=() dtype=float32>
The layer cannot safely ensure proper Variable reuse across multiple
calls, and consequently this behavior is disallowed for safety. Lambda
layers are not well suited to stateful computation; instead, writing a
subclassed Layer is the recommend way to define layers with
Variables.

2nd Error

data = pd.read_csv('/content/Stock_data.csv', parse_dates=False,
                   index_col=1)

tensor = data.close.head(10).to_numpy(dtype='float64')
tensor = tensor / max(tensor)
print(tensor)
# tensor = tf.convert_to_tensor(tensor)

print(tensor)
p = model.predict(tensor)
print(p)

error:

[1.         0.98370762 0.97857038 0.98590929 0.98693674 0.99515632
 0.99471598 0.99163364 0.99251431 0.98811096]
[1.         0.98370762 0.97857038 0.98590929 0.98693674 0.99515632
 0.99471598 0.99163364 0.99251431 0.98811096]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-59-5028b0f38a02> in <module>()
      8 
      9 print(tensor)
---> 10 p = model.predict(tensor)
     11 print(p)

1 frames
<ipython-input-54-a3aac9d10348> in multiple_min_max_pooling(sequences)
      1 def multiple_min_max_pooling(sequences):
----> 2     return tf.map_fn(min_max_pooling, sequences)

ValueError: Exception encountered when calling layer "lambda_layer" (type Lambda).

in user code:

    File "<ipython-input-53-495c7ee6064b>", line 10, in min_max_pooling  *
        for i in range(window, len(sequence) + window, window):

    ValueError: len requires a non-scalar tensor, got one of shape []


Call arguments received:
  • inputs=tf.Tensor(shape=(10,), dtype=float32)
  • mask=None
  • training=False

it's not useful uploading the same image 3 times, upload it 1 time and explain what is its meaning — Alberto Sinigaglia, Aug 15 '22 at 10:56
on the other hand, posting the _all kinds of error_ and a minimal reproducible example (input-code-output-desired output, as numbers, not as graph), might help — Alberto Sinigaglia, Aug 15 '22 at 11:03

Alberto Sinigaglia · Answer 1 · 2022-08-15T11:31:54.807

Apart from the fact that usually TF uses float32 in my experience, as float 64 is double the memory, and usually the additional precision/big numbers are not useful, your problem is that you are not considering that TF uses batches of data

In other words, your layer will receive a batch of sequences, not a single one. Your code works fine for a single one, thus can be easily (but not efficiently) fixed with a tf.map_fn:

def multiple_min_max_pooling(sequences):
    return tf.map_fn(min_max_pooling, sequences)

...
tf.keras.layers.Lambda(multiple_min_max_pooling, name="lambda_layer")(input_layer)

About the time window, you have 2 choices, either defined a custom layer (more elegant and readable), or define a function that return a function (this "design pattern" is called higher-order functions if you want to read more about it online to understand how it works):

def multiple_min_max_pooling(window=5):
    def fn(sequences):
        return tf.map_fn(min_max_pooling(window), sequences)
    return fn
def min_max_pooling(window=5):
    def fn(sequence):
        output = tf.constant([],dtype='float32')
        max_ = tf.Variable(0,dtype = 'float32')
        min_ = tf.Variable(0,dtype = 'float32')

        # loop over sequence in chunks, get the min max values and concat all of them into single

        for i in range(window, len(sequence) + window, window):
            chunk = sequence[i - window:i]
            print(i)

            # get the max and min values from chunk

            max_.assign(chunk[tf.argmax(chunk)])
            min_.assign(chunk[tf.argmin(chunk)])

            # get the index of max and min values from chunk

            max_index = tf.argmax(chunk)
            min_index = tf.argmin(chunk)


            # append values to output tensor according to the original sequence
            # if min was first in sequence than max i,e.  tf.greater(max_index , min_index) == True,
            # append min first and then max else vice versa

            if tf.greater(max_index , min_index):
                output = tf.concat([output, [min_]],-1)
                output = tf.concat([output, [max_]],-1)

            else:
                output = tf.concat([output, [max_]],-1)
                output = tf.concat([output, [min_]],-1)

        return tf.convert_to_tensor(output)
    return fn

# and now you can do this:
tf.keras.layers.Lambda(multiple_min_max_pooling(window=2), name="lambda_layer")(input_layer)

thanks that worked i change the data types to float32 and implemented map function but i got this new error i put that above.It say to use subclassed Layer. do i need to implement this using Layers class in tensorflow. How to do it — AB Music Box, Aug 15 '22 at 11:43
@ABMusicBox yes, if you use states (variable) you have to use subclassing... my suggestion is to use `tf.constant` as you are doing for `output`, you don't need a trainable variable — Alberto Sinigaglia, Aug 15 '22 at 11:46
but then i wont be able assign value to it in future. How can i initialize max (chunk[tf.argmax(chunk)) and min (chunk[tf.argmin(chunk)) values directly into tf.constant — AB Music Box, Aug 15 '22 at 11:56
i am an idiot. Thank a lot. One last thing after compiling the model when i pass a sequence to model.predict it throws a error ' len requires a non-scalar tensor, got one of shape []' what am i doing wrong here. — AB Music Box, Aug 15 '22 at 12:09
@ABMusicBox the problem is here `len(sequence)` probably... maybe try with `tf.shape(sequence)[0]`, but I have too few info to debug it — Alberto Sinigaglia, Aug 15 '22 at 12:13
@ABMusicBox it's probably because you are passing a single tensor to the layer... the shape of the input should be `tf.shape(tensor) = (1, something)`, your one is probably `(something)` — Alberto Sinigaglia, Aug 15 '22 at 12:21

Implementing Custom Min_MAX_Pooling Layer in Tensorflow

1 Answers1