5

x is a (64, 1) dimensional vector created randomly using tf.random.uniform((BATCH_SIZE, 1)), where the BATCH_SIZE = 64.

A random initialization looks like this:

tf.Tensor(
[[0.76922464]
 [0.7928164 ]
 [0.91224647]
 [0.41210544]
 [0.33040464]
 [0.20977008]
 [0.96211743]
 [0.59516513]
 [0.67317   ]
 [0.7600033 ]
 [0.93105805]
 [0.55348516]
 [0.50683343]
 [0.7563635 ]
 [0.06255531]
 [0.93398154]
 [0.5622641 ]
 [0.9913852 ]
 [0.3019762 ]
 [0.519048  ]
 [0.57998526]
 [0.21162748]
 [0.9783536 ]
 [0.38307965]
 [0.6527189 ]
 [0.8094288 ]
 [0.97980523]
 [0.5955998 ]
 [0.7002481 ]
 [0.6879872 ]
 [0.50365186]
 [0.57166266]
 [0.97805905]
 [0.458856  ]
 [0.3485204 ]
 [0.29394794]
 [0.19313121]
 [0.29782188]
 [0.45194447]
 [0.49442303]
 [0.04192603]
 [0.26818407]
 [0.822567  ]
 [0.8573874 ]
 [0.15510845]
 [0.76052403]
 [0.4066763 ]
 [0.17861617]
 [0.458804  ]
 [0.25463438]
 [0.89405084]
 [0.854866  ]
 [0.9855745 ]
 [0.04673469]
 [0.6193329 ]
 [0.9060414 ]
 [0.17602026]
 [0.20119262]
 [0.08522642]
 [0.7849103 ]
 [0.34081244]
 [0.2556857 ]
 [0.75679326]
 [0.635311  ]], shape=(64, 1), dtype=float32)

The embedding layer is defined as self.embedding = tf.keras.layers.Embedding(4934, 256)

x, created above, is passed through this embedding layer as follows:

x = self.embedding(x)

x resulting from this embedding has dimensions (64, 1, 256). So each of the 64 float values in x has a 256 dimensional vector representation.

My question is: x was initially a randomly generated float vector with each vector of length 1.

By definition, I understand the embedding layer as mapping from a word to an index and the index has a vector representation of length equal to "embedding dimensions", 256 in this example. So the word that mapped to the index also has this same vector representation.

But x in our example is just a vector of random float values. How did the embedding layer come up with a 256-dimensional vector representation for these float values? Any float value in this list does not represent a word. Why should it have an embedding?

It's line 36 in the image included below (link to code page: Google colab code location

Code Image

today
  • 32,602
  • 8
  • 95
  • 115
Utpal Mattoo
  • 890
  • 3
  • 17
  • 41

1 Answers1

1

Passing float values to an Embedding layer does not raise an error because the layer implementation is such that the input is automatically cast to integer (if it's not an integer). You can confirm this is the case by looking at the relevant section in source code:

def call(self, inputs):
    dtype = K.dtype(inputs)
    if dtype != 'int32' and dtype != 'int64':
      inputs = math_ops.cast(inputs, 'int32')
today
  • 32,602
  • 8
  • 95
  • 115
  • That makes sense but, the resulting integer values are all zero `import tensorflow as tf from tensorflow.python.ops import math_ops inputs = tf.random.uniform((64, 1)) x = math_ops.cast(inputs, 'int32')`. I think you answered my original question, but I am confused why it makes sense to represent a `64-D` vector of `0's` with a `256-dimensional` embeddings – Utpal Mattoo Sep 22 '20 at 19:58
  • @UtpalMattoo Nothing very special is happening there: it just takes the embedding associated with the zero index and returns it as output. Since then input has a shape of `(64, 1)` (which means **a batch of 64 samples** where each sample is a 1D vector), the output would be of shape `(64, 1, emb_size)`. On the other hand, if the input has a shape of `(1, 64)` (which means **a batch of 1 sample** where the sample is a 64D vector), the output would be of shape `(1, 64, emb_size)`. – today Sep 23 '20 at 08:08