5

I have computed the vectors for two same sentences using XLNet embedding-as-service. But the model produces different vector embeddings for both the two same sentences hence the cosine similarity is not 1 and the Euclidean distances also not 0. in case of BERT its works fine. for example; if

vec1 = en.encode(texts=['he is anger'],pooling='reduce_mean')
vec2 = en.encode(texts=['he is anger'],pooling='reduce_mean')

the model (XLNet) is saying that these two sentences are dissimilar.

pppery
  • 3,731
  • 22
  • 33
  • 46
Anoop kottappuram
  • 184
  • 1
  • 5

2 Answers2

1

This is because of to the dropout layers in the model. During inference, the dropout layers should be turned off but there is a bug in the library. It is discussed here and apparently still not fixed.

See the discussion here: https://github.com/amansrivastava17/embedding-as-service/issues/45

In the mean time as suggested by @Davide Fiocco, you can use the straightforward approaches from HuggingFace. Either use forward, generate or pipeline.

Berkay Berabi
  • 1,933
  • 1
  • 10
  • 26
0

As workaround, if you have some flexibility, what about using the vanilla transformers library instead?

Results from

from transformers import pipeline
embedder = pipeline("feature-extraction", model="xlnet-base-cased")
embedder("he is anger")

are deterministic.

Davide Fiocco
  • 5,350
  • 5
  • 35
  • 72