0

I try to use the tokenizer method to tokenize the sentence and then mean pool the attention mask to get the vectors for each sentence. However, the current default size embedding is 768 and I wish to reduce it to 200 instead but failed. below is my code.

from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/bert-base-nli-mean-tokens')
model = AutoModel.from_pretrained('sentence-transformers/bert-base-nli-mean-tokens')
model.resize_token_embeddings(200)
# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, max pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

Error:

   2193     # Note [embedding_renorm set_grad_enabled]
   2194     # XXX: equivalent to
   2195     # with torch.no_grad():
   2196     #   torch.embedding_renorm_
   2197     # remove once script supports set_grad_enabled
   2198     _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2199 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

IndexError: index out of range in self

my expected output is:

when use:

print(len(sentence_embeddings[0]))
-> 200
Progman
  • 16,827
  • 6
  • 33
  • 48
Learner91
  • 103
  • 6

1 Answers1

0

I think you've misunderstood the resize_token_embeddings. According to docs It

Resizes input token embeddings matrix of the model if new_num_tokens != >config.vocab_size.

Takes care of tying weights embeddings afterwards if the model class has a >tie_weights() method.

meaning it is used when you add/remove tokens from vocabulary. Here resizing refers to resizing the token->embedding dictionary.

I guess what you want to do is change the hidden_size of bert model. In order to do that you have to change the hidden_size in config.json which will re-initialize all weights and you have to re-train everything i.e very computationally expensive.

I think your best option is to add a linear layer on top of BertModel of dimension (768x200) and fine-tune on your downstream task.

Aman Chauhan
  • 66
  • 2