Stack Overflow
Stack Overflow

Questions tagged [language-model]

266 questions
1
vote
1 answer

using huggingface's pytorch- transformers GPT-2 for classifcation tasks

I want to use GPT-2 to make a text classifier model. I am not really sure what head should I add after I extracted features through the GPT-2. for eample I have a sequence. import pytorch_transformers as pt import torch text=test.iloc[1,1] text 'If…
Tommy Yu
  • 1,080
  • 3
  • 11
  • 30
1
vote
1 answer

Access spaCy Masked Language Model

As of v2.1, spaCy has a BERT-style language model (LM). It predicts word-vectors instead of words, so I am going to use "words" and "word vectors" interchangeably here. I need to take a sentence with a word masked, and a list of words, and rank the…
Sam H.
  • 4,091
  • 3
  • 26
  • 34
1
vote
1 answer

Use BERT for feature extraction of a unique word

I am using BERT for feature extraction of a word given the text where it appears, but it seems current implementation in bert's official github (https://github.com/google-research/bert) can only compute the features of all the words in text, which…
Rodrigo Serna Pérez
  • 681
  • 7
  • 16
1
vote
3 answers

How to create window/chunk for list of sentences?

I have list of sentence and I want to create skipgram (window size = 3) but I DONT want the counter to span across sentences since they are all unrelated. So, if I have the sentences: [["my name is John"] , ["This PC is black"]] the triplets will…
oren_isp
  • 729
  • 1
  • 7
  • 22
1
vote
1 answer

Check perplexity of a Language Model

I created a language model with Keras LSTM and now I want to assess wether it's good so I want to calculate perplexity. What is the best way to calc perplexity of a model in Python?
Cranjis
  • 1,590
  • 8
  • 31
  • 64
1
vote
0 answers

One hot encoding of 1 million category

For a language model, I have to predict a word for a given sequence of words. My vocabulary contains 1 million words. I'm trying to predict the words from it. I tried to use one hot encoding using keras (to_categorical) for predicted words. But for…
xoxis
  • 11
  • 1
  • 6
1
vote
1 answer

Why "add one smoothing" in language model does not count the in denominator

English is not my native language , Sorry for any grammatical mistakes. I saw many documents for add one smoothing in language model, and I still very confused about the variable V in the formula: P (wi |w_i-1 ) = c(w_i-1 ,wi )+1 / c(w_i-1 )+V as…
Jeffese
  • 11
  • 2
1
vote
1 answer

Sphinx 4 corrupted ARPA LM?

I have an ARPA LM generated by kylm, when running SPHINX I get this exception stack trace: Exception in thread "main" java.lang.RuntimeException: Allocation of search manager resources failed at…
firas
  • 1,463
  • 4
  • 19
  • 42
1
vote
1 answer

NLP - What to do when unigram is not present in corpus while doing stupid backoff smoothing

In stupid backoff for smoothing for trigrams, if trigram is not found then we backoff to bigram , if bigram is also not found we backoff to unigram. But what if unigram is not present in the corpus. In the paper under stupid backoff section it is…
Trevor Philips
  • 13
  • 4
1
vote
1 answer

language modeling in tensorflow - how to tie embedding and softmax weights

As suggested by recent language modeling papers, I want to use weight tying in my RNN language model. That is, I want to share the weights between the embedding and softmax layer. However, I am not sure how this can be done in TensorFlow. My network…
Lemon
  • 1,394
  • 3
  • 14
  • 24
1
vote
1 answer

What is the input to an RNN language model (TensorFlow)?

I want to build a recurrent neural network (RNN) in TensorFlow that predicts the next word in a sequence of words. I have looked at several tutorials, e.g. the one of TensorFlow. I know that each word in the training text(s) is mapped to an integer…
Lemon
  • 1,394
  • 3
  • 14
  • 24
1
vote
1 answer

Getting probability of the text given word embedding model in gensim word2vec model

I am trying to get most probable sequence of word using gensim word2vec model. I have found a pretrained model which provides these files: word2vec.bin word2vec.bin.syn0.npy word2vec.bin.syn1neg.npy This is my code trying to get the probability of…
ida
  • 1,011
  • 1
  • 9
  • 17
1
vote
0 answers

Tensorflow RNN: Perplexity per Epoch remains constant

I am training an RNN-based language-model using Tensorflow. The model is very similar to the PTB model example in the TF tutorials section. However, when I attempt to train the model on my own data, the perplexity of the model does not go down; it…
user227837
  • 129
  • 1
  • 10
1
vote
1 answer

Can Artificial Neural Networks Learn Language Models? Paper 2000 Implementation

I am new to research field in NLP. I want to implement a paper Can Artificial Neural Networks Learn Language Models? In this paper first time a step was taken so that Neural Network can learn Language Model. I have understood the paper, everything…
Hammad Hassan
  • 1,192
  • 17
  • 29
1
vote
1 answer

N-grams - not in memory

I have 3 milion abstracts and I would like to extract 4-grams from them. I want to build a language model so I need to find the frequencies of these 4-grams. My problem is that I can't extract all these 4-grams in memory. How can I implement a…
Dimitris Dimitriadis
  • 163
  • 1
  • 9
1 2 3
…
17 18