Questions tagged [language-model]
266 questions
0
votes
0 answers
Relation between perplexity and number of training samples
I'm trying to calculate the perplexity of some English language texts using NLTK. I'm trying to figure out how a simple n-gram model will perform with less training samples. The thing I don't understand is why does perplexity get lower if I decrease…
mrpostman889
- 1
- 1
0
votes
0 answers
Getting output probabilities from logits in T0
I am trying to do prompting using T0. I want the probability scores of the query outputs. For example, for the given prompt: Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy, I wanted the following…
Aditya Joshi
- 1
- 4
0
votes
0 answers
why input_ids of huggingface bert is different in every batch size?
When I'm using the huggingface transformer to train bert, i see, for every batch the input_ids are different. For example for first batch it is torch.Size([16, 171]) and second batch it is torch.Size([16, 450]). What is the reason?
kowser66
- 125
- 1
- 8
0
votes
1 answer
How bert [cls] can collect the relevant information from the rest of the hidden states
How bert [cls] can collect the relevant information from the rest of the hidden states.??. Does [cls] has mlm information? If i train my bert using only mlm, in this case cls works?
kowser66
- 125
- 1
- 8
0
votes
1 answer
How does BERT loss function works?
I'm confused about how cross-entropy works in bert LM. To calculate loss function we need the truth labels of masks. But we don't have the vector representation of the truth labels and the predictions are vector representations. So how to calculate…
kowser66
- 125
- 1
- 8
0
votes
0 answers
Pre-trained Language Models: Parameters, data, method?
I am doing a research on pre-trained LMs, specifically the following LMs:
BERT
ALBERT
RoBERTa
XLNet
DistilBERT
BigBird
ConvBERT
I am looking for information to compare these LMs like: number of parameters, layers, data on which they were…
Othman El houfi
- 53
- 3
- 9
0
votes
1 answer
OOM while fine-tuning medium sized model with DialoGPT on colab
I am trying to finetune DialoGPT with a medium-sized model, I am getting Cuda error while the training phase, I reduced the batch size from 4, but still, the error persists. My parameters are
#self.output_dir = 'output-small'
…
Sap BH
- 71
- 1
- 6
0
votes
1 answer
Using German GPT-2 in Rasa
Does the Rasa Framework also allow the use of the German GPT-2 model (https://huggingface.co/dbmdz/german-gpt2) with the LanguageModelFeaturizer? I had also tried two older Rasa 2. x versions. There it did not work as well. Does anyone know…
Compty_
- 1
- 1
- 1
0
votes
1 answer
Training a FF Neural Language Model
Consider 3-grams of the sentence "The cat is upstairs" where each word is separated by the rest with @ and ~ symbols.
trigrams = ['@th', 'the', 'he~', '@ca', 'cat', 'at~', '@is', 'is~',
'@up', 'ups', 'pst', 'sta', 'tai', 'air', 'irs',…
john_ny
- 173
- 8
0
votes
1 answer
What is the correct return of BertForMaskedLM?
I'm using huggingface BertForMaskedLM.
For a sentence, I'm getting a 3-dimensional return from BertForMaskedLM.
For example (P,N,V), Here I understand the N is the length of the sentence and V is the vocab size in Bert. But I'm confused about the P.…
kowser66
- 125
- 1
- 8
0
votes
1 answer
How to load spacy language model from local machine?
Since the server I use is not connected to the Internet, I would need to load model from the local disk.
For example, I should be able to run the following code,
from torchtext.data.utils import get_tokenizer
my_language = get_tokenizer('spacy',…
Droid-Bird
- 1,417
- 5
- 19
- 43
0
votes
1 answer
Subprocess call error while calling generate_lm.py of DeepSpeech
I am trying to build customised scorer (language model) for speech-to-text using DeepSpeech in colab. While calling generate_lm.py getting this error:
main()
File "generate_lm.py", line 201, in main
build_lm(args, data_lower, vocab_str)
…
Anjaly Vijayan
- 237
- 2
- 9
0
votes
1 answer
Add custom punctuation to spacy model
How do you add custom punctuation (e.g. asterisk) to the infix list in a Tokenizer and have that recognized by nlp.explain as punctuation? I would like to be able to add characters that are not currently recognized as punctuation to the punctuation…
aoa4eva
- 66
- 5
0
votes
1 answer
Spacy Model load error from local directory
I am trying to find a way to load the downloaded en_core_web_lg ==2.3.1 for Spacy == 2.3.2.
Steps:
Downloaded the tar file
extracted it to path
Code:
import spacy
nlp=spacy.load("path/en_core_web_lg")
Error:
OSERROR: [E053] Could not read…
data_person
- 4,194
- 7
- 40
- 75
0
votes
1 answer
How to get the language modeling loss by passing 'labels' while using ONNX inference session?
When using GPT2 we can simply pass on the 'labels' parameter to get the loss as follows:
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model =…
Sachin Saxena
- 11
- 1