Questions tagged [language-model]
266 questions
0
votes
0 answers
language modeling - model loss and accuracy not improving, model is underfitting
I am trying to build a word-level language model in TensorFlow. My inputs are batches with word id's of shape (batch_size, seq_length), my targets are the inputs shifted one time step to the left (so for each word, the target is the next word in the…
Lemon
- 1,394
- 3
- 14
- 24
0
votes
1 answer
How does language model evaluation work with unknown words?
So for building language models, less frequent words ranked beyond vocabulary size are replaced as 'UNK'.
My question is, how to evaluate such language models that evaluates probabilities based on 'UNK'? Say we want to evaluate the perplexity of…
Ark
- 13
- 3
0
votes
1 answer
Assertion `THIndexTensor_(size)(target, 0) == batch_size' failed
It occured when I writing Python code with PyTorch. I am trying to construct a simple language model using CrossEntropyLoss, but this error came up. Actually, I wrote this piece of code according to this. Here is my code.
import numpy as np
import…
Sun Chuanneng
- 127
- 1
- 9
0
votes
2 answers
character level bidirectional language model in tensorflow
Inspired from Andrej Karpathy Char-RNN, There is a Tensorflow implementation of char-rnn sherjilozair/char-rnn-tensorflow: Multi-layer Recurrent Neural Networks (LSTM, RNN) for character-level language models in Python using Tensorflow. I want to…
Mahdizade
- 3
- 3
0
votes
1 answer
Use Tensorflow LSTM PTB example for scoring sentences
I try to use an example LSTM, trained according to Tensorflow LSTM example. This example allows to get perplexity on whole test set. But I need to use the trained model to score (get loglikes) of each sentence separately (to score hypotheses of STT…
user2098753
- 11
- 4
0
votes
1 answer
How can the perplexity of a language model be between 0 and 1?
In Tensorflow, I'm getting outputs like 0.602129 or 0.663941. It appears that values closer to 0 imply a better model, but it seems like perplexity is supposed to be calculated as 2^loss, which implies that loss is negative. This doesn't make any…
Evan Weissburg
- 1,564
- 2
- 17
- 38
0
votes
1 answer
How to use n-grams for multi-label classification?
I am working on a project to identify the nationality of the person based on the manner in which they communicate in English. I have 10 nationalities & 1000 files, 100 for each nationality. I am using n-grams as features & looking to validate my…
Aazim
- 750
- 2
- 11
- 24
0
votes
1 answer
Keep a language model loaded through multiple program runs
I am loaded a large word2vec language model in Python. Each time I run the program, I need to load the model into memory.
I'm running the same program with different command line arguments from a shell script, e.g.
#!/bin/bash
python processor.py…
Adam_G
- 7,337
- 20
- 86
- 148
0
votes
1 answer
Document Classification tool in C - Compilation error in nested function/scope (may be)
https://stackoverflow.com/questions/43423803/document-classification-tool-in-c-compilation-error/43432470#43432470
In above link-
I get 3 errors after adding -fnested_functions as
gcc -c ./rainbow.c -fnested-functions process_wv.c test_file.c…
Mrinmay
- 41
- 7
0
votes
1 answer
How to build Tensorflow speech recognition integrated with language model
How can I integrate a language model in a tensorflow speech recognition architecture?
There are a bunch of examples out there for building character level speech recognition in Tensorflow (e.g. https://github.com/nervanasystems/neon,…
bear
- 663
- 1
- 14
- 33
0
votes
1 answer
Language Model for PocketSphinx
I've been working with PocketSphinx to make a speech recognizer for natural language. I don't want to use a grammar but rather a language model.
Is it possible that I can't find anything already-done online?
Everybody is linking this page but it…
Giuseppe Trapasso
- 189
- 2
- 14
0
votes
1 answer
How to calculate conditional_frequency_distribution and conditional_probability_distribution for trigrams in nltk python
I want to calculate Conditional Probability Distribution for my language model but I am not able to do because I need Conditional Frequency Distribution which I am not able to generate. This is my code:
# -*- coding: utf-8 -*-
import io
import…
Jai Prak
- 2,855
- 4
- 29
- 37
0
votes
1 answer
Is there any sentence embedding Tensorflow language model?
I found tensorflow 1b_lm project: https://github.com/tensorflow/models/tree/master/lm_1b
I'm just quite confused about the forth example
Give a sentence, dump the embedding from the LSTM state.
However, the results of this example includes 7…
Zachary
- 11
- 2
0
votes
1 answer
Tensorflow: Recurrent neural network training pairs & the effect on the loss function
I am looking at code for an RNN Language Model. I am confused as to 1) how the training pairs (x,y) are constructed and subsequently 2) how the loss is computed. The code borrows from the Tensorflow RNN tutorial ( reader module ).
Within the reader…
ascar
- 3
- 1
0
votes
1 answer
How to run custom seq2seq learning (using pre-calculated word embeddings) encoder-decoder in Tensorflow?
I need to run a encoder-decoder model in Tensorflow. I see that using the available APIs basic_rnn_seq2seq(encoder_input_data, decoder_input_data, lstm_cell) etc, a encoder-decoder system can be created.
How can we enter the embeddings such as…
user3480922
- 564
- 1
- 10
- 22