Questions tagged [learning-rate]
83 questions
0
votes
1 answer
How to set learning rate 0.2 when training transformer with Noam decay?
I am training small transformer encoder - transformer decoder translation model using small datasets.
Size of my dataset is less than 200k.
When training transformer with low resource datasets, below 2 papers suggests to use learning rate 2…
user14096975
- 3
- 2
0
votes
1 answer
Why do I need a very high learning rate for this model to converge?
I have a simple model in tensorflow which is being trained on the first 1000 images in the MNIST datset. From my previous experience the learning rates which I used were of the order of around 0.001, however for my model to converge the learning…
finlay morrison
- 225
- 1
- 5
0
votes
0 answers
How to change config file to start fine tuning?
I am working on word detection and using MMOCR’s ‘DBNetpp’ detection model, please refer [2202.10304] Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion.
I want to use this pre-trained model and fine tune on my…
aarya
- 83
- 1
- 8
0
votes
1 answer
Re-setting learning rate while training in Pytorch
I am training a model using Learning Rate Scheduler in Pytorch to decrease the value of learning rate. By using learning rate scheduler, I reduced learning rate from 0.0001 to 1e-5, and save all the weights, parameters, learning rate values, etc at…
killermama98
- 45
- 5
0
votes
1 answer
How to get the actual learning rate in PyTorch?
I'm trying to find the appropriate learning rate for my Neural Network using PyTorch. I've implemented the torch.optim.lr_scheduler.CyclicLR to get the learning rate. But I'm unable to figure out what is the actual learning rate that should be…
Snehangsu
- 393
- 1
- 3
- 12
0
votes
1 answer
The Nadam optimizer does not support tf.keras.optimizers.LearningRateSchedules as the learning rate
this is my first question here. I'm playing with tensorflow.keras, doing some CNNs, and I would like to know if anyone understands why this conflict arises, thanks.
from tensorflow.keras.optimizers import Nadam
from…
Davi6202
- 1
0
votes
1 answer
How to understand loss-learning rate (log scale) plot using learner.lr_plot in ktrain package?
I am using ktrain package to classify text. My experiment is shown as:
lr_find and lr_plot are functions in ktrain. They can be used to highlight the best learning rate, which is shown as the red dot in the plot.
I do not understand how to…
Feng Chen
- 2,139
- 4
- 33
- 62
0
votes
1 answer
Recurrent Neural Network - Fail to apply learning rate reduction
I am building a RNN model using Tensorflow Keras, and I want to reduce learning rate whenever an validation accuracy does not increase. However, I have received an error message indicating container does not exist. I have tried to explicitly…
0
votes
1 answer
learner.recorder.plot() raises attribute errror for "plot"
I'm new to neural network, I'm using fast.ai to train a Cifar10 dataset using residual neural network.
After running learner.lr_find(), I tried plotting it using learner.recorder.plot(),
But then it raised an Attribute error: 'wideResNet' object has…
Syre Musk
- 57
- 7
0
votes
0 answers
The Loss number of both train set and validation set both went down at the begining and then went up again
I try to use the siamese network which combines with two resnet networks (Pretrained) to solve the few shot problem. The loss function is contrassive loss and the optimizer is Adam with 0.001 learning rate.
Both train set loss and validation loss…
user17182657
- 31
- 3
0
votes
1 answer
Loss seems to be proportional with learning rate in Keras
I'm training a simple neural network with a single Dense layer on the MNIST dataset in Keras.
This is the code:
model = Sequential()
model.add(Input(shape=(28, 28)))
model.add(Flatten())
model.add(Dense(10, activation='sigmoid'))
model.compile(
…
Moaz Ashraf
- 210
- 1
- 3
- 11
0
votes
0 answers
Whether keras model.fit() validate calling optimizer?
I write a cosineannealingLRScheduler:
class CosineAnnealingLRScheduler(optimizers.schedules.LearningRateSchedule):
def __init__(self, epochs, train_step, lr_max, lr_min, warmth_rate=0.2):
super(CosineAnnealingLRScheduler, self).__init__()
…
Foot Runist
- 23
- 2
0
votes
2 answers
Change learning rate based on iteration
I am trying to write a custom learning rate scheduler for SGD in Keras, which change the learning rate base on iteration. However, the LearningRateScheduler CallBack only accepts a function which takes epoch only. My learning rate function looks…
Casey Werner
- 1
- 3
0
votes
1 answer
Get Learning Rate from Object
How can I get the value of the learning rate updated at each on_train_batch_begin?
lr_decayed_fn = tf.keras.experimental.CosineDecay(initial_lr, decay_steps)
model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=lr_decayed_fn))
I tried this…
SimoX
- 250
- 6
- 13
0
votes
1 answer
Keras : load pic classification model which has learning rate scheduler but raise value error 'unsupported type () to a Tensor.'
I cut the code because it is quite durity,
Here is code for learning rate scheduler and the model use that optimizer.
initial_learning_rate = 0.001
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
initial_learning_rate,
…
Ha yeong Yoon
- 11
- 2