Stack Overflow
Stack Overflow

Questions tagged [automatic-mixed-precision]

24 questions
0
votes
0 answers

Implementation of Gradient Accumulation with SAM Optimizer together with Mixed-Precision

I'm trying to use Sharpness-Aware Minimization (SAM) optimizer in my code, using the already built Pytorch code from here. Then, I would also like to use gradient accumulation, but I have no idea how to make this works properly. Using the proposed…
Gregor Isack
  • 1,111
  • 12
  • 25
0
votes
0 answers

Issues with from apex import amp

In my Google Colab notebook, I am attempting to utilize Nvidia's apex for the amp feature. However, when I attempt to add it, it states "No module named 'apex'" I tried this but it still resulted in an error: !git clone…
Ram Bhagat
  • 1
0
votes
0 answers

mixed_precision.experimental.LossScaleOptimizer() in TF version 2.2.0

I am trying to use mixed_precision in TensorFlow 2.2.0. Since I wrote the code in tf version 2.9.2 and need to run it on a computer with version 2.2.0. I couldn't find a page for the experimental mixed_precision page since it is embedded into core…
bidon
  • 27
  • 4
0
votes
0 answers

mixed precision training cause numerical overshoot in Conv2D

Training CNN model in keras 2.4 and 2.9 using mixed precision Model sporadically diverge during training with loss=nan In model layer, found it caused when a Conv2D multiply two values and outcome exceed 65,504. Not sure how come 'mixed precision'…
YoavEtzioni
  • 85
  • 10
0
votes
0 answers

Scaler.update() - AssertionError: No inf checks were recorded prior to update

I am new to Pytorch and trying to implement ViT on a spectrograms of raw audio . My training input consists of tensors [1,80,128] (almost 1M) of them and I am exploring AMP to speed up my training on a V100(16GB). My training loop is as below scaler…
Devesh Khandelwal
  • 1
0
votes
1 answer

PyTorch loading GradScaler from checkpoint

I am saving my model, optimizer, scheduler, and scaler in a general checkpoint. Now when I load them, they load properly but after the first iteration the scaler.step(optimizer) throws this error: Traceback (most recent call last): File…
Jarartur
  • 149
  • 1
  • 10
0
votes
1 answer

Sigmoid vs Binary Cross Entropy Loss

In my torch model, the last layer is a torch.nn.Sigmoid() and the loss is the torch.nn.BCELoss. In the training step, the following error has occurred: RuntimeError: torch.nn.functional.binary_cross_entropy and torch.nn.BCELoss are unsafe to…
Celso França
  • 653
  • 8
  • 31
0
votes
0 answers

How to use automatic mixed precision with TensorFlow?

I don't manage to use Automatic Mixed Precision with TensorFlow 2.3.2 (on Windows 10). I have a TF_ENABLE_AUTO_MIXED_PRECISION variable set to 1 in my system environment. I have enabled memory growth using explicit Python code. I am doing model…
Martin Frank
  • 199
  • 1
  • 13
0
votes
1 answer

tf2.4 mixed_precision with float16 return 0 gradient

This question was posted here before, and I re-opened it here to draw more attentions. The main issue is that when testing in normal float32 env, the tensorflow returns gradient like, but after I shift to…
wingsofpanda
  • 5
  • 3
1
2