7

I am using google-colaboratory to train some convnets on image classification with Keras. And I get often disconnected from the notebook which I've found (on some forums) to be a frequent issue with some colaboratory users. And when I try to reconnect it gets stuck on "initializing":

enter image description here

So I decided to use Keras Checkpoint to save the model I'm training on my google drive, after each epoch.

During the training the notebook disconnected after 2 epochs, but when I monitored my google drive I've observed that models are being saved after each epoch, despite my colaboratory being disconnected.

enter image description here

My guess is that programs continue their execution in the background even if my noteboook is disconnected.

My question is, Am I guessing it right? And what do these disconnections really mean? (Are our variables, etc., preserved)?

sideshowbarker
  • 81,827
  • 26
  • 193
  • 197
Alka
  • 767
  • 1
  • 6
  • 13

2 Answers2

9

Normally, notebook connection expires after 12 hours and all the data loaded into colab will also be expired with session.

When you run a process and the notebook get disconnected after some time but before 12 hours, the code run in background.

So, if you try to reconnect the notebook, it get stuck on "initializing" until the completion of code execution or session expiry (whichever occurs first). After that, the connection is established again.

Waqas Shiekh
  • 106
  • 5
0

I am new to Google Colab and also faced this problem. I have to run a project and the process will take more than 24 hours. I know that the virtual machine will terminate anyway after 12h (or 24h for pro), so I tried to save the intermediate results to Google Drive. Now my notebook has been running for about 6 hours, and it was discontinued to VM like 1 hour before. But I found it still keep generating intermediate results to my google drive. I think it will terminate after 12/24h or completion of my code execution.

Chim
  • 1
  • 2