0

So I tried running some long calculations in a Datalab notebook. It should have finished overnight. It did not.

What happened is Google shut down my datalab instance a bit after midnight, without stating any reason I can find. Why did they do so? It appear google shut it down on their end. The computer did fall asleep at one point, when it became unplugged. Maybe if the computer falls asleep, it still looks like google closed it from their end. This is completely unacceptable default behavior. Google shouldn't be assuming people have their computers running 24/7.

How can i prevent google from shutting down instances which are still working?

On a semi-related note, how can I keep notebooks running even if my computer doesn't maintain the connection? If my internet goes down, if the computer goes to sleep, etc, I want my cloud notebook to continue running. That's part of the point of the cloud afterall.

Chad Bernier
  • 386
  • 1
  • 10
  • I'd suggest you'd better use a VM instance and schedule a job via `linux` command tools (or if you prefer - `ML Engine`). Datalab is not the best place to run such computations. – irudyak Jul 19 '18 at 08:55

1 Answers1

1

See this: https://cloud.google.com/datalab/docs/concepts/auto-shutdown

After your computer is shutdown, your notebook's kernel will keep running on the VM (until it's auto-stopped, or you shut it down later), so if you were in the middle of a long-running notebook cell it should keep running to end. After the connection to the notebook is lost, the rest of the notebook won't run, and you won't be able to see the results of your commands inside the notebook.

yelsayed
  • 5,236
  • 3
  • 27
  • 38
  • Right, I think I jumped the gun a bit. It seems to be better with my next test. The fact does remain though that it did shut itself down last night while running code. I don't know the cause of that. – Chad Bernier Jun 28 '18 at 20:59
  • It shut itself down again this afternoon. The turned the computer off around 5, at 6:40, shut itself down, despite having code running. – Chad Bernier Jun 29 '18 at 01:11
  • Are you getting some errors or something else in the logs that might be interesting? Also what kind of operations are you running, do the instances have enough resources? – komarkovich Jun 29 '18 at 11:15
  • Did you go through the link I added above? You can turn off auto-shutdown from the Datalab UI, otherwise, it defaults to 90 minutes of shutdown I think. – yelsayed Jun 29 '18 at 18:40