2

I was running a Google Cloud Managed Instance Group (Ubuntu) to run a web server, and this morning the server went offline. I checked on the server status and it appeared that all data went missing. All apache2 files, the apache2 and PHP services, modules, etc. are all missing as if it's a fresh install.

At first I thought it had to do with the fact that I hadn't assigned a static IP to my server, but after logging in to the SSH I noticed that all www and apache2 files from my web server were gone.

I noticed it happened this morning when my server reached the autoscaling point for CPU usage. Apparently it created another instance, but there's no data on it.

I'm currently setting up another VM Instance, but obviously something went terribly wrong. What should I do to prevent this from happening/

Dxx
  • 161
  • 1
  • 5
  • In your configuration of autoscaling are you using existing virtual machine's image to start the new image ? – terminal ninja May 17 '16 at 11:50
  • @terminalninja No, this was indeed the problem. I hadn't configured autoscaling properly and it started a new image. Is it possible to (and should I) make a new instance automatically use the data on the current machine? Or should I be using an additional persistent disk for my web files and assign it to an instance template? – Dxx May 17 '16 at 12:06
  • Here is how I used to do in my AWS, autoscaling start new instance from the base image of the existing machine. Then on top of that chef runs the cookbooks and clone project from git. If you use a new image you will have to handle the delay of setting up and configuring the whole system. – terminal ninja May 17 '16 at 12:08
  • Thanks @terminalninja. Currently trying with an external hosted hard drive, will try git as well! – Dxx May 18 '16 at 10:50
  • Take a look at my answer to a similar question on [this thread](https://groups.google.com/d/msg/gce-discussion/UeG7j7Kw1ow/loONWunRDAAJ). – Kamran May 19 '16 at 03:22

1 Answers1

1

Data loss

It seems from the comments that the data is gone because it resided in an instance that was part of a managed instance group. When the group required additional resources, a new instance was created based on the instance template. Presumably, the template did not contain any data that had been gathered on a given instance, thus the new instance contained no data. This is precisely how it is possible to lose the data stored on these instances.

Data discrepancies

In addition, storing data directly on these instances will almost immediately lead to data discrepancies if there's no system in place to synchronize data across instances. Given a group with 2 instances (A and B), a load balancer sends a request to instance A resulting in a write. Moments later, the load balancer sends a new request to instance B seeking out the data that was recently written. It will find nothing. Thus, the data must either be synchronized which can be very costly and error prone, or better yet migrated outside the scaling instance group.

Stateless instance design

In a GCE Discussion group post about scaling single VMs, Kamran describes how to prevent this sort of data loss/discrepancy. With instances having no persistent state (continuing to exist after they've been shut down), they can be considered stateless. The advantage to making all instances in a given group stateless is that one can expect any instance in the group to behave the same way and handle tasks the same way. This, therefore lends itself quite well to horizontal scaling (having more instances to process more work rather than simply having a more powerful instance).

Storage outside the instance group

Generally, this stateless design requires that you migrate your data/file storage outside the instance group. In the case of SQL, one could move the data to an instance of Cloud SQL accessible to the instance group or Cloud Datastore for a NoSQL alternative. For file storage, one could migrate files to Cloud Storage. You could even use Cloud Storage FUSE to mount a given GCS bucket as a file system on your managed instances.

Nicholas
  • 236
  • 1
  • 7