0

I’m trying to come up with a strategy to backup data in my apache ignite cache hosted as a stateful set in google cloud Kubernetes. My ignite deployment uses ignite native persistence and runs a 3 node ignite cluster backed up by persistence volumes in Kubernetes. I’m using a binaryConfiguration to store binary objects in cache.

I’m looking for a reliable way to back up my ignite data and be able to restore it.

So far I’ve tried backing up just the persistence files and then restoring them back. It hasn’t worked reliably yet. The issue I’m facing is that after restore, the cache data which isn’t binary objects is restored properly, e.g. strings or numbers. I’m able to access numeric or string data just fine. But binary objects are not accessible. It seems the binary objects are restored, but I’m unable to fetch them.

The weird part is that after the restore, once I add a new binary object to the cache all the restored data seems to be accessed normally.

Can anyone please suggest a reliable way to back up and restore ignite native persistence data?

Rico
  • 58,485
  • 12
  • 111
  • 141
Chinmoymohanty85
  • 127
  • 11
  • How are you performing the backup? Are you shutting down the cluster before taking the copy? – Stephen Darlington Jan 31 '19 at 10:26
  • @StephenDarlington I'm not shutting down the cluster before taking a copy. All the primitive data in the backup is accessible after the restore, but only binary data is causing an issue. – Chinmoymohanty85 Feb 01 '19 at 04:35
  • But if the cluster is running, your backup may not be consistent. You need the data files and WAL from all the nodes to be at exactly the same time. Do you get the same issue if you perform your backup when the cluster is down? – Stephen Darlington Feb 01 '19 at 14:46

2 Answers2

0

Apache Ignite providers ACID transactions which are pretty reliable. The cache also uses its own mechanism for primary backups and copies and assuming you have its WAL enabled some stuff is kept in memory.

The most likely thing happening is that you do your restore and the moment you make an initial write memory starts populating allowing you to see what's on disk (cache). This is not really a supported restore mechanism (there isn't one in the docs) but it could work that way where after the restore you run a minor sample irrelevant write. I advise testing this thoroughly though.

Rico
  • 58,485
  • 12
  • 111
  • 141
0

You should either backup ${ignite.work.dir}/marshaller directory, or call ignite.binary().type(KeyOrValue.class) for every type you have in cache to prime binary marshaller.

alamar
  • 18,729
  • 4
  • 64
  • 97
  • Hi @alamar . I think this is something that could work. Looks like the binary marshaller somehow doesn’t kick in after the restore. I’ll try this ASAP. – Chinmoymohanty85 Jan 31 '19 at 14:57
  • unfortunately this doesn't seem to work. The only way I can get it to work is by adding/updating a binary object with the same type in the cache. Is there a recommended way to backup ignite persistence caches so that restoration would be reliable. – Chinmoymohanty85 Feb 01 '19 at 04:32
  • @Chinmoymohanty85 as I have already said, backing up `marshaller` directory should be reliable. That or using [GridGain snapshots](https://docs.gridgain.com/docs/data-snapshots). – alamar Feb 01 '19 at 09:47
  • This seems to work now. somehow I was missing a few steps while restoring. Restoring the marshaller folder actually solves the issue. Thank you. – Chinmoymohanty85 Feb 02 '19 at 20:53