0

Currently I'm running a number of back-up servers to back-up other servers with, e.g. b01, b02, .., bn, all with their own IP's, running their own FTP / SSH services. However, I would like to create just one interface to store and retrieve backups, making it easier for myself and clients by always connect to the same host while the actual data is stored on multiple back-end servers, and also improve the scalability of the system.

I'm currently using ZFS with snapshots (and compression/dedup) to store the backups, each server which is backed-up has it's own volume (20-500G) on a ZFS backup server, which is snapshotted every day for retention.

Does there exist a program or technique to mount / simulate a directory from another (backup) server on an incoming FTP / SSH connection to the "connection server(s)"? It should be scalable and if possible redundant, I have been unable to find anything.

I'm also open to other solutions and entirely changing the current backup setup, but it has some requirements:

  1. Snapshot backups for retention, store differences only
  2. FTP / SSH (rsync) access
  3. If possible, apply some compression and/or deduplication to save disk space
  4. Scalable to hundreds of TB's
  5. Perform well
  6. Redundant

I have been exploring the possibility of using an object store like Openstack Swift, but snapshot are not possible.

Therefore my question is, how can I achieve my goal of creating some kind of back-up cluster with one interface, to replace the current setup existing of standalone servers.

Evianon
  • 3
  • 1

1 Answers1

0

Not sure if this is exactly what you seek, but basically it sounds like you're looking for a distributed filesystem.
There exists several such products (starting from drbd, through, ceph, lustre, and gluster. I'm sure there are more). Because of the ZFS existing infrastructure, I would advise either lustre (also see zol ) or any distributed fs that allows another fs on top of it.

Lustre has a draw back in that it is designed primarily for hpc scratch data - that means high performance, low reliability storage, and is thus not optimized as a backup solution.

Ceph might be a better solution for your needs, but it's support of zfs is still lacking

That said - I would advise looking into gluster which has community support for such setup, Although their route is gluster on top of zfs (which means that snapshots are at the individual pool level, instead of at the filesystem namespace level).

I would still advise against drbd for anything mission critical, but if your data is backed up further (e.g. to tape) then drbd above/beneath zfs might be a viable solution as well.

drbd on top of zfs is probably safe enough, but you still loose global namespace wide snapshots.

Dani_l
  • 498
  • 2
  • 8
  • I've been looking into distributed filesystems (in particular Ceph), but they always lack snapshots, are slow or ain't able to compress data. Also, I'd still have to mount the disks on another server to access the data in one interface using FTP / SSH protocols. I'll look into this further, but maybe there is another solution. – Evianon Jul 20 '14 at 09:58
  • use zfs for snapshots, lustre has good performance and integrates well with zfs (google e.g. warp lustre zfs). In LLNL they achieved a 1TB/s bandwidth to a lustre/zfs storage, but that uses IB cards amongst other things. As for clients - use one (or more, with loadbalancing) of the existing bn servers as ftp server. Each can see the entire namespace. – Dani_l Jul 20 '14 at 11:25