0

We are looking into using a configuration mangement tool (like Salt or Puppet) to automate what is currently a lot of manual work. One requirement is that we have to distribute rather large files / directories to our servers (in sum about 400gb). In the future, we plan to update these files at least monthly, maybe weekly, therfore it would be nice if the process could be fully automated.

Searching online it looks like the usual CM tools are not made for handling large files. Do you know of any CM tool that is good at this task? And if not, what are other common solutions to the problem?

Thanks!

Edit, to clarify:

We want to use a CM tool for a lot of other things as well, not just to copy files. Therfore I was looking for a tool that can handle both tasks well.

Leifb
  • 115
  • 1
  • 9

2 Answers2

2

A configuration management tool is in the wrong category for this kind of task.

While all/most tools in that domain will be able to manage file synchronization it is not worth the effort to implement a configuration management tool just for that. In general it is a good idea to use a configuration management tool but not necessary for file synchronization.

Looking for example at Ansible's synchronize module it is just a wrapper around rsync. A small bash script around rsync is what I would use for such a task.

Henrik Pingel
  • 9,380
  • 2
  • 28
  • 39
  • We want to use a CM tool for a lot of other things, synchronizing large files was just one aspect. Therfore I was trying to find out of there as one that can handle usual CM tasks as well as copying files. Still thanks for the answer! – Leifb Aug 10 '20 at 11:03
0

You can use the CM tools to store the configuration instructions and store the data separately. The instructions would fetch the data files from elsewhere, deploy and configure. The data could be stored and managed on a filesystem or a using an artifact repository manager like Nexus OSS or Artifactory.

Most Version Control Tools (VCS, eg;git) do not handle binary artifacts well. They are not able to calculate usable deltas efficently and end up storing entire copies of binaries, even for minor changes (like the timestamp in a MANIFEST.MF of a war file).They may deduplicate if identical copies are stored in multiple locations in a filesystem. It takes added knowledge of the internals of the binary to do better.

Ian W
  • 239
  • 1
  • 2
  • 8
  • While I have never used on, I feel like Nexus OSS or Artifactory are a bit overkill, as we don't need all the pckage management stuff. Maybe the esiest thing would be to create a custom little program that checks the checksum of the files and gets now ones using ftp or something similar. Thanks! – Leifb Aug 10 '20 at 11:06