0

Background:

Several production machines produce text files with various information that we want to aggregate centrally.

The machines are running all versions of WinXP, 7, 10.

The machines are tightly locked on segmented networks. They do not have access to a Windows or Samba share.

The files themselves are in order of a few hundred kiB per day.

Goal:

I want to set up a central tool that can fetch these text files from each machine at regular intervals.

Fetching must be done from sever side.

The files are appended to every few minutes in each machine, so it is important that the file copy mechanism can deal with files that are locked for writing and not loose data etc.

Each machines files must be stored locally on each machine because the application itself is too sensitive to store directly on a network share (it fails, crashes or looses data).

It would be possible to install some kind of daemon or server (sftp, ftp,..) but the server software must be pretty stable and light weight.

Server side can be Linux or Windows

Failed connections, failed transfers should be logged, and possibly emailed.

A side note: Many years back i remember I used a Windows file sync software that could do this; fetch/sync from multiple sources, check file locking status, send to multiple targets, including versioning, and also run various commands on success or failed transfers. But I cannot remember what it was.

AL3
  • 23
  • 5

1 Answers1

0

If the transfers are always uni-directional, from the production hosts to the aggregation server, then rsync is a standard choice. If it's bi-directional, Unison is a good choice and you can configure it to deal with conflicts in different ways.

Either of those would need to be available on both the aggregation and production hosts. You can get them on the production hosts in either WSL or Cygwin.

Andrew Schulman
  • 8,811
  • 21
  • 32
  • 47
  • It seems that rsync with cygwin is a good idea. I made a test setup and it handles open files gracefully. One problem with cygwin is to be able to make a portable installer. But I think it is possible using https://vegardit.github.io/cygwin-portable-installer/ – AL3 Nov 04 '20 at 12:38
  • See also [Can I script a Cygwin install to include certain packages?](https://stackoverflow.com/questions/745275/can-i-script-a-cygwin-install-to-include-certain-packages/7366200) I think the main problem will be keeping Cygwin up-to-date on all of your production hosts. – Andrew Schulman Nov 04 '20 at 12:40
  • I'll go with Cygwin. Was able to create a portable package using the portable-installer and including custom configurations for rsync etc. Thanks for the suggestion! – AL3 Nov 05 '20 at 11:25