0

We have a spring integration application which is running in a cluster on multiple nodes. A NFS drive is mounted across all nodes and is readable by all nodes.

On NFS there is a folder which is being watched by file:inbound-channel-adapter of all nodes with a polling interval of 5 seconds.

The problem is 1) One node should not pick up a file which has been picked up by other node

2) If a file is picked up by a thread on one node, second thread which runs after polling interval should not pick it up- I have in place AcceptOnceFileListFilter for this one, but wondering if there is a better way.

One of options which we have looked at already:

1) Create a record in DB with unique filename and every other node checks on same DB before starting processing

Have to yet look at NIOLocker - but need some inputs on it's reliability and possible options in this scenario.

Vishal Biyani
  • 4,297
  • 28
  • 55

1 Answers1

0

have you tried renaming the file once it has been accepted? that is, the first thing you do is rename the file to file.txt.lck meaning that other nodes would no longer find it matching their filter criteria.

should a node fail, the file can be manually reverted and processed by another node

incomplete-co.de
  • 2,137
  • 18
  • 23
  • 1
    Well, it has it's set of issues. In between one node renaming, if the other node might get same set of files (I know possibility is less, but exists) and then throw FileNotFound. Anyway, in a general sense, we concluded that this is a wrong way to solve problems. From EIP book, if two systems are looking greedily at same resource, then mediator pattern is a better solution. Have a mediator lookup files and push to nodes for processing. And I think that is a better solution personally – Vishal Biyani Jan 03 '13 at 06:52
  • 1
    very true - the mediator with a proper management for redundancy and failure would work. this would mean one node is nominated as the 'master' and the remainder as the 'slave' – incomplete-co.de Jan 03 '13 at 12:40