-2

I want to know the best practice to ftps batch of new documents. Working on a service which which will take batch of documents and ftp it to remote server. I will maintain transaction log of each document which arrived for reporting purposes for example how many docs arrived in a particular day? How many couldn't be transferred due ftp connection failure, etc. In case the app fails for some reason, it should be able to process from where it left.

The possibilities I can think of is below:

  1. Load the documents in database as well along with transaction details for each doc so that upon failure recovery it can start from where it left of.

  2. Store the docs in local filesystem and write a fileWatcher on this folder to ftp.

  3. Ftp the docs as soon as it arrives. In that case if lots of docs arrives, it might result in OutOfMemoryError I think.

which is good approach in terms of error recovery and non-memory intensive?

nanosoft
  • 2,913
  • 4
  • 41
  • 61
  • This is a very broad question, with *many* ways to approach it (including several ways not included in your list of suggestions). Unfortunately, this type of question is off-topic for StackOverflow, as it will only get you opinion and discussion. – David Makogon Oct 10 '17 at 11:49

1 Answers1

-1

"Best" is a pretty broad term, and I'm not sure about your application requirements but, on the face of it, this looks like a good application for Apache Camel. Camel can be embedded in a Java application, or run as a stand-alone application, or integrated with Spring, or run on an application server. You can specify its behaviour in Java code, or XML, among other things.

It's trivially easy (if you know Camel) to write a Camel route that will monitor a particular filesystem directory for files that match a particular pattern, and upload to a remote machine using FTP or FTPS (or many other ways). In simple cases, only one or two lines of Java or XML are needed to specify this behaviour. You can add logging and data collection to the code easily enough.

Camel is very widely used for applications of this sort, and it doesn't seem to struggle with huge numbers of files.

If course, whether this is an appropriate approach or not depends on how well you know Camel, or whether you're willing to learn about Camel. I'm sure there are other approaches that are equally good.

Kevin Boone
  • 4,092
  • 1
  • 11
  • 15