2

We currently developing an application that read files from file system/ftp/http and upload this files to mongodb. After that we want to analyze this files and write analyzed documents in mongodb. Requirements for this process:

  1. Real-time uploading and analyzing.
  2. Failure handling and restarting process.
  3. Possibility to re-analyzing files.

How properly organize this process in Spring XD? In what cases we should use streams or batch jobs? If anyone can tell best practices for using streams instead batch jobs and vice versa, it is very much appreciated.

Anton Semahko
  • 21
  • 1

1 Answers1

0

From my experience, For real-time uploading, analysis and storing (mongodb in your case) the best way is with streams. the reason is that you don't really want to start and handle jobs in terms of how many times should it run in a minutes / hour and so on ...

Restarting and failure handling should be up to you, you'll need to write a script to redeploy the streams and jobs upon failure.

re-analysis can be done using jobs as it is done in specific times

milonimrod
  • 306
  • 2
  • 3