0

I have four child pipelines in my project and the output for these is ingested in the main pipeline. I want the output files for the child pipelines to be automatically backed up after baseline in some directory. This will help if i disable individual forges and want to restore previous index. Please help

Anuj Sharma
  • 11
  • 2
  • Are you sure this is what you want to do? Sounds like the wrong solution to start with. Have you looked at any solutions regarding Search Federation? Through federation (and there are commercial packages that can help federate the queries) you can have partitioned data sets. So if each forge represents one of these data sets then by federating the queries/results you can protect the integrity of individual partitions. There are other considerations though but without understanding why you have parallel forges running in the first place (and how they join up) I can't really comment further. – radimpe Apr 05 '13 at 08:21

1 Answers1

1

It should be straight-forward to add copy or archive jobs to your baseline update script to execute this sort of thing before your main forge is run.

radimbe, as forge itself is single-threaded (except for the rarely implemented multi-threaded java manipulator) and monolithic, this sort of architecture is commonly used to take better advantage of multi-processor machines and multi-threaded CPU cores. In addition, if data is made available at different times or with different frequencies, you can decouple child forges from your main baseline process, improving its overall turnaround time. And from a strategy POV, this approach could decompose what might be a large, unwieldy job into perhaps simpler, more focused and more easily maintained components.

matgeech
  • 21
  • 1