How to have disposable routes in Camel?

Question

Usecase: We generate a csv file when user hits an endpoint, which we want to upload to different sources (FTP & S3). We need to encrypt & transform the files before uploading. Here's how the routes look like (oversimplified):

FTP Route:

from("file:temp?include=sample.csv&noop=true")
.routeId("ftpRoute" + LocalDateTime.now())
.marshal().pgp("file:temp/encryptionKey.asc", "someuserid", null, true, true)
.process(new SomeComplexProcessor())
.to("sftp:localhost:22/destinationDir?username=username&password=RAW(password)")
.setHeader(FILE_NAME, "metadata.txt")
.process(new MetadataFileGenerator())
.marshal()
.bindy(BindyType.Csv, MetadataFile.class)
.to("sftp:localhost:22/destinationDir?username=username&password=RAW(password)")
.process(new KillRouteProcessor());

S3 Route:

from("file:temp?include=sample.csv&noop=true")
.routeId("s3Route" + LocalDateTime.now())
.marshal().pgp("file:temp/encryptionKey.asc", "someuserid", null, true, true)
.process(new SomeComplexProcessor())
.to("aws-s3:bucketName?accessKey=ACCESS_KEY&secretKey=RAW(SECRET_KEY)&region=REGION")
.setHeader(FILE_NAME, "metadata.txt")
.process(new MetadataFileGenerator())
.marshal()
.bindy(BindyType.Csv, MetadataFile.class)
.to("aws-s3:bucketName?accessKey=ACCESS_KEY&secretKey=RAW(SECRET_KEY)&region=REGION")
.process(new KillRouteProcessor());

What's working: S3 & SFTP upload routes are working correctly.

Requirements:

It'd be great if the code can be shared. Some different parameters / processors could be present in both routes.
File needs to be deleted after the execution of both routes.
We need to kill the routes after upload, as each request will have a different filename.
Above two routes cannot be merged into one, as there are too many if-else conditions inside the actual routes, which'd further complicate this.
Routes should be optional (eg. Choice of upload to s3/sftp or both should be available)

What I've tried:

Camel Direct: It helps with code reuse, but it doesn't allow multiple consumers (In my case these two routes)
Camel Seda: It allows multiple consumers, but it doesn't seem to allow synchronous routes.
Deleting the file outside of camel context. This is a problem, as we won't know how much time will the routes take to upload files.

Environment:
Camel 3.4.3, Spring Boot 2.2.3, Java8,

The uri in the from endpoint is a string. You can build it anyway you want! As for your second requirement, you can add the `delete=true` option in your from uri instead of `noop=true`. For your 3rd requirement, could you please clarify why you have to delete the route? You can also have optional routing using camel choice which works exactly like an if else block https://camel.apache.org/components/latest/eips/choice-eip.html. If you want multiple consumers using direct, you can have a look at multicast https://camel.apache.org/components/latest/eips/multicast-eip.html — Sneharghya Pathak, Aug 28 '20 at 06:00
@SneharghyaPathak It'll even work if i remove `noop=true` without `delete=true`. The concern here is that I have to execute both routes, and after making sure that both are executed, we've to delete the file. Also multicast component looks different. It allows to call multiple direct endpoints, which is not my case. And i'm deleting the routes, because I'll need a different route everytime for a different filename. If it's possible to change filename in 'from' string for a route dynamically, it'd be great. — Lakshmikant Deshpande, Aug 28 '20 at 06:27
If you remove the include option in the from endpoint, it can pickup any file irrespective of the name. Also, if the logic for generating the csv file is same in both cases, you can do it in the same route. something like `.to(ftpendpoint).to(s3endpoint)`. — Sneharghya Pathak, Aug 28 '20 at 06:32
@SneharghyaPathak Yes, that's possible. But in my use case, different customers will call an endpoint which calls this camel route. So if i keep that route running, one customer's file could be sent to another customer which is not what we want. That's why I wanted to know if it's possible to create, use and dispose camel routes on the fly — Lakshmikant Deshpande, Aug 28 '20 at 06:50
Also `.to(ftpendpoint).to(s3endpoint)` is doable, but sftp may need encryption, s3 may not need. It'd add a lot of if else conditions, and complexity if we use the same route. So I have separated the routes. — Lakshmikant Deshpande, Aug 28 '20 at 06:58

Lakshmikant Deshpande · Accepted Answer · 2020-10-02T10:26:02.313

I managed to get it working by using a static route id (Removed LocalDateTime.now()). Here's what I learnt while fixing Camel issues.

Always provide route ids to every route. Especially while using 'direct' routes.
Never use dynamic route id. This is very important. I was seeing this issue when I had used LocalDateTime.now() in my route id. I was seeing this error before I changed:

Multiple consumers for the same endpoint is not allowed: direct://routeName .....

When using loops or calling any direct routes, always use enrich.

eg. .enrich("direct:subroute", AggregationStrategies.useOriginal()).
This will share a copy of parent route's headers to the subroute. It'll help you to avoid some weird issues.

Feel free to comment in case if you'd like to know more.

How to have disposable routes in Camel?

1 Answers1