1

I have existing Spring Boot application that uses Camel framework for getting data from different folders. All routes poll data in similar way:

from(fileUriWithCurrentDay(path, config.getParams())).routeId(ROUTE_ID)

where method defined as follows:

public static String fileUriWithCurrentDay(String path, Map<String, String> params) {
path = path.endsWith(SEPARATOR) ? path : path + SEPARATOR;
return "file:" + path + currentDayPath() + fileParams(params);
}

and returns something like depending on the day:

c:/test/data/2021/242?noop=true&idempotent=false&autoCreate=false&includeExt=xml&scheduler=spring&scheduler.cron = "0 0 13 ? * *"

The cron is set to start one time per day. When cron time is reached the route starts to perform its job and finishes it correctly. But on the next day it starts to poll the same folder, e.g. 242 but it should be 243.

So the problem is: from endpoint should be calculated dynamically at its start.

I've read about dynamic from: in Camel, but didn't find any information that such option is supported out of the box. I've tried to use pollEnrich() option using header that is calculated, but it seem doesn't support dynamic endpoints as returns next error:

org.apache.camel.ResolveEndpointFailedException Dynamic expressions with ${ } placeholders is not allowed. Use the fileName option to set the dynamic expression.

How from endpoint can be refreshed after route is started?

I also see 2 possible solutions but didn't know will it help:

  1. Restart routes for recalculating from endpoint uri.
  2. Restart whole Camel context. Currently all routes are marked as Spring @Component.
Andrey
  • 433
  • 1
  • 5
  • 19

2 Answers2

3

Unlike other EIP component, string in routeId and from endpoint are fixed once the route is created (unless you recreate the route with another from endpoint url). Thus, 2 possible direction to move on. One way is to recreate the route with new from endpoint url and another is to lookup parameter in the component to support your action.

  1. Recreate consumer route with same routeId in same CamelContext

    • Trigger below action daily (e.g. use timer route)
    • Use RouteBuilder to build route config (with same consumer route id and new from endpoint url)
    • recreate route using addRoutes method of CamelContext class with new route config

  2. Leverage filterDirectory in file component

    • Set file path to common root folder (i.e. c:/test/data)
    • Enable recursive to lookup sub-directories
    • Use filterDirectory to filter directory base on simple language
      • From simple language documentation, date command of simple language is using java.text.SimpleDateFormat
      • From Oracle doc, SimpleDateFormat do support both year and Day in year (assume 242 and 243 in your example are day in year)

Disclaimer

I did not use filterDirectory before, way 2 is deduce from Camel documentation.

hk6279
  • 1,881
  • 2
  • 22
  • 32
  • 1
    I tried to use 2nd option but it seems it doesn't filter and select required sub-directory. I didn't understand how it could be used at all due to lack of info, so I posted new thread for it - https://stackoverflow.com/questions/69345083/what-filterdirectory-of-camel-file-component-does. – Andrey Sep 27 '21 at 10:35
  • So after research I found out, that `filterDirectory ` is suitable if you need to go only to the inner subfloder, e.g. in 2021 not to 2021/242. The 1st option is ok but it takes a lot of code plus there are questions about its reliability. So I solved the issue using custom `GenericFileFilter`. – Andrey Sep 29 '21 at 09:01
1

Looking at the documentation for the File component what you are trying to do is not possible purely with the from-endpoint URI.

I would suggest using a common base directory in the URI and creating an implementation of GenericFileFilter that you'd use with the from-endpoint. It allows for a lot of freedom in implementing the logic for which files are and which aren't processed.

To read files from base/path/2021 create something like this:


@Component
public class FileFilter<T> implements GenericFileFilter<T> {
    @Override
    public boolean accept(GenericFile<T> file) {
        if (file.isDirectory()) {
            return "2021".equals(file.getFileName());
        }
        return file.getFileName().startsWith("2021/");
    }
}

with endpoint URI like this:

file://base/path?recursive=true&filter=#fileFilter

When implementing the GenericFileFilter you should be mindful that accept() will be called for the directories as well.

jwwallin
  • 118
  • 7