I am using the Apache Camel File component to read from a local directory and upload to an AWS S3 bucket. This route has worked flawlessly in the past, but is having issues with certain files now-a-days.
In my debug investigations thus far (3 days of misery), I have found that the route reaches the DelegateSyncProcessor class and does not have any exception set on the exchange (for reference see the relevant code of the previously mentioned class below). Considering there aren't any exceptions, I can't seem to figure out why the onCompletion() methods never fire for specific files. There weren't/aren't any exceptions for problematic files; however, the file and the lock continue to exist after all the logic has executed (including the final transfer of the message to the .to() endpoint) hinting that there is some internal Camel issue. I suspect this because the route behaves normally in every aspect except the file and lock deletion.
After enabling debug logs for Camel, I was frustrated to see that there were no logs related to an error during the route execution. I would love to hear any advice about what could be happening under the hood.
A few extra things to note:
- I am running camel 2.16.0
- No easily detectable issues were seen while debugging through the camel code.
- The logic in the
.process()sections runs all the way through (some exceptions occurred, but they are properly handled.
Update:
More debugging with debug logs enabled for camel yielded no results. I have only found that the logic within my .process() code is the issue. When the code takes too long to execute (even without exceptions), the file deletion fails.
Update 2:
I found where the camel route actually falls apart. The CamelEventLogger.java class completely fails while trying to "logEvent". The related code to that class is added below. When the code reaches matcher.find(), it times out.
MyRouteClass.java:
from(importProcessingEndpoint)
.convertBodyTo(byte[].class)
.process((exchange) -> {
// some logic here
})
.to(outgoingEndpoint)
.threads(MAX_NUMBER_OF_CAMEL_THREADS)
.process((exchange) -> {
// some logic here
})
.log("Finished processing import file.")
.to(outgoingEndpoint);
DelegateSyncProcessor.java:
@Override
public boolean process(Exchange exchange, AsyncCallback callback) {
// force calling the sync method
try {
processor.process(exchange);
} catch (Throwable e) {
// must catch throwable so we catch all
exchange.setException(e);
} finally {
// we are bridging a sync processor as async so callback with true
callback.done(true);
}
return true;
}
CamelEventLogger.java
private void logEvent(final String label, final Exchange exchange, final Endpoint endpoint,
final long elapsedTime, final boolean logTID) {
Matcher matcher = PATTERN.matcher(extractMessage(exchange));
if (matcher.find()) {
//CHECKSTYLE:OFF
String evtType = matcher.group(1);
String evtName = matcher.group(2);
String guid = matcher.group(3);
String tid = matcher.group(4);
//CHECKSTYLE:ON
if (tid == null || !logTID) {
tid = "";
} else {
tid = "intuit_tid=" + tid;
}
log.info(LOG_FORMAT, label, exchange.getExchangeId(), guid, evtName, evtType, endpoint, elapsedTime, tid);
} else { // the message is not parseable, fall back to minimum log entry
log.info("Event {} id={} {} elapsedTimeMs={}", label, exchange.getExchangeId(), endpoint, elapsedTime);
}
}
