Error when defining Camel S3 source connector

Question

I am trying to define a Camel S3 Source Connector in our confluent environment.

This is the configuration I am using

{
    "name": "CamelAWSS3SourceConnector",
    "config": {
        "connector.class": "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
        "camel.source.maxPollDuration": "10000",
        "topics": "TEST-S3-SOURCE-POC",
        "camel.source.path.bucketNameOrArn": "json-poc",
        "camel.component.aws-s3.region": "us-east-1",
        "tasks.max": "2",
        "camel.source.endpoint.autocloseBody": "true"
    }
}

And this is the error I receive, when I try to define the connector

{
    "error_code": 405,
    "message": "HTTP 405 Method Not Allowed"
} {
    "name": "CamelAWSS3SourceConnector",
    "connector": {
        "state": "RUNNING",
        "worker_id": "confluent-connect-server2:8083"
    },
    "tasks": [{
        "id": 0,
        "state": "FAILED",
        "worker_id": "confluent-connect-server2",
        "trace": "org.apache.kafka.connect.errors.ConnectException: Failed to create and start Camel context
    at org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:118)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:215)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Option bucketNameOrArn is required when creating endpoint uri with syntax aws-s3://bucketNameOrArn
    at org.apache.camel.support.component.EndpointUriFactorySupport.buildPathParameter(EndpointUriFactorySupport.java:53)
    at org.apache.camel.component.aws.s3.S3EndpointUriFactory.buildUri(S3EndpointUriFactory.java:103)
    at org.apache.camel.kafkaconnector.utils.TaskHelper.buildUrl(TaskHelper.java:68)
    at org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:98)
    ... 8 more"
    }],
    "type": "source"
}

What would be the cause of above error?

I am told that since the connect server is an ec2 instance, I don't have to define AWS parameters here. Is that correct?

Thank you

Note: Wanted to add that there are two connect servers and the error shows up for only one of them in the output

Does anybody have any suggestion where I should look to be able to resolve this issue? Thanks — adbdkb, Dec 12 '20 at 21:44

score 0 · Accepted Answer · answered Dec 17 '20 at 16:21

Just wanted to post an answer here in case someone else runs into this.

The issue was when using PUT to add or update a connector the url format of the curl request needs to be http(s)://<serverurl>/<connectorName>/config. My json contained the key "name:" and url was just http(s)://. Initially, I created the connector using POST and then was trying to update using PUT to update ( actually, add ) the bucketNameOrArn key which was not getting updated in reality.

Error when defining Camel S3 source connector

1 Answers1