0

I am trying to start Kafka connect in distributed mode even in standalone also i am not able to proceed

This is my elastic search sink properties 

name=elasticsearch-sink
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=5
topics=fsp-audit
key.ignore=true
connection.url=https://****.amazonaws.com
type.name=kafka-connect
errors.tolerance = all
errors.deadletterqueue.topic.name = fsp-dlq-audit-event

this is my connect-distributed.properties

bootstrap.servers=***:9092,***:9092,***:9092
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=1
schema.enabled=false
config.storage.topic=connect-configs
config.storage.replication.factor=1
status.storage.topic=connect-status
status.storage.replication.factor=1
offset.flush.interval.ms=10000
plugin.path=/usr/local/confluent/share/java

I also have created three topi upfront 

connect-offsets
connect-configs
connect-status

I am running this on EC2 and using MSK as Kafka . I checked connectivity from my EC2 to MSK and i am able to telent

The error i get this 

[2020-01-30 08:53:12,126] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:237)
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
[2020-01-30 08:53:12,145] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:237)
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
[2020-01-30 08:53:12,149] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:83)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:94)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)

Question :If i have to run Kafka connect in distributed mode,do i have to use more than one EC2/vm ?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Sudarshan kumar
  • 1,503
  • 4
  • 36
  • 83

2 Answers2

1

Ok after looking into more details i found out that issue was in NACL which was blocking few subnets ip addresses .

So, I checked the MSK side Security group/Network ACL/Route table and found them to be fine. This means that the issue could be with the EC2 instance so I checked the Security group/Route table of the instance and found them to be properly configured.

However, on checking the Network ACL (acl-***) attached with the EC2 instance, I saw that there is an Inbound rule allowing 0.0.0.0/0 for the ephemeral ports which should allow the brokers to talk to the EC2 instance. However, looking at the Outbound rules, I saw it allowing only the subnet range where b-2 is present but it didn't have any explicit Outbound rule to allow either b-3 (10.**.**.0/24) or b-4 (10.**.**.0/24) subnet ranges.
When i added new rule then i was able to ping and connect success fully

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Sudarshan kumar
  • 1,503
  • 4
  • 36
  • 83
0

If i have to run Kafka connect in distributed mode,do i have to use more than one EC2/vm ?

No, you can run one "psuedo-distributed" instance. The main differerence between standalone and distributed is how it handles storage of offsets and configurations.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Hi MSK does give PLAINTEXT addresses .When we click on view client information in the MSK ui ..We can see that ..It might be earlier this option were not available .But i can see that now – Sudarshan kumar Jan 31 '20 at 05:22
  • Are you sure it isn't PLAINTEXT SASL? I'd be shocked if there wasn't some authentication layer. Did you add MSK and your EC2 box to the same VPC? – OneCricketeer Jan 31 '20 at 06:01
  • Yes i am sure ..I feel some connectivity issue on NACL level internal issue ..But thanks for explaining distributed mode ..First time i am trying out that so little confused .I have created EC2 in same subnet with all same setting and i am able to start the connector ..But Records are not going into Elastic search i am not getting any error also . – Sudarshan kumar Jan 31 '20 at 06:11
  • 1
    I get something like this `[2020-01-31 06:08:23,268] INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1125) [2020-01-31 06:08:23,330] INFO [Worker clientId=connect-1, groupId=connect-cluster] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1438)` – Sudarshan kumar Jan 31 '20 at 06:12
  • By default, connect reads the latest offsets of the topic,so send some data – OneCricketeer Jan 31 '20 at 06:22
  • Also, you need to HTTP post the Elasticsearch connector config, you can't use a properties file. Refer the REST API – OneCricketeer Jan 31 '20 at 06:23
  • Yes i am sending data and i can read from topic ..I did not get your point HHTP post the elastic search . – Sudarshan kumar Jan 31 '20 at 06:26
  • Sorry, 1) HTTP Port on `connection.url=https://****.amazonaws.com` 2) HTTP POST to add connectors with JSON, not the shown property file https://docs.confluent.io/current/connect/references/restapi.html – OneCricketeer Jan 31 '20 at 17:04