I am trying to add a Kinesis connector to connect my Kinesis Stream with my AWS MSK Cluster, so that I can send messages from the Kinesis to a certain topic in the Kafka cluster. This is the connector I am trying to add: https://www.confluent.io/hub/confluentinc/kafka-connect-kinesis
The problem is that the creation fails, and when checking in the logs, I see this message:
[Worker-09c2e6449b0bd54f5] [2023-08-03 09:30:01,966] WARN [msk-kinesis-connector|worker] [AdminClient clientId=msk-kinesis-connector-license-manager] Error connecting to node <bootstrap server URL>:9098 (id: -2 rack: null) (org.apache.kafka.clients.NetworkClient:1011)
[Worker-09c2e6449b0bd54f5] java.net.UnknownHostException: <bootstrap server URL>
The cluster has both SASL/SCRAM and IAM enabled, so I am trying to access via IAM, with just Private Endpoint (single-VPC), it has no public endpoint.
The connector is in the same subnet and has the same security groups as the Cluster, and they allow themselves as inbound traffic, so I think that there should be no problem with connectivity.
In the connector configuration, I tried with multiple options, right now I have the following:
connector.class=io.confluent.connect.kinesis.KinesisSourceConnector
producer.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
consumer.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
confluent.topic.bootstrap.servers=<bootstrap server URL>:9098,<bootstrap server URL>:9098,<bootstrap server URL>:9098
producer.security.protocol=SASL_SSL
tasks.max=1
kinesis.region=eu-central-1
consumer.sasl.mechanism=AWS_MSK_IAM
kinesis.stream=kinesis_stream_name
confluent.license=
producer.sasl.mechanism=AWS_MSK_IAM
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
confluent.topic.replication.factor=3
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
name=msk-kinesis-connector
kafka.topic=testing
consumer.security.protocol=SASL_SSL
I got some of them from https://docs.confluent.io/kafka-connectors/kinesis/current/kinesis_source_connector_config.html and I think I am not missing anything related to connectivity
For the VPC ACL, I have the following:
10 SSH (22) TCP (6) 22 0.0.0.0/0 Deny
11 RDP (3389) TCP (6) 3389 0.0.0.0/0 Deny
100 All traffic All All 0.0.0.0/0 Allow
* All traffic All All 0.0.0.0/0 Deny
And in one of the security groups, all inbound traffic from itself is allowed, and also all outbund traffic.
I also tried to add all the msk policies to the role, just in case something was missing, but no luck:
{
"Statement": [
{
"Action": [
"kafkaconnect:*",
"kafka-cluster:*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:kafkaconnect:*:<accountId>:worker-configuration/*/*",
"arn:aws:kafkaconnect:*:<accountId>:custom-plugin/*/*",
"arn:aws:kafkaconnect:*:<accountId>:connector/*/*",
"arn:aws:kafka:*:<accountId>:topic/*/*/*",
"arn:aws:kafka:*:<accountId>:cluster/*/*",
"arn:aws:kafka:*:<accountId>:group/*/*/*",
"arn:aws:kafka:*:<accountId>:transactional-id/*/*/*"
]
},
{
"Action": [
"kafkaconnect:ListConnectors",
"kafkaconnect:CreateCustomPlugin",
"kafkaconnect:CreateConnector",
"kafkaconnect:DeleteConnector",
"kafkaconnect:ListCustomPlugins",
"kafkaconnect:DeleteCustomPlugin",
"kafkaconnect:CreateWorkerConfiguration",
"kafkaconnect:UpdateConnector",
"kafkaconnect:ListWorkerConfigurations"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": [
"kafka:ListClustersV2",
"kafka:ListVpcConnections",
"kafka:DescribeClusterOperation",
"kafka:GetCompatibleKafkaVersions",
"kafka:ListClusters",
"kafka:ListKafkaVersions",
"kafka:GetBootstrapBrokers",
"kafka:ListConfigurations",
"kafka:DescribeClusterOperationV2"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": "kafka:*",
"Effect": "Allow",
"Resource": [
"arn:aws:kafka:*:<accountId>:vpc-connection/*/*/*",
"arn:aws:kafka:*:<accountId>:cluster/*/*",
"arn:aws:kafka:*:<accountId>:configuration/*/*"
]
}
],
"Version": "2012-10-17"
}
Any idea on why it is not reaching the MSK cluster? I thought that setting it on the same VPC with the same SG would work.
