0

I have configured the MSK cluster and allowed public access through SASL/SCRAM authentication method. Now I am facing the issue where I do not have the necessary permissions when using these credentials (specified in the Secrets Manager created with a custom key). The connecting client can perform certain operations (e.g. retrieve metadata) but fails to fetch or create topics or publish a new message to the existing topic. I am using Confluent as the library and here is a simplified example of the configuration that I am using (this is probably not relevant at all but is here to support an explanation of the issue).

BootstrapServers = Config.KafkaBootstrapServers,
SaslMechanism = SaslMechanism.ScramSha512, // only supported option by AWS
SecurityProtocol = SecurityProtocol.SaslSsl,
SaslUsername = Config.Username, // username from secrets manager
SaslPassword = Config.Password, // password from secrets manager
ClientId = Config.Client,
Acks = Acks.All

The error I get is Confluent.Kafka.Admin.CreateTopicsException: An error occurred creating topics: [topic]: [Authorization failed.]

How can I assign higher permissions? Since it is a managed Kafka service, there is no option to modify this on broker level directly. And since there is no user behind these credentials (since I am not using IAM auth method), I cannot assign a specific policy to it allowing certain operations like topic creation. What are the options here?

This page explains how ACLs are generally configured on Kafka but not on MSK. Am I missing something here?

hcerim
  • 959
  • 1
  • 11
  • 27
  • what error do you get? If you just enable authentication, you shouldn't have issues with ACLs. Please post an exact error you are getting (remove sensitive info though) – EdbE Feb 27 '23 at 23:58
  • From confluent I get Confluent.Kafka.Admin.CreateTopicsException: An error occurred creating topics: [topic]: [Authorization failed.]. – hcerim Feb 28 '23 at 06:59

2 Answers2

0

In MSK with SASL/SCRAM, authorizations are performed using ACLs. However, note that MSK sets "allow.everyone.if.no.acl.found" to true by default

From public docs:

This means that with Amazon MSK clusters, if you don't explicitly set ACLs on a resource, all principals can access this resource. If you enable ACLs on a resource, only the authorized principals can access it.

I suspect there is some existing ACL on these resources which is preventing you to produce/create topics. Would you be able to list ACLs for this cluster and verify this?

Sankalp
  • 75
  • 7
  • allow.everyone.if.no.acl.found must be false if you want to make the cluster publicly available which I did because I am accessing the service from outside of AWS. Also, it is not possible to configure ACLs on MSK as it is a managed service and you cannot directly access certain configurations (e.g. server.properties). So, their documentation is not really helpful as it explains how ACLs are managed on Kafka and not on MSK – hcerim Mar 01 '23 at 08:23
  • That's not true. You can set ACLs in MSK using the Kafka ACL APIs. Check this doc: https://docs.aws.amazon.com/msk/latest/developerguide/msk-acls.html – Sankalp Mar 09 '23 at 18:30
0

Kafka ACLs commands are working perfectly fine with MSK to control authorization policies on a cluster.

Define policies for the user that you are connecting with - Config.Username, with appropriate permissions.

# allow topic creation
kafka-acls zookeeper.connect=z-1:2181 \
--add --allow-principal User:THE_USER \
--operation DESCRIBE \
--operation CREATE \
--cluster


# allow read from and write to a topic
kafka-acls zookeeper.connect=z-1:2181 \
--add --allow-principal User:THE_USER \
--operation read --operation write --topic some-kafka-topic
EdbE
  • 204
  • 1
  • 4