1

Everything works if run a simple job with kafka to kafka without kerberos security. I need do same but behind kerberos security. Take a look at my job code below:

job.name=Kafka2KafkaSimple
job.group=Kafka
job.description=This is a job that runs forever, copies an input Kafka topic to an output Kafka topic
job.lock.enabled=true

kafka.brokers=[host1]:9092

source.class=org.apache.gobblin.source.extractor.extract.kafka.UniversalKafkaSource
gobblin.source.kafka.extractorType=DESERIALIZER
kafka.deserializer.type=BYTE_ARRAY
extract.namespace=org.apache.gobblin.extract.kafka

bootstrap.with.offset=latest
topic.whitelist=logging_application_access

writer.builder.class=org.apache.gobblin.kafka.writer.KafkaDataWriterBuilder
writer.kafka.topic=logging_application_access
writer.kafka.producerConfig.bootstrap.servers=[host2]:9092
writer.kafka.producerConfig.value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
writer.output.format=TEXT

data.publisher.type=org.apache.gobblin.publisher.NoopPublisher

gobblin.yarn.keytab.file.path="/etc/security/keytabs/[secret].keytab"
gobblin.yarn.keytab.principal.name=[secret]

The output issue below:

2019-01-21 14:40:34 BRST WARN  [kafka-producer-network-thread | gobblin] org.apache.kafka.common.network.Selector  276 - Error in I/O with [hostname]/[ip]
java.io.EOFException
  at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62)
  at org.apache.kafka.common.network.Selector.poll(Selector.java:248)
  at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
  at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
  at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
  at java.lang.Thread.run(Thread.java:748)
alex
  • 12,464
  • 3
  • 46
  • 67
Bruno Wego
  • 2,099
  • 3
  • 21
  • 38
  • 1
    Curious why you'd use Gobblin between two Kafka Clusters instead of MirrorMaker? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_kafka-component-guide/content/running-mirrormaker-with-kerberos.html – OneCricketeer Jan 22 '19 at 05:50
  • We have Mirror Maker not, working, but for test purposes and compare, we wish run with Gobblin. – Bruno Wego Jan 22 '19 at 10:17
  • 1
    Might be worth opening a new question with what isn't working with it. From what I understand, Gobblin runs batches, not streaming between clusters. – OneCricketeer Jan 22 '19 at 16:34

0 Answers0