0

I have installed CDH 5.16 Express using packages in a RHEL server. I am trying to install Kafka now and i observed that it can be installed only if CDH is installed as parcels.

1) Is it possible to install Kafka or confluent platform" separately in the server and use it along withCDH` components.

2) Is there any other workaround to install Kafka using Cloudera Manager

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Vaishak
  • 607
  • 3
  • 8
  • 30

1 Answers1

1

In order use the CDK 4.0 (cloudera distribution of Kafka) with Cloudera 5.13, I was forced to install CDK 4.0 as a parcel.

I had a cloudera quickstart docker VM that I downloaded. It runs without Kerberos authentication. After starting the quickstart VM, I separately installed the quickstart Kafka from Apache kafka's website. This was required as the kafka packaged within cloudera was a older version. Since, this was non kerberos environment, the Kafka server upon startup started using the zookeeper that was running in quickstart VM. This way I achieved connection of Kafka with cloudera VM.

If you are new to CDH/CM then I suggest you first try and use the Kafka service that is bundled within Cloudera. Go to 'Add Service' within Cloudera drop-down and select kafka. Enabling this Kafka service will give you a set of brokers for kafka to run. Also, Kafka needs Zookeeper to run. Zookeeper comes by default in Cloudera. So, you would get a working cluster with kafka enabled in it. You can think of changing to the latest version of Kafka (using the approach mentioned above) once you are comfortable with inbuilt tools of CDH/CM.

Prashant
  • 702
  • 6
  • 21
  • my concern was whether Kafka installed separately(like you did) will be able to interact with CDH components such as HDFS or Spark). your experience shows that its possible..i will give it a try once...Did you install Kafka in your quickstart VM? – Vaishak May 28 '19 at 10:24
  • In a non-kerberos environment it works for sure as I have used Kafka + Hive + Spark2 and HDFS. So, it should be possible in an kerberos enabled environment as well. – Prashant May 28 '19 at 10:26
  • you mentioned you installed CDK 4.0 as a parcel. Did you install CDH using parcel as well? as per the documents, CDK can be installed only if CDH is also deployed through parcels. Unfortunately i followed package installation. – Vaishak May 28 '19 at 10:29
  • Yes, i removed the packaged installation and installed cloudera using parcel installation and then the CDK 3.1 parcel. It worked after that. – Prashant May 28 '19 at 10:31
  • i am yet to figure out how the entire process of CDH actualy works. If kafka installed separately was able to interact with CDH components, then adding kafka thorugh CM just allows you to control and monitor the kafka services? – Vaishak May 28 '19 at 11:00