1

I am trying to write a Spark dataframe to Kudu DB, but I do not know the Kudu master. The cluster I am using is a Cloudera cluster.

How do I find Kudu master in the cluster?

tk421
  • 5,775
  • 6
  • 23
  • 34
Karthik reddy
  • 79
  • 9
  • Did you look at https://stackoverflow.com/questions/45109355/how-to-get-current-kudu-master-or-tserver-flag-value? – tk421 Sep 14 '18 at 03:16

3 Answers3

0

Here is a very basic example using Cloudera Manager Java client (https://cloudera.github.io/cm_api/docs/java-client-swagger/)

package cloudera.kudu_example;

import java.io.IOException;

import com.cloudera.api.swagger.HostsResourceApi;
import com.cloudera.api.swagger.ServicesResourceApi;
import com.cloudera.api.swagger.client.ApiClient;
import com.cloudera.api.swagger.client.ApiException;
import com.cloudera.api.swagger.client.Configuration;
import com.cloudera.api.swagger.model.ApiHost;
import com.cloudera.api.swagger.model.ApiRole;
import com.cloudera.api.swagger.model.ApiRoleList;

public class App {
    public static void main( String[] args ) throws IOException {
        ApiClient cmClient = Configuration.getDefaultApiClient();

        cmClient.setBasePath(args[0]);
        cmClient.setUsername(args[1]);
        cmClient.setPassword(args[2]);

        cmClient.setVerifyingSsl(false);
        HostsResourceApi hostsApiInstance = new HostsResourceApi();
        ServicesResourceApi servicesApiInstance = new ServicesResourceApi();
        try {
            ApiRoleList apiRoles = servicesApiInstance.readRoles("Cluster 1", "KUDU-1");
            for(ApiRole role : apiRoles.getItems()) {
                if(role.getType().equalsIgnoreCase("KUDU_MASTER")) {
                    ApiHost host = hostsApiInstance.readHost(role.getHostRef().getHostId(), "full");
                    System.out.printf("Kudu master runs at %s. IP: %s, status %s", host.getHostname(), host.getIpAddress(), host.getEntityStatus());
                }
            }

        } catch (ApiException e) {
          System.err.println("Exception when calling ClustersResourceApi#readClusters");
          e.printStackTrace();
        }
    }
}
Greg S
  • 466
  • 3
  • 5
0

Here is python example using Python client v3 (https://cloudera.github.io/cm_api/docs/python-client-swagger/):

#!/usr/local/bin/python
import cm_client

# Configure HTTP basic authorization: basic
#configuration = cm_client.Configuration()
cm_client.configuration.username = 'admin'
cm_client.configuration.password = 'admin'

# Create an instance of the API class
api_client = cm_client.ApiClient("http://your-cdh-cluster-cm-host:7180/api/v30")

# create an instance of the ServicesResourceApi class
service_api_instance = cm_client.ServicesResourceApi(api_client)

# create an instance of the HostsResourceApi class
host_api_instance = cm_client.HostsResourceApi(api_client)

# find KUDU_MASTER roles in the CDH cluster
cluster_roles = service_api_instance.read_roles("Cluster 1", "KUDU-1")
for role in cluster_roles.items:
  if role.type == "KUDU_MASTER":
    role_host = host_api_instance.read_host(role.host_ref.host_id, view="full")
    print("Kudu master is located on %s\n" % role_host.hostname)
Greg S
  • 466
  • 3
  • 5
0

I know this is not a best way but it is one quick way to do. Let us assume we already have a kudu table with you(In case even if you don't create a test/temporary table through impala ), just do a describe formatted to that table. You will be getting bunch of details including kudu master details(host name), where port will be 8051. I believe once you know the host and port details you can explore many for your spark dataframe.

Temp table syntax:

CREATE TABLE kudu_no_partition_by_clause ( id bigint PRIMARY KEY, s STRING, b BOOLEAN ) STORED AS KUDU;

Syntax for describe: Describe formatted table_name;

FYR:

Kudu web administration details: https://kudu.apache.org/releases/0.6.0/docs/administration.html

Kudu with spark examples: https://kudu.apache.org/docs/developing.html

Cheers!!

Tony79
  • 21
  • 1
  • 3