0

Please help how to pass --props file and --source-class file to LIVY API POST .

spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.5.3,org.apache.spark:spark-avro_2.11:2.4.4 \
 --master yarn \
 --deploy-mode cluster \
 --conf spark.sql.shuffle.partitions=100 \
 --driver-class-path $HADOOP_CONF_DIR \
 --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer \
 --table-type MERGE_ON_READ \
 --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
 --source-ordering-field tst  \
 --target-base-path /user/hive/warehouse/stock_ticks_mor \
 --target-table test \
 --props /var/demo/config/kafka-source.properties \
 --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
 --continuous
Amila Senadheera
  • 12,229
  • 15
  • 27
  • 43
codek
  • 65
  • 1
  • 6
  • please check this. https://stackoverflow.com/questions/68021329/need-help-on-submitting-hudi-delta-streamer-job-via-apache-livy. – shiva nagesh Oct 07 '21 at 15:25
  • --props and --source-class cannot be passed through args..It needs --props and --source-class as well. Help me on this. – codek Oct 08 '21 at 09:13

1 Answers1

0

I have converted the configs you are using in a json file to be passed to LIVY API

{
  "className": "org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer",
  "proxyUser": "root",
  "driverCores": 1,
  "executorCores": 2,
  "executorMemory": "1G",
  "numExecutors": 4,
  "queue": "default",
  "name": "stock_ticks_mor",
  "file": "hdfs://tmp/hudi-utilities-bundle_2.12-0.8.0.jar",
  "conf": {
    "spark.sql.shuffle.partitions": "100",
    "spark.jars.packages": "org.apache.hudi:hudi-spark-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:3.0.2",
    "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
    "spark.task.cpus": "1",
    "spark.executor.cores": "1"
  },
  "args": [
    "--props","/var/demo/config/kafka-source.properties",
    "--table-type","MERGE_ON_READ",
    "--source-class", "org.apache.hudi.utilities.sources.JsonKafkaSource",
    "--target-base-path","/user/hive/warehouse/stock_ticks_mor",
    "--target-table","test",
    "--schemaprovider-class","org.apache.hudi.utilities.schema.FilebasedSchemaProvider",
    "--continuous"
  ]
}

You can submit this json to the LIVY endpoint like

curl -H "X-Requested-By: admin" -H "Content-Type: application/json" -X POST -d @config.json http://localhost:8999/batches

For reference : https://community.cloudera.com/t5/Community-Articles/How-to-Submit-Spark-Application-through-Livy-REST-API/ta-p/247502

Vinay Patil
  • 105
  • 2
  • 10