how to submit a pyspark job in google cloud shell; how to pass files and arguments in pyspark submit; how to read them in pyspark code

Question

Can anyone please help how to

1. submit a pyspark job in google cloud shell
2. pass files and arguments in pyspark submit
3. read those files and arguments in pyspark code

Answer 1

1. To submit a job to a Dataproc cluster, run the gcloud CLI gcloud dataproc jobs submit command locally in a terminal window or in Cloud Shell. Here is the detailed official documentation.

2. With the help of spark-submit you can pass program arguments you'll see that spark-submit has following syntax:

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

You can use either application-arguments and conf to pass required configuration to the main method and SparkConf respectively.Here is the documentation.

3.Some information on reading files with Spark:

1. If using a path on the local filesystem, the file must also be accessible at the same path on worker nodes. Either copy the file to all workers or use a network-mounted shared file system.

2. All of Spark’s file-based input methods, including textFile, support running on directories, compressed files, and wildcards as well. For example, you can use textFile("/my/directory"), textFile("/my/directory/.txt"), and textFile("/my/directory/.gz"). T

3. he textFile method also takes an optional second argument for controlling the number of partitions of the file. By default, Spark creates one partition for each block of the file (blocks being 64MB by default in HDFS), but you can also ask for a higher number of partitions by passing a larger value. Note that you cannot have fewer partitions than blocks.Here is blog explaining it.

Here is the guide for all processes.