Stack Overflow
Stack Overflow

Questions tagged [spark-operator]

39 questions
2
votes
1 answer

Helm install spark-operator tries to download a version that does not exist and cannot be force to use the correct one

I am trying to install ANY working version of spark-operator https://console.cloud.google.com/gcr/images/spark-operator/GLOBAL/spark-operator?tag=v1beta2-1.3.1-3.1.1 on my local kubernetes. However, spark pod is stuck on ImagePullBackOff trying to…
Tomáš Sedloň
  • 153
  • 1
  • 9
2
votes
2 answers

Argo Workflow + Spark operator + App logs not generated

Am in very early stages of exploring Argo with Spark operator to run Spark samples on the minikube setup on my EC2 instance. Following are the resources details, not sure why am not able to see the spark app logs. WORKFLOW.YAML kind:…
Gurupraveen
  • 181
  • 1
  • 13
2
votes
1 answer

spark-operator logging in k8s

I am using spark-operator for Kubernetes. https://github.com/GoogleCloudPlatform/spark-on-k8s-operator I am able to run the jobs successfully but there is a lot of trouble for monitoring and troubleshooting the job since the pods are dynamic. I want…
Sumit G
  • 436
  • 8
  • 21
1
vote
0 answers

spark streaming from kafka on spark operator(Kubernetes)

I have a spark structured streaming job in scala, reading from kafka and writing to S3 as hudi tables. Now I am trying to move this job to spark operator on EKS. When I give the option in the yaml file. spark.jars.packages:…
haripriya rajendran
  • 11
  • 2
1
vote
0 answers

JMX exporter & spark-on-k8s-operator

I'm trying to submit a spark application using spark operator and to expose metrics using JMX exporter. I'm using Spark 3.1.1 & spark operator v1beta2-1.3.3-3.1.1 Here is a snippet from the configuration. monitoring: exposeDriverMetrics: true …
Alex L
  • 36
  • 5
1
vote
2 answers

How to deploy a Beam Application on already deployed Spark application on Kubernetes?

I have already deployed Spark on Kubernetes, below is the deployment.yaml, apiVersion: "sparkoperator.k8s.io/v1beta2" kind: SparkApplication metadata: name: pyspark-pi namespace: default spec: type: Python pythonVersion: "3" mode: cluster …
user19930511
  • 299
  • 2
  • 15
1
vote
0 answers

Failed to connect to spark-master:7077

I am trying to deploy my spark application on Kubernetes. I followed the below steps: Installed spark-kubernetes-operator: helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator helm install gcp-spark-operator…
user19930511
  • 299
  • 2
  • 15
1
vote
0 answers

Spark Streaming on Kubernetes - Executor Pods not Restarted/Rescheduled

We are running DStream applications on Kubernetes cluster using Spark Operator (Spark 2.4.7). Sometimes due to various reasons (OOM's, Kubernetes node restarts) executor pods are getting lost, and while many times Spark sees this and reschedules a…
eugen-fried
  • 2,111
  • 3
  • 27
  • 48
1
vote
1 answer

aws libraries compatible with spark 3.1.1

I have a spark operator with sparkVersion: "3.1.1" and would like to use it for structured streaming to/from minIO. However, I have not been able to find a compatible combination of libraries for anything newer than hadoop 2.7.0. (which does not…
Tomáš Sedloň
  • 153
  • 1
  • 9
1
vote
0 answers

spark streaming - waiting for a dead executor

I have a spark streaming application running inside a k8s cluster (using spark-operator). I have 1 executor, reading batches every 5s from a Kinesis stream. The Kinesis stream has 12 shards, and the executor creates 1 receiver per shard. I gave it…
Malki
  • 2,335
  • 8
  • 31
  • 61
1
vote
1 answer

executor.CoarseGrainedExecutorBackend (Logging.scala:logInfo(57)) - Driver commanded a shutdown - how I can debug on driver side?

I'm getting that logs from the executor (beginning at the buttom): 2021-11-30 21:44:42 2021-11-30 18:44:42,911 INFO [shutdown-hook-0] util.ShutdownHookManager (Logging.scala:logInfo(57)) - Deleting directory…
Eljah
  • 4,188
  • 4
  • 41
  • 85
1
vote
1 answer

Spark-operator on EKS Apache spark failed to create temp directory

I am trying to deploy simple spark-pi.yaml onto AWS EKS using spark-operator. I have successfully deployed spark-operator Refer deployment YAML here spark-operator example I am getting the following error when I do helm install Events: Type …
Anand Jain
  • 603
  • 7
  • 20
1
vote
0 answers

Spark on K8s: UnknownHostException when spark app is trying to resolve DNS (getting) of another pod in diffrent namespace on same cluster

I am able to execute SparkPi in k8s and deployed (in GKE) as well. But, when I am trying to broadcast PI value to my microservice which is in toys-broadcast-svc.toys.svc.cluster.local I am unable to resolve DNS (getting UnknownHostException) . Can…
Jacksquad
  • 23
  • 6
1
vote
0 answers

Kubernets Spark Operator: Configuring Automatic Application Restart and Failure Handling

On kubernetes a Container restart policy can be configured with an optional field .spec.restartPolicy which can be set to type: OnFailure. I read here that there is a cap of 300 secs (5 mins) on an exponential back-off delay before a failed pod is…
Azeem
  • 33
  • 7
1
vote
0 answers

Spark on Kubernetes troubleshooting

I have a kubernetes cluster where I try to run a spark example application (spark-pi). I have setup a service account = spark Scenario When I do a spark-submit from the command line like below, I am getting a container error. bin/spark-submit…
user3370144
  • 99
  • 1
  • 5
1
2 3