Questions tagged [cluster-mode]
22 questions
0
votes
2 answers
Spark Pod restarting every hour in Kubernetes
I have deployed spark applications in cluster-mode in kubernetes. The spark application pod is getting restarted almost every hour.
The driver log has this message before restart:
20/07/11 13:34:02 ERROR TaskSchedulerImpl: Lost executor 1 on…
Sprasad
- 25
- 7
0
votes
1 answer
Writing to a local FS in cluster mode SPARK
For spark jobs, we are trying to add a logging framework that creates a custom log file on a local FS.
In client mode, everything is fine, the files are created on the local FS with the user who launched the spark-submit.
However in cluster mode,…
Amir haroun
- 21
- 3
0
votes
1 answer
How can i use babel interpreter on pm2 cluster mode
I'm having an error about my ES codes because the babel-node doesn't work when I run the app in pm2 cluster mode, but in fork mode, it run successfully, it says here that --interpreter can't work in cluster mode (-i) due to its operating…
Dylan
- 1,121
- 1
- 13
- 28
0
votes
0 answers
Confluent-kafka (with kerberos) Error when spark-submit python job in cluster Mode
I am facing the following error while submitting a python job in a cluster mode:
appcache/application_1548793257188_803870/container_e80_1548793257188_803870_01_000001/environment/lib/python2.7/site-packages/confluent_kafka/init.py",
line 2, in
…
Elisabetta
- 328
- 3
- 9
0
votes
0 answers
Console output from worker nodes to a file in Spark cluster mode
I am running a pyspark script using spark-submit. The job runs succesfully.
Now I am trying to collect console output of this job to a file like below.
spark-submit in yarn-client mode
spark-submit --master yarn-client --num-executors 5…
nmr
- 605
- 6
- 20
0
votes
1 answer
Spark Cluster mode issue to read Hive-Hbase table on Kerberized Environment
Error description
We are not able execute our Spark job in yarn-cluster or yarn-client mode, though it is working fine in the local mode.
This issue occurs when we try to read the Hive-HBase tables in a Kerberized cluster.
What we have tried so…
Puneet Babbar
- 105
- 1
- 1
- 7
-3
votes
1 answer
PM2 stop in cluster mode
By mistake , I run the following command on the server
pm2start app.js -i 3500
Now on server multiple instances are creating in cluster mode. Can anyone suggest how can i stop this ? And how can i delete already instances ?
Thanks
