4

Running an application in in client mode, the driver logs are printed with the below info messages, any idea on how to resolve this? Any spark configs to be updated? or missing? 

[INFO ][dispatcher-event-loop-29][SparkRackResolver:54] Got an error when resolving hostNames. Falling back to /default-rack for all

The jobs runs fine, this msg is not in the executor logs.

Marc
  • 19,394
  • 6
  • 47
  • 51
Darklord
  • 61
  • 1
  • 5

2 Answers2

0

Check this bug:

https://issues.apache.org/jira/browse/SPARK-28005

If you want to suppress this in the logs you can try to add this into your log4j.properties

log4j.logger.org.apache.spark.deploy.yarn.SparkRackResolver=ERROR
Marek Rehora
  • 1
  • 1
0

This can happen while using spart-submit with master yarn in a deploy mode local (not using --deploy-mode cluster) and the path to topology.py script is not correct into your core-site.xml.

Path to core-site.xml can be set via environment variable HADOOP_CONF_DIR (or YARN_CONF_DIR).

Check the path in the param net.topology.script.file.name value of core-site.xml.

If the path is incorrect, deploying driver in local mode will lead to error of executing with the following warning:

23/01/15 18:39:43 WARN ScriptBasedMapping: Exception running /home/alexander/xxx/.conf/topology.py 10.15.21.199
java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py" (in directory "/home/john"): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
...
23/01/15 18:39:43 INFO SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
Alexander Grigorev
  • 11
  • 3