0

I'm building Spark 3.2.0 against Hadoop 2.7 but failed.

$ git clone -b v3.2.0 https://github.com/apache/spark
$ mv spark spark-3.2.0
$ nohup sh -x dev/make-distribution.sh  --tgz -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.2 -Phive -Phive-thriftserver -DskipTests > build.log 2>&1 &

errors

[ERROR] [Error] /home/myuser/spark-3.2.0/core/src/test/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManagerSuite.scala:145: method reset in class UserGroupInformation cannot be accessed in object org.apache.hadoop.security.UserGroupInformation

...

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile (scala-test-compile-first) on project spark-core_2.12: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile failed.: CompileFailed -> [Help 1]

enter image description here

AppleCEO
  • 63
  • 7
  • What is wrong with the dropdown "Pre built for Hadoop 2.7" ? https://spark.apache.org/downloads.html – OneCricketeer Jan 24 '22 at 16:33
  • I use Pre-built user-provied , but when I execute： ./bin/pyspark Python 2.7.5 (default, Apr 2 2020, 13:16:51) ... print("Could not find valid SPARK_HOME while searching {0}".format(paths), file=sys.stderr) ^ SyntaxError: invalid syntax >>> seems python version not compatible, my version is 2.7.5 – AppleCEO Jan 25 '22 at 13:06
  • many unexpected errors occur when i use the pre-built version because i need it to be compatible with my present env. so i still need to built it myself – AppleCEO Jan 25 '22 at 14:11
  • You definitely should not be using Python2 anymore. Your error seems to be related to the fact that Python2 support was dropped (from everything, not only Spark), not with running Spark. In other words, run `spark-shell` if you are trying to write Scala code. Otherwise just do `python3 -m pip install pyspark` – OneCricketeer Jan 25 '22 at 14:54
  • 1
    i installed python3, the error above disappear but other strange errors occur, so i think i need to back to the Building error – AppleCEO Jan 25 '22 at 14:58
  • after installing python3 and pyspark, RUN `./bin/pyspark --master yarn ` is ok but RUN `./bin/spark-submit --master yarn wordcount.py`, error `pyspark.sql.utils.IllegalArgumentException: ` – AppleCEO Jan 25 '22 at 15:20
  • I'm just trying to undertand why you think building will solve any problem here (all these scripts are going to work the same). Does `--master 'local[*]'` work? Otherwise, you will need to show your wordcount file – OneCricketeer Jan 25 '22 at 15:40
  • Thanks for all your helps. I give up building it myself and using the prebuilt version @OneCricketeer – AppleCEO Feb 05 '22 at 12:26

0 Answers0