I am trying to run a simple word counter example on a 3 node cluster, one with master-worker, and 2 being sole worker nodes. When I execute the "spark-submit CountWord.jar" in terminal I get the error saying TransportRequestHandler: Error...closing connection. I am putting my codes below along with the errors.
public static void main(String[] args) {
// TODO Auto-generated method stub
SparkConf sparkConf = new SparkConf().setMaster("spark://210.107.197.209:7077").setAppName("JD Word Counter")
.set("spark.shuffle.service.enabled", "false").set("spark.driver.blockManager.port", "10026")
.set("spark.driver.port", "10027").set("spark.cores.max", "3").set("spark.executor.memory", "800m")
.set("spark.driver.host", "210.107.197.209").set("spark.shuffle.service.enabled", "false")
.set("spark.dynamicAllocation.enabled", "false").set("spark.shuffle.blockTransferService", "nio");
;
try (JavaSparkContext jsc = new JavaSparkContext(sparkConf)) {
JavaRDD<String> inputFile = jsc.textFile("input.txt");
JavaRDD<String> wordsFromFile = inputFile.flatMap(content -> Arrays.asList(content.split(" ")).iterator());
JavaPairRDD countData = wordsFromFile.mapToPair(t -> new Tuple2(t, 1))
.reduceByKey((x, y) -> (int) x + (int) y);
int random = (int) (Math.random() * 100);
countData.saveAsTextFile("CountData/result_" + random);
}
}
Error!!
aavash@mainMaster:~/Desktop$ spark-submit CountWord.jar
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.2.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
22/02/22 17:58:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/02/22 17:58:25 INFO SparkContext: Running Spark version 3.2.0
22/02/22 17:58:25 INFO ResourceUtils: ==============================================================
22/02/22 17:58:25 INFO ResourceUtils: No custom resources configured for spark.driver.
22/02/22 17:58:25 INFO ResourceUtils: ==============================================================
22/02/22 17:58:25 INFO SparkContext: Submitted application: JD Word Counter
22/02/22 17:58:25 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 800, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/02/22 17:58:25 INFO ResourceProfile: Limiting resource is cpu
22/02/22 17:58:25 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/02/22 17:58:25 INFO SecurityManager: Changing view acls to: aavash
22/02/22 17:58:25 INFO SecurityManager: Changing modify acls to: aavash
22/02/22 17:58:25 INFO SecurityManager: Changing view acls groups to:
22/02/22 17:58:25 INFO SecurityManager: Changing modify acls groups to:
22/02/22 17:58:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(aavash); groups with view permissions: Set(); users with modify permissions: Set(aavash); groups with modify permissions: Set()
22/02/22 17:58:25 INFO Utils: Successfully started service 'sparkDriver' on port 10027.
22/02/22 17:58:25 INFO SparkEnv: Registering MapOutputTracker
22/02/22 17:58:25 INFO SparkEnv: Registering BlockManagerMaster
22/02/22 17:58:25 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/02/22 17:58:25 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/02/22 17:58:25 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/02/22 17:58:25 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-281e35c3-8e10-4e2c-8543-95545acc9c08
22/02/22 17:58:25 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
22/02/22 17:58:25 INFO SparkEnv: Registering OutputCommitCoordinator
22/02/22 17:58:25 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/02/22 17:58:25 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://210.107.197.209:4040
22/02/22 17:58:25 INFO SparkContext: Added JAR file:/home/aavash/Desktop/CountWord.jar at spark://210.107.197.209:10027/jars/CountWord.jar with timestamp 1645520305139
22/02/22 17:58:25 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://210.107.197.209:7077...
22/02/22 17:58:26 INFO TransportClientFactory: Successfully created connection to /210.107.197.209:7077 after 23 ms (0 ms spent in bootstraps)
22/02/22 17:58:26 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20220222175826-0005
22/02/22 17:58:26 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220222175826-0005/0 on worker-20220222170512-210.107.197.222-42867 (210.107.197.222:42867) with 1 core(s)
22/02/22 17:58:26 INFO StandaloneSchedulerBackend: Granted executor ID app-20220222175826-0005/0 on hostPort 210.107.197.222:42867 with 1 core(s), 800.0 MiB RAM
22/02/22 17:58:26 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220222175826-0005/1 on worker-20220222170512-210.107.197.226-35963 (210.107.197.226:35963) with 1 core(s)
22/02/22 17:58:26 INFO StandaloneSchedulerBackend: Granted executor ID app-20220222175826-0005/1 on hostPort 210.107.197.226:35963 with 1 core(s), 800.0 MiB RAM
22/02/22 17:58:26 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220222175826-0005/2 on worker-20220222163637-210.107.197.209-43757 (210.107.197.209:43757) with 1 core(s)
22/02/22 17:58:26 INFO StandaloneSchedulerBackend: Granted executor ID app-20220222175826-0005/2 on hostPort 210.107.197.209:43757 with 1 core(s), 800.0 MiB RAM
22/02/22 17:58:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 10026.
22/02/22 17:58:26 INFO NettyBlockTransferService: Server created on 210.107.197.209:10026
22/02/22 17:58:26 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/02/22 17:58:26 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 210.107.197.209, 10026, None)
22/02/22 17:58:26 INFO BlockManagerMasterEndpoint: Registering block manager 210.107.197.209:10026 with 434.4 MiB RAM, BlockManagerId(driver, 210.107.197.209, 10026, None)
22/02/22 17:58:26 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 210.107.197.209, 10026, None)
22/02/22 17:58:26 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220222175826-0005/1 is now RUNNING
22/02/22 17:58:26 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220222175826-0005/0 is now RUNNING
22/02/22 17:58:26 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 210.107.197.209, 10026, None)
22/02/22 17:58:26 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220222175826-0005/2 is now RUNNING
22/02/22 17:58:26 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
22/02/22 17:58:26 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 219.7 KiB, free 434.2 MiB)
22/02/22 17:58:26 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.1 KiB, free 434.2 MiB)
22/02/22 17:58:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 210.107.197.209:10026 (size: 32.1 KiB, free: 434.4 MiB)
22/02/22 17:58:26 INFO SparkContext: Created broadcast 0 from textFile at WordCounter.java:21
22/02/22 17:58:26 INFO FileInputFormat: Total input files to process : 1
22/02/22 17:58:27 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
22/02/22 17:58:27 INFO HadoopMapRedCommitProtocol: Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
22/02/22 17:58:27 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/02/22 17:58:27 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/02/22 17:58:27 INFO SparkContext: Starting job: runJob at SparkHadoopWriter.scala:83
22/02/22 17:58:27 INFO DAGScheduler: Registering RDD 3 (mapToPair at WordCounter.java:25) as input to shuffle 0
22/02/22 17:58:27 INFO DAGScheduler: Got job 0 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
22/02/22 17:58:27 INFO DAGScheduler: Final stage: ResultStage 1 (runJob at SparkHadoopWriter.scala:83)
22/02/22 17:58:27 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
22/02/22 17:58:27 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
22/02/22 17:58:27 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at WordCounter.java:25), which has no missing parents
22/02/22 17:58:27 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 7.6 KiB, free 434.1 MiB)
22/02/22 17:58:27 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 434.1 MiB)
22/02/22 17:58:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 210.107.197.209:10026 (size: 4.1 KiB, free: 434.4 MiB)
22/02/22 17:58:27 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1427
22/02/22 17:58:27 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at WordCounter.java:25) (first 15 tasks are for partitions Vector(0, 1))
22/02/22 17:58:27 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0
22/02/22 17:58:28 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (210.107.197.209:50864) with ID 2, ResourceProfileId 0
22/02/22 17:58:28 INFO BlockManagerMasterEndpoint: Registering block manager 210.107.197.209:44335 with 300.0 MiB RAM, BlockManagerId(2, 210.107.197.209, 44335, None)
22/02/22 17:58:28 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (210.107.197.222:47732) with ID 0, ResourceProfileId 0
22/02/22 17:58:28 INFO BlockManagerMasterEndpoint: Registering block manager 210.107.197.222:33217 with 300.0 MiB RAM, BlockManagerId(0, 210.107.197.222, 33217, None)
22/02/22 17:58:28 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (210.107.197.226:48806) with ID 1, ResourceProfileId 0
22/02/22 17:58:28 INFO BlockManagerMasterEndpoint: Registering block manager 210.107.197.226:35719 with 246.9 MiB RAM, BlockManagerId(1, 210.107.197.226, 35719, None)
22/02/22 17:58:28 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (210.107.197.209, executor 2, partition 0, PROCESS_LOCAL, 4497 bytes) taskResourceAssignments Map()
22/02/22 17:58:29 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 210.107.197.209:44335 (size: 4.1 KiB, free: 300.0 MiB)
22/02/22 17:58:29 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 210.107.197.209:44335 (size: 32.1 KiB, free: 300.0 MiB)
22/02/22 17:58:29 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (210.107.197.209, executor 2, partition 1, PROCESS_LOCAL, 4497 bytes) taskResourceAssignments Map()
22/02/22 17:58:29 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 619 ms on 210.107.197.209 (executor 2) (1/2)
22/02/22 17:58:29 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 50 ms on 210.107.197.209 (executor 2) (2/2)
22/02/22 17:58:29 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/02/22 17:58:29 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at WordCounter.java:25) finished in 2.220 s
22/02/22 17:58:29 INFO DAGScheduler: looking for newly runnable stages
22/02/22 17:58:29 INFO DAGScheduler: running: Set()
22/02/22 17:58:29 INFO DAGScheduler: waiting: Set(ResultStage 1)
22/02/22 17:58:29 INFO DAGScheduler: failed: Set()
22/02/22 17:58:29 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at saveAsTextFile at WordCounter.java:29), which has no missing parents
22/02/22 17:58:29 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 102.1 KiB, free 434.0 MiB)
22/02/22 17:58:29 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 36.9 KiB, free 434.0 MiB)
22/02/22 17:58:29 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 210.107.197.209:10026 (size: 36.9 KiB, free: 434.3 MiB)
22/02/22 17:58:29 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1427
22/02/22 17:58:29 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at saveAsTextFile at WordCounter.java:29) (first 15 tasks are for partitions Vector(0, 1))
22/02/22 17:58:29 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks resource profile 0
22/02/22 17:58:29 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2) (210.107.197.209, executor 2, partition 0, NODE_LOCAL, 4275 bytes) taskResourceAssignments Map()
22/02/22 17:58:29 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 210.107.197.209:44335 (size: 36.9 KiB, free: 299.9 MiB)
22/02/22 17:58:29 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 210.107.197.209:50864
22/02/22 17:58:29 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3) (210.107.197.209, executor 2, partition 1, NODE_LOCAL, 4275 bytes) taskResourceAssignments Map()
22/02/22 17:58:29 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 204 ms on 210.107.197.209 (executor 2) (1/2)
22/02/22 17:58:29 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 65 ms on 210.107.197.209 (executor 2) (2/2)
22/02/22 17:58:29 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
22/02/22 17:58:29 INFO DAGScheduler: ResultStage 1 (runJob at SparkHadoopWriter.scala:83) finished in 0.284 s
22/02/22 17:58:29 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/22 17:58:29 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished
22/02/22 17:58:29 INFO DAGScheduler: Job 0 finished: runJob at SparkHadoopWriter.scala:83, took 2.628934 s
22/02/22 17:58:29 INFO SparkHadoopWriter: Start to commit write Job job_202202221758275785522858291556929_0005.
22/02/22 17:58:29 INFO SparkHadoopWriter: Write Job job_202202221758275785522858291556929_0005 committed. Elapsed time: 37 ms.
22/02/22 17:58:29 INFO SparkUI: Stopped Spark web UI at http://210.107.197.209:4040
22/02/22 17:58:29 INFO StandaloneSchedulerBackend: Shutting down all executors
22/02/22 17:58:29 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
22/02/22 17:58:29 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/02/22 17:58:29 INFO MemoryStore: MemoryStore cleared
22/02/22 17:58:29 INFO BlockManager: BlockManager stopped
22/02/22 17:58:29 INFO BlockManagerMaster: BlockManagerMaster stopped
22/02/22 17:58:29 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/02/22 17:58:29 ERROR TransportRequestHandler: Error sending result StreamResponse[streamId=/jars/CountWord.jar,byteCount=87448104,body=FileSegmentManagedBuffer[file=/home/aavash/Desktop/CountWord.jar,offset=0,length=87448104]] to /210.107.197.222:47734; closing connection
io.netty.channel.StacklessClosedChannelException
at io.netty.channel.AbstractChannel.close(ChannelPromise)(Unknown Source)
22/02/22 17:58:29 ERROR TransportRequestHandler: Error sending result StreamResponse[streamId=/jars/CountWord.jar,byteCount=87448104,body=FileSegmentManagedBuffer[file=/home/aavash/Desktop/CountWord.jar,offset=0,length=87448104]] to /210.107.197.226:48810; closing connection
io.netty.channel.StacklessClosedChannelException
at io.netty.channel.AbstractChannel.close(ChannelPromise)(Unknown Source)
22/02/22 17:58:29 INFO SparkContext: Successfully stopped SparkContext
22/02/22 17:58:29 INFO ShutdownHookManager: Shutdown hook called
22/02/22 17:58:29 INFO ShutdownHookManager: Deleting directory /tmp/spark-74aef123-cbb8-47af-878e-e40461d760a2
22/02/22 17:58:29 INFO ShutdownHookManager: Deleting directory /tmp/spark-f45c9452-8fcb-4833-8203-05e3ad9011ef
