from single machine to parallel processing?

Question

i am new to Spark Streaming and Big Data in general. I am trying to understand the structure of a project in spark. I want to create a main class lets say "driver" with M machines, each machine keeps an array of counters and its values. In single machine and not in Spark, I would create a class for the machines and a class for the counters and i would do the computations that i want. But i am wondering if that is happening in Spark too. Would the same project but in Spark, have the structure I am quoting bellow?

class Driver {
    var num : Int = 100
    var machines: Array[Machine] = new Array[Machine](num)
    //split incoming dstream and fill machines' queues
}
class Machine {
 var counters = new Queue[(Int,Int)]() // counter with id 1 and value 25
 def fillCounters: Unit = { ... } //function to fill the queue counters
}

First of all, I'd suggest not to use term `machine` while dealing with Spark's ideoms. There is general term `node` which more clearly denotes any host in networking/distributed compation. Also, there are `driver` and `executor` for specific Spark roles. — gemelen, Jun 16 '18 at 09:42

score 2 · Accepted Answer · answered Jun 16 '18 at 09:56

In general, you could imagine Spark application as a driver part of application which runs all coordination tasks, constructs graph (you will find mentions of direct acyclic graph or DAG in theoretical parts of tutorials on Spark and distributed computations) of operations to take place over your data, and executor part which results in many copies of the code, sent to each node of the cluster to run over the data.

Main idea is that driver extracts part of your application's code that needs to be run locally with data on nodes, serializes it, sends over network to each executor, launches, manages and collects results.

Spark framework hides this details for simplicity of usage, so applications being developed and looks like single-threaded application.

Developer could separate contexts that run on driver and executors, but this is not very common for tutorials (again, for simplicity).

So, to the answer for the actual question above: you do not need to design your application in a way you demonstrated above, unless your certainly want to. Just follow official Spark tutorial to get viable solution and split it afterwards with contexts of execution.

There is good post, summarizing a lot of Spark turorials, videos and talks - you could find it here at SO.

So in my case do you think that I should split the data on the driver class into nodes/executors, where they are going to make the computations I need (take averages etc) and when that happens I will return them to the driver. Maybe to use some broadcast variables? Can I use one executor for every "Machine" where each machine keeps its data. And if so, what action does that? Parallelize? — Suzy Tros, Jun 16 '18 at 11:44
@SuzyTros I suggest to start with application made in generic way, like one you find in Spark tutorials. In other words application that has one main class, which creates Spark contexts, launches data ingestion and transforms data in your desired way. — gemelen, Jun 16 '18 at 12:38

from single machine to parallel processing?

1 Answers1