Stack Overflow
Stack Overflow

Questions tagged [executors]

Use for questions related with machines playing the role of an executors, for example inside a cluster in a distributed computing environment.

The notion of an executor is:

A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.

taken from Apache-Spark's webpage, which is a popular framework in .

110 questions
150
votes
6 answers

How to get thread id from a thread pool?

I have a fixed thread pool that I submit tasks to (limited to 5 threads). How can I find out which one of those 5 threads executes my task (something like "thread #3 of 5 is doing this task")? ExecutorService taskExecutor =…
serg
  • 109,619
  • 77
  • 317
  • 330
35
votes
6 answers

Java Executor with throttling/throughput control

I'm looking for a Java Executor that allows me to specify throttling/throughput/pacing limitations, for example, no more than say 100 tasks can be processed in a second -- if more tasks get submitted they should get queued and executed later. The…
mrip
  • 14,913
  • 4
  • 40
  • 58
18
votes
2 answers

How to avoid Spark executor from getting lost and yarn container killing it due to memory limit?

I have the following code which fires hiveContext.sql() most of the time. My task is I want to create few tables and insert values into after processing for all hive table partition. So I first fire show partitions and using its output in a…
Umesh K
  • 13,436
  • 25
  • 87
  • 129
12
votes
1 answer

Abort countDownLatch.await() after time out

I am using an ExecutorService to implement a 3-thread pool, and CountDownLatch to monitor the completion of all threads, for further processing. ExecutorService threadExecutor = Executors.newFixedThreadPool(3); CountDownLatch countDownLatch = new…
user2122524
  • 516
  • 1
  • 7
  • 9
10
votes
7 answers

Java concurrency: executing many "infinite" tasks with few threads

I'm building a (concurrent) simulator for a set of N particles that are moving in a space according to the Newton's laws. My idea is model each particle as a task, which interacts with other particles (tasks) in order to get their positions and…
metaphori
  • 2,681
  • 1
  • 21
  • 32
10
votes
3 answers

How to cancel pending items from a ScheduledThreadPoolExecutor?

I have a task which requires me to schedule tasks and remove them when a certain event occurs. I'm using a ScheduledThreadPoolExecutor to schedule the tasks, that's pretty straightforward. But, I found two ways to cancel the pending items, both of…
Tamas Rev
  • 7,008
  • 5
  • 32
  • 49
9
votes
2 answers

ScheduledExecutorService, how to stop action without stopping executor?

I have this code: ScheduledExecutorService scheduledExecutor; ..... ScheduledFuture result = scheduledExecutor.scheduleWithFixedDelay( new SomethingDoer(),0, measurmentPeriodMillis, TimeUnit.MILLISECONDS); After some event I should stop…
user2541398
  • 113
  • 1
  • 1
  • 6
8
votes
1 answer

What is the executor pattern in a C++ context?

The author of asio, Christopher Kohlhoff, is working on a library and proposal for executors in C++. His work so far includes this repo and docs. Unfortunately, the rationale portion has yet to be written. So far, the docs give a few examples of…
Praxeolitic
  • 22,455
  • 16
  • 75
  • 126
8
votes
2 answers

spark executor out of memory in join and reduceByKey

In spark2.0, I have two dataframes and I need to first join them and do a reduceByKey to aggregate the data. I always got OOM in executor. Thanks in advance. Data d1 (1G, 500 million rows, cached, partitioned by col id2) id1 id2 1 1 1 3 1 4 2 …
xuan
  • 270
  • 1
  • 2
  • 15
8
votes
3 answers

Spark - How many Executors and Cores are allocated to my spark job

Spark architecture is entirely revolves around the concept of executors and cores. I would like to see practically how many executors and cores running for my spark application running in a cluster. I was trying to use below snippet in my…
Krishna Reddy
  • 1,069
  • 5
  • 12
  • 18
7
votes
3 answers

Producer Consumer - Using Executors.newFixedThreadPool

My understanding of a Producer-Consumer pattern is that it could be implemented using a queue shared between the producer and the consumer. Producer submits work to a shared queue, consumer retrieves it and processes it. It could also be implemented…
Oxford
  • 139
  • 1
  • 2
  • 5
7
votes
3 answers

java thread reusage via executor

I am confused on the following: To use threads in a Java program, the simplest way is to extend Thread class and implement the runnable interface (or simply implement runnable). To start the thread's execution. we must call the Thread's method…
Cratylus
  • 52,998
  • 69
  • 209
  • 339
7
votes
3 answers

Is there a way to put tasks back in the executor queue

I have a series of tasks (i.e. Runnables) to be executed by an Executor. Each task requires a certain condition to be valid in order to proceed. I would be interested to know if there is a way to somehow configure Executor to move tasks in the end…
Cratylus
  • 52,998
  • 69
  • 209
  • 339
5
votes
4 answers

How to get to FutureTask execution state?

I have a singleThreadExecutor in order to execute the tasks I submit to it in serial order i.e. one task after another, no parallel execution. I have runnable which goes something like this MyRunnable implements Runnable { @Override public void…
Svilen
  • 1,377
  • 1
  • 16
  • 23
5
votes
2 answers

Programmatically add/remove executors to a Spark Session

I'm looking for a reliable way in Spark (v2+) to programmatically adjust the number of executors in a session. I know about dynamic allocation and the ability to configure spark executors on creation of a session (e.g. with --num-executors), but…
Will Boulter
  • 66
  • 1
  • 4
1
2 3 4 5 6 7 8