Stack Overflow
Stack Overflow

Questions tagged [backpressure]

Backpressure refers to the build-up of data at an I/O switch when buffers are full and not able to receive additional data. No additional data packets are transferred until the bottleneck of data has been eliminated or the buffer has been emptied.

Backpressure refers to the build-up of data at an I/O switch when buffers are full and not able to receive additional data. No additional data packets are transferred until the bottleneck of data has been eliminated or the buffer has been emptied.

Source - Techopedia

164 questions
4
votes
4 answers

How to handle backpressure using google cloud functions

Using google cloud functions, is there a way to manage execution concurrency the way AWS Lambda is doing? (https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html) My intent is to design a function that consumes a file of tasks and…
simon
  • 559
  • 1
  • 6
  • 19
4
votes
1 answer

Does the C# Implementation of gRPC have streaming back-pressure?

I have a gRPC service that accepts streaming messages from a client. The client sends a finite sequence messages to the server at a high rate. The result is the server buffering a large number of messages (> 1GB) and it's memory usage skyrocketing…
AlgorithmsAreCool
  • 185
  • 2
  • 9
4
votes
0 answers

How to do akka-http request-level backpressure?

In akka-http, you can: Set akka.http.server.max-connections, which will prevent more than that number of connections. Exceeding this limit means clients will get connection timeouts. Set akka.http.server.pipelining-limit, which prevents a given…
randomstatistic
  • 800
  • 5
  • 11
4
votes
1 answer

How to create a Source which is not affected by backpressure

I want to test some Akka streams features like conflate. For this, I need to construct a source which is not affected by backpressure within a simple unit test. Naive attempts like Source.tick(1.milli, 1.milli, "tick").map(_ =>…
kostja
  • 60,521
  • 48
  • 179
  • 224
4
votes
2 answers

Ways to maintain back pressure in Akka Streams involving multiple JVMs

I'm aware that as of Akka 2.4.16, there is no "remote" implementation of Reactive Streams. The specification focuses on a stream running on a single JVM. However, considering the use case to involve another JVM for some processing while maintaining…
Toaditoad
  • 254
  • 2
  • 12
3
votes
3 answers

Flink co group outer join fails with High Backpressure

I have two streams in Flink stream1 has 70000 records per sec and stream2 may or may not have data. // Ingest the High Frequency Analog Stream SingleOutputStreamOperator stream1 = environment …
Neha Jirafe
  • 741
  • 5
  • 14
3
votes
1 answer

How to handle backpressure when using apache camel and Kafka?

I am trying to write an application that will integrate with Kafka using Camel. (Version - 3.4.2) I have an approach borrowed from the answer to this question. I have a route that listens for messages from a Kafka topic. The processing of this…
Raghu
  • 909
  • 7
  • 23
3
votes
1 answer

ReactiveX Backpressure doesn't work as expected

I am trying to make a flowable with a backpressure. My idea is that new item of the flowable won't be emitted until one of the current items finishes its processing. I am using a ResourceSubscriber and subscribeWith() method to achieve that. Each…
Konstantin Abakumov
  • 380
  • 2
  • 10
3
votes
1 answer

Why does `Publishers.Map` consume upstream values eagerly?

Suppose I have a custom subscriber that requests one value on subscription and then an additional value three seconds after it receives the previous value: class MySubscriber: Subscriber { typealias Input = Int typealias Failure = Never …
jjoelson
  • 5,771
  • 5
  • 31
  • 51
3
votes
1 answer

RxJava: how to process Flowable interactively from console

I've created a Flowable (RxJava v3) that parses a file. I'd like it to support backpressure. This is important because the files can be quite large, and I don't want them to be loaded into memory at once. Here is my first attempt: public…
Mister Smith
  • 27,417
  • 21
  • 110
  • 193
3
votes
1 answer

Akka Streams' ActorPublisher as a Source for web response - how back-pressure works

I use akka-streams' ActorPublisher actor as a streaming per-connection Source of data being sent to an incoming WebSocket or HTTP connection. ActorPublisher's contract is to regularly request data by supplying a demand - number of elements that can…
Tvaroh
  • 6,645
  • 4
  • 51
  • 55
3
votes
2 answers

RxJava onBackpressureBuffer not emitting items

I've witnessed a weird behavior with onBackpressureBuffer, I'm not sure if it is a valid behavior or a bug. I'm having a tcp call that is emitting items in a certain rate (using streaming and inputStream but that just for some info) On top of it…
ndori
  • 1,934
  • 1
  • 18
  • 23
3
votes
1 answer

Apache Flink: How does it handle the backpressure?

For an operator, the input stream is faster than its output stream, so its input buffer will block the previous operator's output thread that transfers the data to this operator. Right? Do the Flink and the Spark both handle the backpressure by…
jiecxy
  • 31
  • 1
  • 3
3
votes
1 answer

How does Backpressure happen internally in RxJava

Ive been reading a few docs for backpressure in RxJava, but i cant find a detailed explanation like how does it happen internally in the Library, everyone just summarize it like "producer" is too fast and "consumer" is too slow. For example like the…
Qing
  • 1,401
  • 2
  • 21
  • 41
3
votes
1 answer

How to process RxJS stream n items at a time and once an item is done, autofill back to n again?

I have a stream of events and I would like to call a function that returns a promise for each of those events, the problem is that this function is very expensive, so I would like to process at most n events at a time. This pebble diagram is…
martincastell
  • 177
  • 8
1 2
3
…
10 11