Stack Overflow
Stack Overflow

Questions tagged [dataflow]

Dataflow programming is a programming paradigm in which computations are modeled through directed graphs: nodes are instructions and data flows through the connections between them.

Dataflow programming is a programming paradigm which models programs as directed graphs and calculation proceeds in a way similar to electrical circuits. More precisely:

  • nodes are instructions that takes one or more inputs, perform calculation on them and present the result(s) as output;
  • edges connects inputs and outputs of the instructions -- this way the output of an instruction can be fed directly to the input on another node to trigger another calculation;
  • data "travels" using the directed edges and triggers the instructions as they pass through the nodes.

Often dataflow programming languages are visual, the most prominent example being LabView.

Resources

1152 questions
5
votes
1 answer

Apache Beam MinimalWordcount example with Dataflow Runner on eclipse

I am trying to run the MinimalWordCount example using the DataFlowRunner from Eclipse on windows using MinimalWordCount -->Run As Java Application from with in eclipse , its the same stock code from the example using my gcs bucket , however I…
user1965449
  • 2,849
  • 6
  • 34
  • 51
5
votes
1 answer

How can one create a data flow graph (DFG/SDFG) for any application from its source code

I have done a lot of research to figure out how a DFG can be created for an application from its source code. There are DFG's available online for certain applications such as MP3 Decoder, JPEG compression and H.263 Decoder. I haven't been able to…
a_sid
  • 577
  • 10
  • 28
5
votes
1 answer

Share state among operators in Flink

I wonder if it is possible in Flink to share the state among operators. Say, for instance, that I have partitioning by key on an operator and I need a piece of state of partition A inside partition C (for any reason) (fig 1.a), or I need the state…
affo
  • 453
  • 3
  • 15
5
votes
3 answers

Why do blocks run in this order?

This is short code sample to quickly introduce you what is my question about: using System; using System.Linq; using System.Threading.Tasks; using System.Threading.Tasks.Dataflow; namespace DataflowTest { class Program { static void…
kseen
  • 359
  • 8
  • 56
  • 104
5
votes
2 answers

Memory issue in TPL Dataflow implementation of IO read write operation

I have tried to implement Read write operation using File IO operations and encapsulated those operations into TransformBlock so as to make these operation thread safe instead of using locking mechanism. But the problem is that when I try to write…
Balraj Singh
  • 3,381
  • 6
  • 47
  • 82
5
votes
1 answer

Run Magento 1.9.1.0 Dataflow Import Profile Programmatically

I've tried to get this working, but it can't seem to find a solution. I'm looking to run an existing dataflow profile which has an ID = 3, and has the import file name already configured. All the research I've done, leads to some variation of the…
Alex
  • 153
  • 1
  • 6
5
votes
2 answers

TPL Dataflow block never completes on PropagateCompletion

Since the last alteration to my propagated-completion pipeline, one of my buffer blocks never completes. Let me summarize what was working and what isn't anymore: Previously working: A.LinkTo(B, PropagateCompletion); B.LinkTo(C,…
Luis Ferrao
  • 1,463
  • 2
  • 15
  • 30
5
votes
3 answers

TPL Dataflow: design for parallelism while keeping order

I have never worked with TPL before so I was wondering whether this can be done with it: My application creates a gif image animation file from a lot of frames. I start with a list of Bitmap which represents the frames of the gif file and need to do…
p4gefault
  • 63
  • 4
5
votes
3 answers

Confusion between Behavioural and Dataflow model Programs in VHDL

I'm using the textbook "VHDL: Programming By Example" by Douglas L Perry, Fourth Edition. He gave an example of the Dataflow programming model in page 4: Code I: ENTITY mux IS PORT ( a, b, c, d : IN BIT; s0, s1 : IN BIT; x, : OUT BIT); END…
Analon
  • 111
  • 1
  • 2
  • 8
5
votes
2 answers

Will TPL Dataflow work for this design type?

I'm currently looking into ways to simulate the transmission of resources and messages between connected objects such as power to devices or control messages over a networked system: CLICK FOR LARGER IMAGE. I've been recently looking into TPL…
S.Richmond
  • 11,412
  • 6
  • 39
  • 57
5
votes
3 answers

How to reconfigure the column information on a flat file connection manager?

I have a Flat File Source that is reading data from a flat file. We have recently added a new column to this flat file. The flat file data is inserted into a database table. To accommodate the new field in the destination component, I used the ALTER…
ETLUser
  • 371
  • 4
  • 7
  • 19
5
votes
3 answers

SSIS - fill unmapped columns in table in OLE DB Destination

As you can see in the image below, I have a table in SQL Server that I am filling via a flat file source. There are two columns in the destination table that I want to update based on the logic listed below: SessionID - all rows from the first CSV…
sion_corn
  • 3,043
  • 8
  • 39
  • 65
5
votes
1 answer

What exactly does "context" mean in context-(in)sensitive analysis?

This question was asked earlier but the answers are very generic. I am interested to know what context means in the context of static code analysis, specifically with Java and when used in conjunction with the term context (in)sensitive…
Jus12
  • 17,824
  • 28
  • 99
  • 157
5
votes
1 answer

How do I continue the process after inserting non-matching lookup rows into a table?

Inside a data flow, is it possible to do a lookup, insert the non-matching rows, re-lookup and then continue with the full (original) data set? I can't see how you get/re-use the inserted rows after they've gone to an OLE DB Destination, nor can I…
PeterX
  • 2,713
  • 3
  • 32
  • 42
4
votes
1 answer

Programming toolchain supporting "Incremental computing" (maybe even C/C++)

I'd like to write algorithms, that prepare results for big data sets. Than, when each dataset changes, incrementally update all affected outputs. It's called : Incremental computing. Are there programming tools, libraries, compiler, program analysis…
Grzegorz Wierzowiecki
  • 10,545
  • 9
  • 50
  • 88
1 2 3
…
76 77