Stack Overflow
Stack Overflow

Questions tagged [dataflow]

Dataflow programming is a programming paradigm in which computations are modeled through directed graphs: nodes are instructions and data flows through the connections between them.

Dataflow programming is a programming paradigm which models programs as directed graphs and calculation proceeds in a way similar to electrical circuits. More precisely:

  • nodes are instructions that takes one or more inputs, perform calculation on them and present the result(s) as output;
  • edges connects inputs and outputs of the instructions -- this way the output of an instruction can be fed directly to the input on another node to trigger another calculation;
  • data "travels" using the directed edges and triggers the instructions as they pass through the nodes.

Often dataflow programming languages are visual, the most prominent example being LabView.

Resources

1152 questions
0
votes
1 answer

SSIS Lookup, create new records for no-match output and get newly generated keys

I have a Hotel table. create table DimHotel ( Hotel_ID int Identity(1, 1), HotelName nvarchar(max) ) Now i get some feeds as HotelNames, now our business logic is use Hotel_ID for existing HotelName and create new record for HotelName which…
Pritesh
  • 1,938
  • 7
  • 32
  • 46
0
votes
1 answer

Magento dataflow takes too long to load CSV file

I have a large CSV file containing Inventory data to update (more than 35,000 rows). I created a method which extends Mage_Catalog_Model_Convert_Adapter_Productimport to do the inventory update. Then I used an Advanced Profile to do the update which…
Alex Phelan
  • 1
  • 1
0
votes
1 answer

magento dataflow: how to call action once before processing any row

Context: Magento 1.7.0.0 version. I have to import csv data, with magento dataflow advanced profiles. I have made an Adapter that implements Mage_Dataflow_Model_Convert_Adapter_Abstract. I've implemented saveRow() method for processing each row,…
Katapofatico
  • 750
  • 1
  • 10
  • 29
0
votes
2 answers

How to create data flow diagrams using Java

I'm an engineering student; I studied dataflow graphs. I already made parser using Antlr, but I don't know how to make dataflow diagrams. I read some paper that one can make dataflow graphs using Java. Please help me.
user1378187
  • 31
  • 1
  • 5
-1
votes
2 answers

How to transform hh:mm:ss to total seconds in azure data factory

In Azure Data Factory, what can I do to transform "hh:mm:ss" to total seconds. In "Add dynamic content" part, I want to use "@activity('dataflow').Duration" in Azure Data Factory to get the duration of my current activity. However, I found that…
lalalalaalalallll
  • 1
  • 2
-1
votes
1 answer

Is there a way to draw a diagram that represents the relationship of variables in formula/conditional statement?

I have an algorithm like this : A = 0 if B == 1 or C == 1 else A = 1 D = A + E F = D/2 I'd like to represent relationships between variables as a dataflow or a tree diagram. Something like this I have a bunch of such formula which are to be updated…
JIHYUNG RYU
  • 1
  • 1
-1
votes
1 answer

Performing Left Join on two CSV files in Apache Beam Python

For my practice I tried joining two CSV in Apache Beam on a condition with no success. Also checked multiple posts, however nothing works. I have two CSV input file, tab separated, and I want to extract Column A, C from Primary file and Column A…
zigbee
  • 1
  • 2
-1
votes
1 answer

How to install Dataflow library in .Net v 4 project?

I need to install system.threading.task.dataflow library in a project targeting .netv 4 client profile. How can I install it in my project as I cannot update my project framework? Thanks in advance if anyone helps.
saba
  • 9
  • 3
-1
votes
1 answer

How to create multi zone(regional) dataflow using terraform?

dataflow list I want to change instance group location from asia-northeast1-a(zonal) to asia-norhteast1(regional) but dataflow job automatically select single zone. I wrote terraform code below as reference but maybe this is not a terraform issue…
hiko1129
  • 13
  • 2
-1
votes
1 answer

How to do streaming insert data from one bigquery table to another table after flattening the nested fields?

I have an intermediate table that holds send_timestamp and JSON data in its columns. I'm using dataflow to insert data from pubsub to this intermediate table. Now my use case is to authenticate the data and flatten the nested JSON in the…
Mohsin Mumtaz
  • 63
  • 1
  • 11
-1
votes
1 answer

Dependent data processing pipelines where files arrive asynchronously

I have several data dependent tasks/pipelines of which some depend on the completion of another. What makes it even harder is that the data can arrive asynchronously, meaning that certain tasks need to wait until all the files or tasks in the…
WJA
  • 6,676
  • 16
  • 85
  • 152
-1
votes
1 answer

Why my node returns me a websocket error?

I'm putting my application on an IIS HTTP server, but I get an error: WebSocket connection to 'wss://xxxx.xxxx.xx:xxxxx/?designer=1&token=&baa=' failed: Error during WebSocket handshake: net::ERR_CONNECTION_RESET. What could cause it?
Maxence.sailly
  • 11
  • 2
-1
votes
1 answer

How to create gcloud dataflow from linux console

I have already created gcloud dataflow on the Google cloud platform and now I need to do it from the Linux console. The main questions: How should it look? How can I work with --parameters of gcloud?
Alexander Antonenko
  • 1
  • 3
-1
votes
2 answers

Adding support for JS UDF in Google Dataflow template

I have this code from Google Cloud Platform Dataflow Templates. I wish to add more functionalities to it, namely, I wish to add support for JavaScript UDF. When I try to compile the file, using this: mvn compile exec:java…
tHappy
  • 91
  • 8
1 2 3
…
76
77