Stack Overflow
Stack Overflow

Questions tagged [incremental-load]

33 questions
0
votes
1 answer

Handling Null Values in ADF Pipeline

I am performing an incremental load using timestamp as watermark column. I have few null values in the date column in my source. When I am replicating the data using copy activity, only rows whose date column is not null are getting copied, but I…
gmp
  • 1
  • 3
0
votes
2 answers

Incremental load in Azure Data Factory

I am replicating my data from Azure SQl DB TO Azure SQL DB. I have some tables with date columns and some tables with just the ID columns which are assigning primary key. While performing incremental load in ADF, I can select date as watermark…
gmp
  • 1
  • 3
0
votes
1 answer

Azure Delta Load won't recognize Epoch timestamp (ms) as Watermark Column Name

I am trying to perform a delta load (incremental load) with Azure Data Factory from SQL Server to Blob Storage. My tables have an updateStamp column that is Epoch Time in milliseconds, numeric(19,0) data type. When I look to select the Watermark…
TomH
  • 3
  • 1
0
votes
1 answer

Displaying images in gridview using paging

I have a gridview with incremental loading that displays book covers from a local folder. XAML:
Rose
  • 613
  • 4
  • 22
0
votes
0 answers

Incremental load in SSIS

I have a source table that has some millions of rows and the job runs the SSIS package to load the data daily. Now the daily job is taking for hours to run the entire load due to several rows. I want to just do the incremental logic on the SSIS…
veda
  • 33
  • 3
0
votes
1 answer

Azure Synapse - Pipelines - Copy Data - incremental load without table

I am using a very simple architecture to copy data from an external source into Azure Data Lake Storage gen 2 and serve it to PowerBI via a Serverless pool (where I perform some aggregations). For the initial load, I used CopyData activity (Synapse…
ciufy13
  • 5
  • 1
0
votes
1 answer

Aggregation in Azure Data Flow is Returning Invalid Value

I have created a data flow in Data Factory. Step 1. Read the parquet file. Step 2. Aggregate the file to get the Max(DateField) Step 3. Use a derived column to write in a Value. Step 4. Alter row task with Value and the DateField. Step 5. Sink…
grimleyed
  • 1
  • 1
0
votes
1 answer

Custom Pagination of Rest API in Azure Data Factory

I would like to retrieve all results from Rest API endpoint.The URL has the below form https://myapi.com/relativeapi?project=&repo=&prId=&page=&pageSize=&startTime=&endTime By default when requesting data it is returned only the first page. A sample…
christi08
  • 5
  • 3
0
votes
1 answer

Incremental load from multiple tables for join queries in Nifi

For Incremental load we will be using QueryDatabaseTable processor which extracts data incrementally from one table. For writing sql query which extracts data from multiple tables we are using ExecuteSQL processor. How can we extract incremental…
Anil Krishna Parvathaneni
  • 43
  • 1
  • 5
0
votes
2 answers

Microsoft PowerApps Excel incremental load

I'm very new to PowerApps and have some task of making the app which will involve incremental data load. Currently we have chosen Excel sheets as the data source. So the actual task was make the data to be read from one Excel file and written to…
Leo
  • 37
  • 6
0
votes
2 answers

Azure Data Factory Error: "incorrect syntax near"

I'm trying to do a simple incremental update from an on-prem database as source to Azure SQL database based on a varchar column called "RP" in On-Prem database that contains "date+staticdescription" for example: "20210314MetroFactory" 1- I've…
Mo Ay
  • 3
  • 1
  • 2
0
votes
0 answers

REST API to Excel integration Azure Logic Apps

I have data extracted through REST API from one of my applications and store it in a spreadsheet. I am using Azure Logic apps to extract data based on the date range for incremental loading everyday. In worst case, if I run the workflow twice in a…
Raj001
  • 1
  • 1
  • 2
0
votes
1 answer

How can I do incremental load into Elasticsearch using logstash+jdbc and Database Trigger

I have elasticsearch cluster and i am using logstash + jdbc to load the data. I Would like to know is there any way that we can do a incremental load based on database trigger instead of depending on query that uses last modified value column Any…
Ravi Teja Muddada
  • 341
  • 3
  • 18
0
votes
1 answer

Best way to replicate MongoDB NoSQL into SQL tables

How can i replicate (incremental load) MongoDB (NoSQL) to SQL tables. We have a web-based solution that loading data into MongoDB. The data size is almost 1TB. We need to do BI Reporting in the Looker BI tool. but looker doesn't support MongoDB…
Adeel Nazir
  • 54
  • 10
0
votes
1 answer

How to perform hourly incremental extracts from an oracle datasource using last_mod_dt (timestamp) in Airflow?

Need to do hourly table refresh/loads using Airflow/Python from an oracle datasource using last_modified_dt timestamp column. In Airflow there's airflow.models.taskinstance API which exposes data from task_instance metadata table and which has…
ManiK
  • 377
  • 1
  • 21
1
2
3