Questions tagged [data-integration]

Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution encompasses discovery, cleansing, monitoring, transforming and delivery of data from a variety of sources.

Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution encompasses discovery, cleansing, monitoring, transforming and delivery of data from a variety of sources.

It is a huge topic for the IT because, ultimately aims to make all systems work seamlessly together.

Example with data warehouse

The process must take place among the organization's primary transaction systems before data arrives at the data warehouse.
It is rarely complete, unless the organization has a comprehensive and centralized master data management(MDM) system.

Data integration usually takes the form of conforming dimensions and facts in the data warehouse. This means establishing common dimensional attributes across separated databases. Conforming facts is making agreement on common business metrics such as key performance indicators (KPIs) across separated databases, so these numbers can be compared mathematically.

332 questions
1
vote
2 answers

Moving data from a database to Google Big Query

I am trying to figure out a way to move data in batches(not streamed in) from on-prem database (Oracle) to Google Bigquery. I am trying to do this via Google Cloud Composer however I understand that I need to get data into GCS first before I can…
1
vote
1 answer

Flat File to SQL Server (BULK INSERT) ODI 11g Knowledge Module Only Loading Half of Records from Flat File

I am using Oracle Data Integrator 11g Interface to interface a flat file containing over a million records to a SQL Server table. Given some performance issues with using the general knowledge modules - I am leveraging the use of the Flat File to…
1
vote
1 answer

Creating a custom step in Oracle Data Integrator ODI using Groovy to delete duplicate rows

I have a case where I need to populate a target table, I have a column "CUSTOMER_ID" which has duplicate values in various rows. I need to populate target table with only one row from CustomerID no matter which one (it can be picked randomly).…
1
vote
1 answer

Data Ingestion: Receive and decode Message files(.txt) in Real Time

We have a requirement in our project to read and decode Message Text files in real time. The files are received by a Third Party using MQ Setup. The requirement is to extract the files from MQ Server, Store it in our system, decode using a spec and…
AngiSen
  • 915
  • 4
  • 18
  • 41
1
vote
1 answer

Input xlsx or csv file in Kettle from microsoft sharepoint that needs login

I want to input xlsx files or csv files from a microsoft sharepoint site that i need to login. I have account on the sharepoint but Kettle Microsoft excel input step does not have options for login. The result is that the transformation gives an…
1
vote
2 answers

What is the real benefit of using GraphQL?

I have been reading about the articles on the web about the benefits of graphql but so far I have not been able to find a single benefit of it. One of the most common benefits mentioned in those articles are below? No Overfetching with…
ralzaul
  • 4,280
  • 6
  • 32
  • 51
1
vote
2 answers

Delete lines based on two tables

I have one problem, where i have one main table one other table. I have in one table (Accidents), attribute named Accident_ID (Values goes like this 1,4,7,10 and so on) In other table named casualties, i have the same attribute Accident_ID (But…
1
vote
2 answers

SSIS package creation for integrating MSSQL and MySql dbs

I am trying to create an SSIS package for integrating between MSSQL and MYSQL. I have no prior experience of working with Bids or SSIS and following the instructions from here. I added the OLE DB Source, Lookup, Conditional Split, OLE DB…
naiquevin
  • 7,588
  • 12
  • 53
  • 62
1
vote
1 answer

Extract data from Excel sheet to multiple SQL Server tables

I have an Excel file with a few columns (20) and some data that I need to upload into 4 SQL Server tables. The tables are related and specific columns represents my id for each table. Is there an ETL tool that I can use to automate this process?
Koji D'infinte
  • 1,309
  • 12
  • 20
1
vote
1 answer

Getting result from SQL Server Stored Procedure

I need to get the result from a stored procedure in a MSSQL Server database. I tried different ways: Using a Table Input Step, with this query: exec dbo.storedproc @param1 = 5, @param2 = 12, @param3 = null, @param4 = null, @param5 = '2017/08/29',…
1
vote
1 answer

Data Integration Tool For HANA

Is there a good data integration tool available for SAP HANA which could accomplish the following : Consuming data periodically (a user defined interval) from REST based web service (A simple URL containing XML) Parsing the XML and extracting the…
Shubha Lakshmi
  • 127
  • 2
  • 2
  • 15
1
vote
1 answer

Talend: How to get instance of a component in tjava

I want ask if there is way to get an instance of a component if a job (ex:tmap, tmysqlinput) in tjava code and then manipulate it manually using code? Thank you
1
vote
3 answers

Talend : Create a Component using java code

I am new in user of Talend open studio I want to find a way to add component like tinputfile or tligrow without the drag and drop tools , but with java code manually Help please Thank you very much
1
vote
1 answer

How to call Data Integration when interaction is involved?

First let's define Data Integration: "combine data from different sources and present users with a unified view" I interpret data sources as databases with tables, files and more. My question: What do you call it when, on the functional level,…
hsmit
  • 3,906
  • 7
  • 34
  • 46
1
vote
1 answer

Using Talend Open Studio DI to extract extract value from unique 1st row before continuing to process columns

I have a number of excel files where there is a line of text (and blank row) above the header row for the table. What would be the best way to process the file so I can extract the text from that row AND include it as a column when appending…
mobcdi
  • 1,532
  • 2
  • 28
  • 49