Questions tagged [data-integration]

Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution encompasses discovery, cleansing, monitoring, transforming and delivery of data from a variety of sources.

Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution encompasses discovery, cleansing, monitoring, transforming and delivery of data from a variety of sources.

It is a huge topic for the IT because, ultimately aims to make all systems work seamlessly together.

Example with data warehouse

The process must take place among the organization's primary transaction systems before data arrives at the data warehouse.
It is rarely complete, unless the organization has a comprehensive and centralized master data management(MDM) system.

Data integration usually takes the form of conforming dimensions and facts in the data warehouse. This means establishing common dimensional attributes across separated databases. Conforming facts is making agreement on common business metrics such as key performance indicators (KPIs) across separated databases, so these numbers can be compared mathematically.

332 questions
0
votes
1 answer

Talend string max length

With Talend Data Integration , I read a csv file. I make a transformation with a tmap and I write in another file. My fields must not exceed 40 characters. My repository I set length to 40, but I have no warning when my chain beyond . What to do?
Sworios
  • 37
  • 1
  • 9
0
votes
1 answer

SSIS Data Integration - Data flow

I have 4 steps say 1,2,3,4,5 in SSIS script.The steps 4 and 5 are depedent on 2, hence it starts once the step 2 is completed. Steps 2 and 3 are independent. Normally when the execution begins the step 1 is started and once it is done, step2 and…
Karthi
  • 708
  • 1
  • 19
  • 38
0
votes
1 answer

Split transformation steps based on Parameter in Pentaho Data Integration

I have a transformation with a boolean parameter. If the parameter is 1, I want the data to flow through one transformation path. But if the parameter is 0, I want it to flow through a different transformation path. What step can I possibly use to…
Dezzie
  • 934
  • 3
  • 18
  • 35
0
votes
1 answer

Changing Interface Staging Area in ODI using ODI SDK classes

Hi I am new to ODI and ODI SDK. I have created some sample mappings using ODI SDK and now I am wondering if there is a way to change the Staging Area for my interface. Till now I have used default target server as my staging area. Basically, I…
BigD
  • 117
  • 3
  • 10
0
votes
0 answers

What is the best object mapper for merging DTOs from many sources?

I need to map many properties from two or more DTOs to common one. The original problem is that I need to use the domain model of other software in data integration layer. After research I have an idea to use "object to object mapper" framework (or…
Viper
  • 597
  • 1
  • 8
  • 24
0
votes
1 answer

SAS DI Error 22-232 in ROW_NUMBER () OVER (PARTITION BY construction

I have 2 SQL Join transforms (one using only SELECT, the other using SELECT and WHERE) and I have syntax error near OVER when submitting the following code: select row_number () over ( partition by t0.A order by t0.B DESC )…
G1N000
  • 3
  • 2
0
votes
1 answer

How do I get the right values out of this JSON?

I've got a really stupid question but I can't seem to find any answer whatsoever. I have this JSON from an online weather API, the JSON looks like…
E. V. d. B.
  • 815
  • 2
  • 13
  • 30
0
votes
0 answers

Pentaho DI missing or skipping rows

I have problem with pentaho data integration tool. I made some ETL to insert rows to fact table, but the problem is that not all rows are inserted to fact table. I ran pentaho several times and every time it gives me different results. Sometimes…
xurca
  • 2,426
  • 3
  • 24
  • 29
0
votes
0 answers

Row count based on a list of tables in Pentaho

I am using an Input Table step to retrieve a list of owners and tables from Oracle's ALL_TABLES, then I want to pass to another step to, for each of this owner.table entries, it performs a SELECT COUNT(*) FROM owner.table. The final result I want…
Lucas Rezende
  • 2,516
  • 8
  • 25
  • 34
0
votes
1 answer

How pentaho's copy block handles duplicate data?

I a trying to copy my MySQL DB to HANA DB through Pentaho's Copy Table Wizard. It automatically created a workflow but I am confused if the destination DB is partially filled, then would it handle duplicate rows or it just copy them anyways?
erogol
  • 13,156
  • 33
  • 101
  • 155
0
votes
1 answer

PDI: Returning the result of a SELECT-statement to the datastream

Using PDI (Kettle) I am filling the entry-stage of my database by utilizing a CSV Inputand Table Output step. This works great, however, I also want to make sure that the data that was just inserted fulfills certain criteria, e.g. fields not being…
daZza
  • 1,669
  • 1
  • 29
  • 51
0
votes
2 answers

Hive Data Integration error in Pentaho

I am using Bigdata Lite3.0, Pentaho 5.1, CDH5.0. I want to connect Hive tables using Pentaho. While trying to connect, i am getting below errors. Error connecting to database [Hive] : org.pentaho.di.core.exception.KettleDatabaseException: Error…
Jiten Patel
  • 11
  • 2
  • 4
0
votes
1 answer

map two table columns using pentaho Data integration

I am using pentaho data integration in which I am able to get table columns but while editing I want the columns to save as edited but it giving some error able to save in .txt file and want to get as input for next transformation. How to map two…
0
votes
1 answer

REST Client step in Pentaho DI (Kettle) says I do not have permission to API

I am using version 5.1 of Pentaho DI (Kettle) and am having an issue with the "Rest Client" step. The step executes properly and successfully, but the result says I do not have permission to use the API. I have confirmed in another tool that I do…
JHFB
  • 441
  • 1
  • 5
  • 20
0
votes
1 answer

JSONpath Stopper

There is a parent field called "price", but there is also a deeply-embedded child field also called "price" (first one is probably MSRP). How do I get JSONpath to only parse the parent ones? { "name":"Toaster", "price":"19.99", "store":{ …
Jarrett G.
  • 354
  • 3
  • 16