0

I have a usecase. I want to Integrate / Transform data from different / disparate sources without storing it. Data sources are database(oracle,db2,etc), Webservice(Rest/Soap), Flat files(CSV, XML, JSON), MQ dumps, mainframe systems. I want to pull data from these sources and do some kind of intelligent transformation and integration and provide it our customers. It looks like typical ETL scenario, but my situation is different. I am not allowed to store the data given by the desperate sources, that means, for simple example, i pull data from oracle, soap and a rest, and do all my intelligent transformations and integrations on the fly.

I browsed through google and technical stuffs but could not get convincing solution to my problem.

If you guys can help me giving some valuable insight on this problem and give suggestion and probable approaches to it.

Note: Data size from these sources can sometime be really huge.

Thanks in Advance

Sadique
  • 140
  • 8

2 Answers2

0

Take a look at htto://teiid.org

Thst is exactly what it does, and it is Open Source.

Ramesh Reddy
  • 554
  • 1
  • 3
  • 8
  • Thanks Ramesh for your suggestion. I have already reviewed and did some pocs on JDV using teiid. It is really a good tool.But in some scenario it failed like, when i have large data, complex aggregation/federation, when data itself is really complex(multi-level hierarchical data, in xml or Json) and scalability challenges of the tool. If you can suggest any supporting guideline to overcome those typical scenario. – Sadique Sep 27 '17 at 06:44
  • You would need to provide details of each issue where you are failing for any suggestions. Teiid also supports clustering and caching to help with scaling. Suggest different thread for each issue. – Ramesh Reddy Sep 27 '17 at 12:45
0

Talend Open Studio y a great solution as well, I'm using it and it's great and easy to make the ETL workflow.

https://www.talend.com/products/data-integration/data-integration-manuals-release-notes/

You can see a lot of help videos: https://www.youtube.com/results?search_query=talend+studio

marcode_ely
  • 434
  • 1
  • 5
  • 18