-1

We are facing an issue of data migration. We have two heavily loaded tables. We used to make a join and fetch aggregated data from both tables in real time. Since due to increase in size we want to migrate it in mongodb with remodelling having a flat db layout to overcome joins. I researched a lot but i am not able to figure it out the exact tool or framework. Sqoop and flume migrate table from one system to another directly but not remodelling. I am not able to figure it out whether Spring XD or apache nifi transform as per my need.

Pseudo steps are as follows

step 1 (fetch from first system)

1) var a = select id, name, age from info where age > 10;

step 2 (fetch from first system)

2) var b = select address, income from professional_detail where id = a.id;

step 3 (insert in second system)

3) insert into aggregate_table (id, age, name, income, address) values (a.id, a,age,a.name, b.income, b.address)

Which is the exact framework or tool best suited in this? We only want to do sql query, bash command and java code. No other languages

Community
  • 1
  • 1
ravi ranjan
  • 5,920
  • 2
  • 19
  • 18

1 Answers1

0

Based on the questions and comments it is safe to say you need an ETL solution.

As you mention some tools, you seem to be looking at the Hadoop ecosystem, in this space Spark seems to be the most natural fit with your requirements.

Dennis Jaheruddin
  • 21,208
  • 8
  • 66
  • 122