0

What is the best way to replicate data from Oracle Goldengate On premise to AWS (SQL or NOSQL)?

sebanazarian
  • 3
  • 3

2 Answers2

0

I was just checking this for azure,

My company is looking for solutions of moving data to the cloud

  • Minimal impact for on-prem legacy/3rd party systems.
  • No oracle db instances on the cloud side.
  • Minimum "hops" for the data between the source and destination.
  • Paas over IaaS solutions.
  • Out of the box features over native code and in-house development.
  • oracle server 12c or above
  • some custom filtering solution
  • some custom transformations

** filtering can be done in goldengate, in nifi, azure mapping, ksqldb

solutions are divided into:

  1. If solution is alolwed to touch.read the logfile of the oracle server
  • you can use azure ADF, azure synapse, K2view, apache nifi, Orcle CDC adapter for BigData (check versions) to directly move data to the cloud buffered by kafka however the info inside the kafka will be in special-schema json format.
  1. If you must use GG Trail file as input to your sync/etl paradigm you can

  • use a custom data provider that would translate the trailfile into a flowfile for nifi (you need to write it, see this 2 star project on github for a direction

  • use github project with gg for bigdata and kafka over kafkaconect to also get translated SQL dml and ddl statements which would make the solution much more readable

other solutions are corner cases, but i hope this gives you what you needed

Mickey Perlstein
  • 2,508
  • 2
  • 30
  • 37
0

In my company's case we have Oracle as a source db and Snowflake as a target db. We've built the following processing sequence:

  1. On-premise OGG Extract works with on-premise Oracle DB.
  2. Datapump sends trails to another host
  3. On this host we have OGG for Big data Replicat that processes trails and then sends result as json to AWS S3 bucket.
  4. Since Snowflake DB can handle JSON as a source of data and works with S3 bucket it loads jsons into staging tables where further processing takes place.

You can read more about this approach here: https://www.snowflake.com/blog/continuous-data-replication-into-snowflake-with-oracle-goldengate/

Roman
  • 473
  • 5
  • 22