3

Cloud Data Fusion offers the ability to create ETL jobs using their graphical pipeline UI representation whereas Dataproc lets us run previously created Spark/Hadoop/Hive jobs.

With my limited experience in both these services, I have found Cloud Data Fusion to be the easier of the two to use & manage. I would like to know the use cases in which creating & running jobs in Dataproc is preferred over Cloud Data Fusion.

user929287171
  • 91
  • 5

1 Answers1

3

You asked for an opinion, so your question should be closed...

Anyway, it mainly depends on what you prefer! If you are a developer, and you want to handle, manage, customize/tweak all the steps your pipeline for performance, observability or security reason, code, and Dataproc is better for you. Same reason if all your developers already know the Hadoop ecosystem.

If you prefer to focus on the data transformation/wrangling with low/no code solution, Data fusion is for you. Especially if you have a few or no skills in development (business users).

At the end, all the pipeline will run on Dataproc.

guillaume blaquiere
  • 66,369
  • 2
  • 47
  • 76
  • Caveat! While this is true for the generic problems you can solve using Data Fusion. For any simple customization that isn't available in Google Data fusion which surely will be the case. One has to make their customized components and plugin them on the Data Fusion which isn't so easy. It takes good amount of time. – Zahid Khan Dec 11 '22 at 02:51