4

I'm currently new to Talend and I'm learning through videos and documentation, so I'm just not sure how to approach/implement this with best practices.

Goal

Integrate Magento and Quick Book using Talend.

My thoughts

Initially my first thought was I will setup direct DB connection for Magento and will take relevant data which I need and will process it and will send to QuickBook using REST API's(specifically bulk API's in batch)

But then again I thought it would be little hectic for me to query Magento database(multiple joins) so I've another option to use Magento's REST API.

But as I'm not much familiar with the tool I'm struggling little to find best suitable approach, so any help is appreciated.

What I've done till now?

  1. I've saved my auth(for QB) and db(Magento) credentials data in file and using tFileInputDelimited and tContextLoad, I'm storing them in context variables so they can be accessible globally.

enter image description here

  1. I've successfully configured database connection and dbinput but I've not used metadata for connection(should I use that and if Yes how can I pass dynamic values there?). I've used my context variables data in db connection settings.
  2. I've taken relevant fields for now but if I want multiple fields simple query is not enough as Magento stores data in multiple tables for Customer etc but it's not big deal I know but I think it might increase my work.

enter image description here

For now that's what I've built and my next step is send the data to QB using REST while getting access_token and saving it to context variable and again storing the QB reference into Magento DB.

Also I've decided to use QB bulk API's but I'm not sure how I can process data in chunks in Talend(I tried to check multiple resources but no luck) i.e. if the Magento is returning 500 rows I want to process them in chunks of 30 as QB batch max limit is 30, so I will be sending it using REST to QB and as I said I also want to store back QB reference ID in magento(so I can update it later).

Also this all will be on local, then how can I do same in production? how I can maintain development and production environment?

Resources I'm referring

For REST and Auth best practices - https://community.talend.com/t5/How-Tos-and-Best-Practices/Using-OAuth-2-0-with-Talend-to-Access-Goo...

Indrajeet Latthe
  • 374
  • 1
  • 8
  • 22

1 Answers1

0

Nice example for batch processing here: https://community.talend.com/t5/Design-and-Development/Batch-processing-in-talend-job/td-p/51952

Redirect your input to a tFileOutputDelimited.

Enter the output filename, tick the option "Split output in several files" from the "Advanced settings" and enter the value of 1000 into the field "Rows in each output file". This will create n files based on the filename with 1000 in each.

On the next subjob, use a tFileList to iterate over this file list to get records from each file.

Oly
  • 302
  • 1
  • 8