0

I need to download my company's marketing campaigns on Bing Ads from Airflow. I have seen there is a Bing Ads SDK in Python that may be a good starting point to build a custom operator that would make an API call to Bing Ads and upload the resulting csv to S3 using the s3 hook. I previously used the Google Ads client library to download campaigns as a list of Python objects inside the PythonOperator, but this library was way more intuitive than BingAds.

There is no BingAds hook in Airflow unfortunately, so I have to rely fully on this Python library and wrap it inside the PythonOperator. Has someone built a similar solution in the past that can point me in the right direction?

Best,

Ramon Soto
  • 1
  • 1
  • I use a BashOperator for executing a python script that includes the Bing Ads library in python. This can be done as well in PythonOperator as you said. Also, I find more convenient to upload the csv to s3 with pandas and s3fs, using just df.to_csv("s3://bucket/key.csv") – Javier Lopez Tomas Nov 27 '21 at 18:19
  • I use df.to_csv() to write to a file buffer (BytesIO()) and then upload the csv file using the s3 hook. That sounds like a good solution. Can you please share with me your Python script? I'd like to see how to store the results of the API call to a pandas df. The Bing Ads library does not seem intuitive to me. – Ramon Soto Nov 27 '21 at 19:32

1 Answers1

1

I think it's ok to use PythonOperator to run python code - including importing the Bing API - you just have to make sure that the packages installed for scheduler and worker. No need to build custom operator for that.

You can even use task flow API to integrate it nicely https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html

Airlfow is meant to be extended in customized - so writing your Python code is entirely expected. More - if you find it works for you, you might actually to contribute it back to Airflow after adding all the "generalisations" - BingAdsProvider would be nice.

Jarek Potiuk
  • 19,317
  • 2
  • 60
  • 61
  • Yes. there is open feature request for it https://github.com/apache/airflow/issues/8303 (though the request ask for more it must have first a BigAdsHook) – Elad Kalif Nov 28 '21 at 15:28