3

I wanted to use filemask in the GoogleCloudStoragePrefixSensor. I cant use the GoogleCloudStoragePrefixSensor because I also need to see the ending oif the file mask. BAsically my file is like "tv_link_input_*.xml". So, tried using GoogleCloudStorageObjectSensor, but its keep running without any OP. Using airflow 1.10.14. Code

check_file_gcs = GoogleCloudStorageObjectSensor(
            task_id='check_file_gcs',  
            bucket=source_bucket,
            object="processing/tv_link_input_*xml",
            poke='10',
            dag=dag
            )

Thanks

Elad Kalif
  • 14,110
  • 2
  • 17
  • 49
Mohammed
  • 43
  • 5
  • Hi Elad, Its with .xml. The file mask name is exactly "tv_link_input_YYYYMMDDHHMMSS.xml". – Mohammed Feb 17 '21 at 14:31
  • This functionality isn't supported by google storage library. https://github.com/googleapis/python-storage/blob/d35ab3537d1828505f614d32b79b67173c9438c0/google/cloud/storage/client.py#L733 so probably worth opening a github issue requesting that functionality. If added by the google library it will be easy to do this with airflow. Without it - you will need to write custom python code that achieve that. See also https://stackoverflow.com/questions/44702835/google-cloud-storage-ignoring-delimiter – Elad Kalif Feb 17 '21 at 15:39

0 Answers0