0

I need to move some files between buckets after they are processed in the pipeline, however I have come across files that contain characters like "+" or "-" within their name (example: data+1+2132121.json)

following the documentation in this How to set runtime arguments in a CDAP/DATA FUSION pipeline? response

I have set the source path to "gs://mybucket/data/${input}" so that the value of my argument can be "data+1+" and only move those files that begin with "data+1+" are the ones that will be moved from location.however when using wildcards() I cannot move the files.some idea that it may be happening?

DevJonDoe
  • 1
  • 1
  • It looks like wildcard support is an open feature request (https://cdap.atlassian.net/browse/PLUGIN-698), so it is not currently supported. – Albert Shau Sep 10 '22 at 00:04
  • Answered a similar question here https://stackoverflow.com/questions/73815959/move-only-files-that-were-read-google-cloud-data-fusion-pipeline. Hope it helps! – Janaki Ram Goteti Sep 26 '22 at 09:12

0 Answers0