0

I need to enter regular expression in a file path name inside tfileinputdelimited

I have a file watcher that is looking inside a directory for new files to be added. I want the new files have similar word for example file 1 is called apple1.csv and the other file that will be added at another time is called apple2.csv I want a way in order to tell talend to extract the file that contains the word apple, regardless of whats before or after.

Right now I have:

twaitforfile -> tflowtoiterate -> tfileinputdelimited -> tmap -> tdboutput

I believe tfileinputdelimited should have a regular expression as right now I selected a specific file

enter image description here

enter image description here

RealHowTo
  • 34,977
  • 11
  • 70
  • 85
Zuhair Ahmed
  • 17
  • 5

1 Answers1

1

For the twaitforfile component, in the File Mask field specify the following: "*apple*.csv". This should grab only the files that have the word apple in them.

enter image description here

To dynamically grab these files base on their file names through tfileinputdelimited, you can use the FILENAME global parameter of the twaitforfile to achieve this:

enter image description here

Neo
  • 779
  • 5
  • 13
  • Thank you, Im still having the problem that it doesn't add to the database the new file apple2.csv. right now I have twaitforfile -> tflowtoiterate -> tfileinputdelimited -> tmap -> tdboutput . I am thinking maybe because I have tfileinputdelimited which chooses a specific file .. in this case apple1.csv , keeping apple2.csv not loaded. Could you please tell me if my order is correct otherwise how can I map a schema from twaitforfile right away – Zuhair Ahmed Feb 09 '19 at 22:33
  • Another followup: what if I need to make the file watcher also look into subdirectories. I clicked on that checkbox on the twaitforfile, however, although it is detected, it gives an error in mapping, I believe also due to something with tfileinputdelimited. How can I make it look into the directory and also subdirectory – Zuhair Ahmed Feb 10 '19 at 14:39
  • It should work fine. Are you sure you don't have any errors in your CSV? What errors are you getting? – Neo Feb 10 '19 at 15:38
  • it gives the file path along to the file name that is inside the subdirectory.. then it says this : (No such file or directory) – Zuhair Ahmed Feb 10 '19 at 15:46
  • ok so I figured out something. If I take a file from the main directory and copy it to the sub directory then it works. However, if I create a csv file and save it directly inside the subdirectory it doesn't. But I have no clue why – Zuhair Ahmed Feb 10 '19 at 16:02
  • In that case we need to make it more dynamic, it the tFileInputDelimited set the File Name/Stream as (String)globalMap.get("tWaitForFile_1_CREATED_FILE") so that it can grab the entire path of the file and accepts subdirectories – Neo Feb 10 '19 at 16:02
  • Nope this didn't work, right now its not accepting from both the directory and subdirectory. err: path/applycopy.csv (No such file or directory) – Zuhair Ahmed Feb 10 '19 at 16:11
  • Can you attach an image of your Job, the details of the twaitforfile and tfileinputdelimited? – Neo Feb 10 '19 at 16:19
  • I have added the images in the question above – Zuhair Ahmed Feb 10 '19 at 16:23
  • tfileInputDelimited file stream should be equal to (String)globalMap.get("tWaitForFile_1_CREATED_FILE") only. Remove the static part that is at the start. – Neo Feb 10 '19 at 16:25
  • YES that is perfect. I Really appreciate your help.. this is really valuable.. thanks alot – Zuhair Ahmed Feb 10 '19 at 16:35
  • Hello I have another problem, and I thought you could be able to help me due to your expertise in talend and because your suggestion last time helped me a lot. – Zuhair Ahmed Feb 16 '19 at 17:06
  • I need to send an email at the end of ETL process. However I need the email address to be extracted from the tfileinputdelimited.Right now I have twaitforfile -> tflowtoiterate -> tfileinputdelimited -> tmap_1 -> tbdoutput ->tsendemail What I want to do is get the email from the tfileinputdelimited, I dont know the email ahead of time thats why I can't add a predefined list. It all depends on the new file added and detected by the file watcher. I can't also add global variable to tfileinputdelimited since the tfileinputdelimited is connected to other things from both sides as shown above – Zuhair Ahmed Feb 16 '19 at 17:07