How can I use ExecuteScript in Apache Nifi to retrieve a list of files on an FTP server using Python?

Question

I'm new to apache nifi and I am having trouble with writing a script that can give the list of files within a ftp server using the ExecuteScript on apache Nifi.

I have the following script that I'm trying to execute using the ExecuteScript processor:

import ftplib

flowFile = session.get()
if flowFile != None:
 # Get the FTP server details from NiFi's FlowFile attributes
 ftp_host = flowFile.getAttribute('HostName')
 ftp_port = int(flowFile.getAttribute('PortNumber'))
 ftp_user = flowFile.getAttribute('Username')
 ftp_password = flowFile.getAttribute('Password')
 ftp_directory = flowFile.getAttribute('RemotePath')


 ftp = ftplib.FTP()
 ftp.connect(ftp_host, ftp_port)
 ftp.login(ftp_user, ftp_password)
 ftp.cwd(ftp_directory)

 file_list = ftp.nlst()   

 newFlowFile = session.create(flowFile)

 for files in file_list:  
  newFlowFile = session.putAttribute(newFlowFile, 'filename', files)

 session.transfer(newFlowFile, REL_SUCCESS)

The following is the error message I am getting :

"Failed to process session due to javax.script.ScriptException: org.apache.nifi.processor.exception.FlowFileHandlingException: org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=15ef2c95-8f36-4669-9b38-6bb1b39b7e8d,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1685699673185-1, container=default, section=1], offset=30819, length=314],offset=0,name=15ef2c95-8f36-4669-9b38-6bb1b39b7e8d,size=314] transfer relationship not specified in at line number 25: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: org.apache.nifi.processor.exception.FlowFileHandlingException: org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=15ef2c95-8f36-4669-9b38-6bb1b39b7e8d,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1685699673185-1, container=default, section=1], offset=30819, length=314],offset=0,name=15ef2c95-8f36-4669-9b38-6bb1b39b7e8d,size=314] transfer relationship not specified in at line number 25"

Can any please assist me with the above, or any other way I can achieve this with the ExecuteScript

Thank you.

I am expecting to get the list of file names that are present on the ftp server and create flowFiles that will have the names of the files as filename of the flowFile

Did you try ListFTP first? If so what does it not do that you're trying to do? — mattyb, Jun 02 '23 at 18:01
@mattyb I did try the ListFTP processor but my problem with it is, it is not dynamic since it does not take any flowFile Attributes. For Instance in my use case the FTP server details are stored on Microsoft SQL Server database within a table, so I grab that data and I created attributes so I can use them dynamically within the python script. — Athenkosi Lengs, Jun 05 '23 at 07:34
@daggett My apologies, I just edit the post with the code as text this this. — Athenkosi Lengs, Jun 05 '23 at 07:44
i see at least 2 issues in your code: `1/` you have to do something with `flowFile` - remove or transfer it. `2/` in `for files in file_list` loop you are setting attribute `filename` multiple times into the same file - only last value will stay in there. — daggett, Jun 05 '23 at 12:32

score 0 · Answer 1 · answered Jun 07 '23 at 08:35

I manage to get it right with the following code:

import ftplib

flowFile = session.get()
if flowFile != None:
 # Get the FTP server details from NiFi's FlowFile attributes
 ftp_host = flowFile.getAttribute('HostName')
 ftp_port = int(flowFile.getAttribute('PortNumber'))
 ftp_user = flowFile.getAttribute('Username')
 ftp_password = flowFile.getAttribute('Password')
 ftp_directory = flowFile.getAttribute('RemotePath')


 ftp = ftplib.FTP()
 ftp.connect(ftp_host, ftp_port)
 ftp.login(ftp_user, ftp_password)
 ftp.cwd(ftp_directory)

 file_list = ftp.nlst()   



 for files in file_list:  
  newFlowFile = session.create(flowFile)
  newFlowFile = session.putAttribute(newFlowFile, 'filename', files)
  session.transfer(newFlowFile, REL_SUCCESS)

session.remove(flowFile)

As for the creation of the new flowFile with the added attribute 'filename' which comes from the list of files names on the ftp server, using the for loop allowed me to iterate through the list of files and add the filenames as the the attribute each new flowFile that was created

How can I use ExecuteScript in Apache Nifi to retrieve a list of files on an FTP server using Python?

1 Answers1