Unclear if either Pysftp or Dropox is lagging when creating and pushing a file to an SFTP server?

Question

I am having a strange problem with pysftp. My code creates a text file and then almost immediately pushes this to an SFTP server using the pysftp lib. The text file that is created and then pushed is saved on a Dropbox folder.

Intermittently I get 'Errno[2] File Does not exist' errors at the push part of the code. It is almost as though the created file sometimes isn't available quick enough for the next part of code, and I am suspecting this may be due to lag with Dropbox. I could put in a EXISTS part to wait, but wanted to check if my suspicions are correct before proceeding? Its just a bit wierd that this code runs fine 90% of the time.

Code below with *** in place of credentials:

def get_sec_info(bb_req_dir,bb_req_name,bb_firmname,sec_missing):

    #Build request file and return the filepath
    filepath = bb_req_file(bb_req_dir,bb_req_name,bb_firmname,sec_missing)

    #Send the file to Bloomberg, wait and download response file. Returns the filename of the response file.
    dir_response = bb_upload_download(filepath,bb_req_name)

    df_secs = parse_response(dir_response)

    return df_secs



def bb_req_file(bb_req_dir,bb_req_name,bb_firmname,sec_missing):
    #Build req file
    filepath = bb_req_dir + bb_req_name + '.req'
    req_file = open(filepath,'w+')

    with req_file:
        req_file.write('START-OF-FILE')
        req_file.write('\n')
        req_file.write('FIRMNAME=' + bb_firmname)
        req_file.write('\n')
        req_file.write('FILETYPE=pc')
        req_file.write('\n')
        req_file.write('REPLYFILENAME=' + bb_req_name + '.out')
        req_file.write('\n')
        req_file.write('SECMASTER=yes')
        req_file.write('\n')
        req_file.write('DATEFORMAT=ddmmyyyy')
        req_file.write('\n')
        req_file.write('PROGRAMNAME=getdata')
        req_file.write('\n')
        req_file.write('PROGRAMFLAG=adhoc')
        req_file.write('\n')
        req_file.write('MIFIR=yes')
        req_file.write('\n')
        #Add the fields being requested
        req_file.write('START-OF-FIELDS')        
        req_file.write('\n')
        req_file.write('CFI_CODE')
        req_file.write('\n')
        req_file.write('END-OF-FIELDS')
        req_file.write('\n')
        #Add the securities
        req_file.write('START-OF-DATA')
        req_file.write('\n')
        for sec in sec_missing:
            req_file.write(''.join(sec))
            req_file.write(' |Ticker|')
            req_file.write('\n')

        #Close
        req_file.write('END-OF-DATA')
        req_file.write('\n')
        req_file.write('END-OF-FILE')

    req_file.close()

    return filepath



def bb_upload_download(filepath,bb_req_name):

    warnings.filterwarnings('ignore','.*hostkeys')

    cnopts = pysftp.CnOpts()
    cnopts.hostkeys = None

    with pysftp.Connection('****', username='****', password='****', cnopts=cnopts) as sftp:


        sftp.put(filepath)
        response = filepath[:-3] + 'out'
        file_response = bb_req_name + '.out'

        exists = False
        while exists == False:
            exists = sftp.isfile(file_response)

        sftp.get(file_response,response)


    return response

Andy

score 0 · Accepted Answer · answered Apr 20 '19 at 08:01

Ok I seem to have resolved this and thought I would put an answer in case it helps others.

The problem was misleadingly due to the SFTP server I was pushing the file to, processing it so quickly that Pysftp couldn't get a confirmation response that it had uploaded successfully.

Fix is to put confirm=False in the sftp.put() command.

Unclear if either Pysftp or Dropox is lagging when creating and pushing a file to an SFTP server?

1 Answers1