Transfer i/o files & invoke scripts betwen 2 applications running in different Red Hat Linux Server

Question

Issue Background:

1. Moving a Java/J2EE application from dedicated RedHat Linux server to cloud RedHat Linux server.
2. I am analysing the Batch processing jobs involved in this application to implement similar processing in cloud environment.

Current Approach:

We have 2 applications App1 & App2 both are in same RedHat Linux server. Both applications have shared directories in the same server. Also Appl can call shell scripts in App2's directory to get some job done.

App2's process:

1. External system send input file(.DAT) to App2 via NDM Jobs.
2. Received input file(.DAT) will be placed in App2's input file directory.
3. Process the records in file using Java/J2EE program/component
4. Generate the outputfile.
5. Place it in App1's shared directory.
6. App1 has a filewatcher pointed to this directory to consume this file.

Upcoming approach:

1. App2 will be moved to cloud Red Hat Linux server.
2. App2 will be running in atleast 2 nodes.

Challenges:

1. External system job still point to same old directory in non cloud Linux server.
2. After processing, output file must be in App1's shared directory.

Expectation:

1. App2's process running in cloud is, expected to read & process this file.

Request you all to suggest a best approach for this requirement. Can we have FTP or REST-webservice to read the input file from non cloud Linux server?

2) App1 has business requirement to call shell scripts in App2. How can we provide a service to call App2's shell script located in cloud server.

I am new to cloud. Please excuse me if my questions are irrelevant or trivial.

Thank You In Advance.

Answer 1

I Thank You very much and appreciate you for spending your precious time to go through my post and for your quick response.

I had most of the following questions with me while posting this question to stack overflow. Hence I reached out to Stackoverflow for suggestions to model an ideal design for this requirement.

Please find my thoughts below for your questions.

It depends on the capabilities of your cloud servers and your (non functional) requirements. Does the cloud servers have access to App Server 1?

Ans: No. I am discussing with unix/security team to find the feasibility for Mount "i/o & shared directories",If we can mount it might allow App2 to access i/o & shared directories from Cloud server. Cloud server is expected to handle the processing load of App2 to get better performance and i/o files are expected to be in App server 1 always .

Which Ports / Applications can be used?

Ans:This is to be determined yet.

Are there other limits like volume / transfer limits?

Ans:I believe we have no file transfer limits, since both (cloud & App Server1) are belongs to our enterprise.

How do you ensure that the shell script on App Server 2 (now there are two nodes) is executed only once?
How do you ensure that the input file in the two App 2 Server Nodes is processed only once?

Ans:For the above 2 questions,Here are my thought process, I hope we must implement semaphore logic to avoid duplicate processing. we can have a Database entry with process name & node id to be persisted for each request or we can place an empty file with jobnames along with following status as its suffix
.ready -file to notify the process is ready
.Inprocess - to notify that a node is processing the file
.processed - to notify processed state
.rejected - to notify process end with error.

Do the two App 2 Server Nodes use a shared file system?

Ans:This approach is not discussed yet, but we need a share file system. Since each process in respective nodes can write logs and the log should be in one place which can be very useful to find out the issues.

Is it possible to use a shared directory beteween App 1 and the 2 App 2 Server Nodes?

Ans:This is to be determined.I am discussing with Unix team to Mount share directory.

I am having a thought process of replacing the scripts with webservices. I may be wrong but I would like to share my view in this post.
Can we have a webservice call to pass request and transfer/process files between App1 & App2 instead of shell?

Scenario:
1. External system send input file(.DAT) to App2 via NDM Jobs.
2. We can write a script in (server1)place to call App2's webservice to process this file.
    NDM post file copy job will trigger this script.
3. we can have a webservice to transfer the file to app2 in cloud?
4. WebService call will hit the endpoint.I hope load balancer will pass the webservice call to
     one of the node where the service exist and the data's will be processed & output file will
    get generated.
5. We need to have another webservice to send the output file to Server 1 Also, I believe
    we need to have a webservice client in both servers to exchange webservice call/File transfer
    between Server 1 & Cloud server.

The webservice approach which i have suggested here is completely my thought process. I have not implemented it or tested it. If this process is correct. I may need to find out the pros/cons in this approach

Kindly request you to share your comments on my response and also request you to share your suggestions.

Thank You.