1

I have a scenario where I am sending a message to a request queue which in turns calls a lambda. That lambda performs some operation and write the data to a response queue(AWS SQS). How do I retrieve the specific message from my response queue for which request was sent and pass on that to the next step. Steps:

I have a Step function At step 1, I am sending the input data to a request queue ( say ABCQueue) This queue is trigger to a lambda The lambda processes, and writes data to response queue( say XYZQueue) In my step function next step is to receive the message from response queue

How do I maintain the sequence here i.e the data I sent that only should I receive from response queue?

Note: Since there would be multiple requests coming in and each request would be getting completed at different time. Is there any unique id by which I can check the id which I sent is only received back from response queue?

NOTE: I am integrating this with java

enter image description here

I have tried to use task token while using wait for callback. But that task token expires once I send success from lambda And when it goes to next state i.e. receive message the task token is not there and I can't get corresponding message

Archit Agarwal
  • 13
  • 3
  • It is not possible to "retrieve a specific message" from an Amazon SQS queue. Amazon SQS would be an inappropriate technology for this architecture. Why not write the data to an Amazon S3 object instead, and pass the object Key to the next Step? – John Rotenstein Oct 31 '22 at 23:20
  • @JohnRotenstein Actually we are calling the external systems once we put the message in the queue and depending upon the downstream systems messages would be processed and success or failure token would be returned to the state and step function moves to next state. – Archit Agarwal Nov 01 '22 at 03:59
  • What will be the best service to use here in order to acheive this? Can we use kinesis – Archit Agarwal Nov 01 '22 at 03:59
  • It all depends on how the 'downstream system' would send back the message. You could have the external system send a message to SQS, but you would need a Lambda function that is triggered by the message, which then figures out which State machine 'owns' the message, and it could then progress the State machine to the next step. So, rather than have the state machine "ask" for the response, the act of receiving the response in SQS can be used to tell the state machine to continue. No polling of SQS required. – John Rotenstein Nov 01 '22 at 05:29
  • Rhis approach sounds good. Can you help me how to figure out which state machine 'owns' that response. If lets say I have 1000 executions running simultaneously and I start receiving response in my queue from downstream applications how can I determine the message received is of which state machine execution – Archit Agarwal Nov 01 '22 at 05:45
  • Can you pass some information to the 'downstream' app (eg the ID of the state machine), and have it return that ID as part of its response? Or is there some other identifying data in the response that you can use? Worst case, you could use a simple database to map the ID back to the state machine. – John Rotenstein Nov 01 '22 at 05:50
  • @JohnRotenstein the problem is we don't have any unique identifier to determine that. Even if we send ID of state machine that would be part of the body of the sqs message and then we have to poll the messages from queue get the body and check the id of the execution which will impact performance. Can't we use the metadata of the message rhat was send to request queue like messageId, md5BodyofMessage in order to retrieve the corresponding message – Archit Agarwal Nov 01 '22 at 06:24
  • Something like: ReceiveMesageRequest r = new ReceiveMessageRequest().withQurl("").*with MessageId(" ")* – Archit Agarwal Nov 01 '22 at 06:27
  • No polling is necessary if the "return message" comes into the SQS queue and a Lambda function looks at the message, extracts the ID and then (somehow) passes the response to the appropriate state machine. But, feel free to experiment! – John Rotenstein Nov 01 '22 at 07:34
  • Surely will test this. Also will the message attributes be able to help us. I am trying to send message abc to q1 with attributes(key=1 , value = some uuid) the lambda eill do processing and when I am trying to receive a reaponse I can get the message with these attributes thus defining the execution gets correct data – Archit Agarwal Nov 01 '22 at 09:40
  • It is not possible to specifically request a message based on message attributes. They are simply metadata that are available once you receive the message. – John Rotenstein Nov 01 '22 at 10:05

0 Answers0