What is the best way to extract request and response data from a log file using Logstash and regex?

Question

I have application log file which contains the application requests and responses, the complete request and response looks like the below, I tried different patterns using RegEx but unfortunately without any luck, can some one suggest what should I change :

pattern =>

"
(?m)\b\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\b.?\bRequest given by the user\b.?\brspDesc is\b(?!.\n)..?(?=\R|$)
"

2023-05-11 00:20:26,103 [http-apr-7777-exec-46] INFO com.welcome.ws.AccountWebServiceImpl. - Request given by the user

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:GetCustomerBalanceReq xmlns="url" xmlns:ns2="url">

testt
**
xls

ns2:ID/EID/5000000004/123456****1234</ns2:ID>
</ns2:GetCustomerBalanceReq>

2023-05-11 00:20:26,144 [http-apr-7777-exec-46] INFO com.welcome.svc.AccountService. - rspCode is:1001 and rspDesc is:Account or Media does not Exist

I was trying this regex but it wasn't matching anything

"(?m)\b\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\b.?\bRequest given by the user\b.?\brspDesc is\b(?!.\n)..?(?=\R|$) 

Answer 1

To reiterate, you're saying that within the log-file there are sections which start and end with the example text you provided.

To capture these sections, you can use the following regular expression pattern.

(?s)(\d{4}-\d\d-\d\d .+?rspDesc.+?$)

• Use the s flag to enable single-line mode, causing a . to additionally match new-line characters.
• The beginning of the pattern will match the time-stamp.
• Then, match all characters up to "rspDesc".
• Finally, continue until the end of the line is reached.