0

I'm new to Dremio, and I was following this SQS + S3 + Dremiotutorial to learn more about Dremio. In one of the code snippets, it is mentioned that get_messages_from_queue will create a CSV file and which is later used in the upload_file method to upload into S3. However I'm missing that portion of the command which converts into CSV, can anyone help me how to create CSV using pandas? I'm new to Pandas, still learning.

SQS message body looks like this

"Body": "{\"holiday\":\"None\",\"temp\":288.28,\"rain_1h\":0.0,\"snow_1h\":0.0,\"clouds_all\":40,\"weather_main\":\"Clouds\",\"weather_description\":\"scattered clouds\",\"date_time\":\"2012-10-02 09:00:00\",\"traffic_volume\":5545}"
change198
  • 1,647
  • 3
  • 21
  • 60

1 Answers1

0

Add sqs message to dictionary and load it to panda Dataframe. Finally use to_csv to export csv file.

import pandas as pd

df = pd.DataFrame(sqs_message)
df.to_csv('sqs_messages.csv', index = False)  # pass path and file name of csv.
Marios Nikolaou
  • 1,326
  • 1
  • 13
  • 24
  • I don't think you really need a for loop in sqs_message as sqs_message doesn't have a Body key, because sqs_message is already in the right format. But you gave me the right direction to follow from data_dict. – change198 May 08 '22 at 10:33
  • @change198 yes you are right, it's a complete answer. you just need to load it to panda `Dataframe` and use `to_csv()` – Marios Nikolaou May 08 '22 at 10:36