Write AWS Glue DynamicFrame to redshift table

Question

I have a dynamic frame with following schema

root
|-- source_id: long
|-- scrape_timestamp_last: timestamp
|-- scrap_timestamp_orig: timestamp
|-- job_id_init: string
|-- post_date: timestamp
|-- date_posted: string
|-- date_offset: int
|-- target_id: long
|-- job_id: long
|-- status: string
|-- hash_001: string
|-- hash_002: string
|-- update_flag: string

I want to write this dynamic frame to a redshift table. I have created the table in redshift database with similar column names as follows:

create table redshift_table_name
(
    source_id bigint,
    target_id bigint,
    job_id_init varchar(50),
    job_id bigint,
    status varchar(50),
    scrap_timestamp_orig timestamp,
    date_posted varchar(100),
    date_offset int,
    post_date timestamp,
    scrap_timestamp_last timestamp,
    hash_001 varchar(200),
    hash_002 varchar(200),
    update_flag varchar(200)
);

I am writing this frame to this redshift table as following code snippet:

dest_table = "<redshift_schema>.<redshift_table>"

pre_actions = f"DELETE FROM {dest_table} WHERE 1=1"

datasink = glueContext.write_dynamic_frame.from_jdbc_conf(
    frame=<data_frame>,
    catalog_connection="redshift_connection",
    connection_options={
        "preactions": pre_actions,
        "dbtable": dest_table,
        "database": "<redshift_database>",
    },
    redshift_tmp_dir=args["TempDir"],
    transformation_ctx="datasink",
)

job.commit()

AWS Glue job shows the status as succeeded and there is no error in output log as well as error logs. But, when I query the table in redshift, it shows no record in the table. What could be the problem ?. I am new to Glue job and redshift and could not figure out the problem. Please help me.

Write AWS Glue DynamicFrame to redshift table

0 Answers0