Storing error message to Redshift through datapipeline

Question

I am trying to run a SQL activity in Redshift cluster through data pipeline. After SQL activity, few logs need be written to a Table in redshift [such as number of rows affected, the error message(if any)].

Requirement: If the sql Activity is finished successfully, the mentioned table will be written with 'error' column as null, else if the sql Activity fails on any error, that particular error message is need to be updated into the 'error' column in Redshift table.

Can we able to achieve this through pipeline? If yes, How can we achieve this?

Thanks, Ravi.

Is there any way to fetch the error raised in an activity from the next activity? — Ravi Gv, May 02 '16 at 06:26

score 0 · Answer 1 · answered May 03 '16 at 20:02

0

Unfortunately you cannot do this directly with SqlActivity in DataPipeline. The work around is to write a java program (or any executable) that does what you want and schedule it via Datapipeline using ShellCommandActivity.

answered May 03 '16 at 20:02

Ramkumar K Sugavanam

56
1

Storing error message to Redshift through datapipeline

1 Answers1