DataWarehousing design patterns for event-driven semi-structured data

Question

We want to be able to analyze ticket events and trigger internal webhooks in realtime from Zendesk. Zendesk provides an AWS EventBridge connector which we were thinking of leveraging for this use-case.

For the analytical use-case, I was thinking of dumping events from EventBridge into Redshift. In order to do this, I'll create the table schemas up-front for each Zendesk event-type, and then transform the event into that schema with some eventbridge listener. I was thinking of downsides here, and a few came to mind:

1. I have the option to keep only the fields I care about. If down the line, I ever need a field, then my only option is to re-query Zendesk and obtain the data needed.
2. If the Zendesk API changes, then my pipeline may break. It seems like I might be in an endless chase to be in sync with Zendesk schema.

Another option I thought of was to dump the EventBridge output into S3, and then use Glue to compact + crawl the data. I can then query this with AWS Athena (instead of redshift). In this approach, I get the upside that I'll have all fields from the event if I ever need them, and I won't have pipeline breakages. However, I see the following downsides:

1. Analytical scripts / dashboards could still break since schemas changed.
2. In our experience, we've found glue + athena to have poor performance, and also to be expensive. It's also not realtime, glue-jobs will be running every so often.

For the Webhook triggering, I was thinking of outputting the Eventbridge events to SQS (stronger delivery / retry guarantees), and having a listener that processes the events and triggers the webhooks accordingly.

Thoughts here? What other design patterns are common for this kind of use-case?

Answer 1

For your use case of analyzing ticket events and triggering internal webhooks in real-time from Zendesk, your considerations for leveraging AWS EventBridge, Redshift, S3, Glue, and SQS are reasonable. Here are some thoughts and alternative design patterns you can consider:

1. Dumping Events into Redshift:

   • Advantage: Redshift provides a scalable and performant data warehousing solution, allowing you to run complex analytical queries.
   • Downside: As you mentioned, keeping only the fields you care about may limit your ability to query additional fields in the future. Staying in sync with Zendesk's schema changes can be challenging.
   • Alternative: Instead of dumping all events into Redshift, consider using a data lake approach by leaving the events in S3 as Avro, Parquet, or JSON files. This way, you retain the complete event payload for future analysis without being constrained by predefined schemas.

2. Using Glue and Athena for Data Processing:

   • Advantage: Glue and Athena can provide a serverless and cost-effective way to process and query data stored in S3.
   • Downside: Performance limitations and potential schema changes can impact analytical scripts and dashboards.
   • Alternative: You can use AWS Glue for ETL (Extract, Transform, Load) jobs to transform and partition your event data in a format optimized for querying with Athena. This way, you have more control over schema changes and data organization, improving the performance and stability of your analytical workflows.

3. Webhook Triggering with SQS:

   • Advantage: SQS provides reliable message delivery and retries mechanisms for triggering webhooks.
   • Alternative: Consider using AWS Lambda with an EventBridge trigger instead of SQS. This allows you to process events directly with Lambda functions, which can trigger webhooks or perform other actions. Using Lambda provides a more event-driven and real-time approach without an additional queue.

What to consider further ?

• Event Sourcing: Instead of relying solely on downstream event processing, you can consider implementing an event sourcing pattern. This involves capturing the events in a persistent event store and using them as the source of truth for both analytics and triggering webhooks. This way, you have a historical log of all events and can rebuild the state or derive new insights from the event stream if needed.
• Schema Evolution: As you mentioned, changes in Zendesk's schema may impact your pipelines. Consider implementing schema versioning and evolution techniques such as schema registry, forward compatibility, and backward compatibility checks to handle changes more gracefully.