Heron is Twitter's 2nd generation real-time analytics platform that is fully API-compatible with Storm.
Heron is a general-purpose stream processing engine designed for speedy performance, low latency, isolation, reliability, and ease of use for developers and administrators alike. Heron was open sourced by Twitter in May 2016.
Heron's Design Goals
From the beginning, Heron was envisioned as a new kind of stream processing system, built to meet the most demanding of technological requirements, to handle even the most massive of workloads, and to meet the needs of organizations of all sizes and degrees of complexity. Amongst these requirements:
The ability to process billions of events per minute Extremely low end-to-end latency Predictable behavior regardless of scale and in the face of issue like extreme traffic spikes and pipeline congestion Simple administration, including: The ability to deploy on shared infrastructure Powerful monitoring capabilities Fine-grained configurability Easy debuggability To meet these requirements, a few core design goals have guided—and continue to guide—Heron’s development:
- Modularity
- Extensibility
- Isolation
- Constrained resource usage
- Apache Storm compatibility
- Backpressure handling
- Multiple delivery semantics
Resources
Heron Resources includes:
Videos
- Twitter Heron on Apache Aurora - #compute event @Twitter (Apr, 2016)
- Flying Faster with Heron - InfoQ (Apr, 2016)
- Twitter Heron: Stream Processing at Scale - @Scale (Sep, 2015)
- Stream Processing and Anomaly Detection - Velocity O’Reilly - Note: requires O’Reilly login (Jun, 2015)
Official Blog Posts
Community Blog Posts
- Deploying Heron on a Cluster of Machines with Apache Aurora (Supun Kamburugamuve, Jun, 2016)
- Setting up Heron Locally with Apache Aurora (Pulasthi Supun, Jun, 2016)
Slides
- Real-Time Analytics: Algorithms and Systems - Twitter University (May, 2016)
- Stream Processing and Anomaly Detection - Velocity O’Reilly (Jun, 2015)
Press
- Heron, Twitter’s Data Streaming Platform, Has Been Open Sourced (BenZinga, Jun, 2016)
- Twitter Open Sources Heron – Data Streaming For Dummies (Forbes, Jun, 2016)
- Getting Started with Heron on Apache Mesos and Apache Kafka (All Things Hadoop, May, 2016)
- Twitter open-sources Heron, its real-time stream-processing engine (VentureBeat, May, 2016)
- Twitter’s Heron Will Start a New Chapter in Real-Time Streaming (Forbes, Jun, 2015)
- Twitter Has Replaced Storm with Heron (InfoQ, Jun, 2015)
