Questions tagged [lambda-architecture]

Lambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods.

This approach to architecture attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate precomputed views, while simultaneously using real-time stream processing to provide dynamic views. The two view outputs may be joined before presentation.

34 questions
0
votes
1 answer

Can I use Apache Kafka as for Batch Layer to save historical data in Lambda Architecture?

Kafka as a storage system can be a data store for long term data. It can replicate and distribute without problem. So can I create RDD from all historical data in Kafka and create batch view then combine it with Spark Streaming Views?
0
votes
1 answer

Spark Streaming NoClassDefFoundError error

I am trying to create Spark Kafka Cassandra Integration. Now I am able to connect to cassandra but when I m trying to create SparkStreamingContext object using val ssc = new StreamingContext(sparkConf, Seconds(60)) I am able to import and write…
0
votes
1 answer

Unable to create file using Pail DFS

Newbie here. Trying to run the code from Nathan Marz's book Big Data DFS Datastore using Pail. What am I doing wrong? Trying to connect to an HDFS VM. Tried replacing hdfs with file. Any help appreciated. public class AppTest { private App app =…
Win Man
  • 929
  • 2
  • 15
  • 30
0
votes
1 answer

What is the best way to implement Lambda-architecture batch_layer and serving_layer?

If I am build a project applying Lambda-architecture now, should I split the batch layer and the serving layer, i.e. program A do the batch layer's work, program B do the serving layer's? they are physically independent but logically relevant, since…
Eric Zheng
  • 1,084
  • 1
  • 11
  • 23
1 2
3