Questions tagged [alpakka]

Alpakka is the collective name for various Akka Streams connectors, integration patterns, and data transformations.

Alpakka is a community-driven initiative that provides connectors, integration patterns, and data transformations that are built with Akka Streams. This toolkit is meant to be a "modern alternative to Apache Camel" (hence its name, which is a homophone of "alpaca," a relative of the camel, and was first used as a code name for an old akka-camel module).

From an introductory blog post:

Akka Streams already has a lot that are useful for integrations. Defining processing pipelines is what the Akka Streams DSL is all about and that is exactly what you need for operating on streaming data that cannot fit in memory as a whole. It handles backpressure in an efficient non-blocking way that prevents out-of-memory errors, which is a typical problem when using unbounded buffering with producers that are faster than consumers.

Connectors:

  • AMQP
  • Apache Geode
  • AWS DynamoDB
  • AWS Kinesis
  • AWS Lambda
  • AWS S3
  • AWS SNS
  • AWS SQS
  • Azure Storage Queue
  • Cassandra
  • Elasticsearch
  • File
  • FTP
  • Google Cloud Pub/Sub
  • HBase
  • IronMq
  • JMS
  • MongoDB
  • MQTT
  • Server-sent Events (SSE)
  • Slick (JDBC)
  • Spring Web

Integration Patterns:

  • Splitter

Data Transformations:

  • Parsing Lines
  • JSON
  • Compressing/decompressing
  • Comma-Separated Values (CSV)
  • RecordIO Framing
  • Extensible Markup Language (XML)

Additional Links:

217 questions
2
votes
1 answer

Processing a big table with Slick fails with OutOfMemoryError

I am querying a big MySQL table with Akka Streams and Slick, but it fails with an OutOfMemoryError. It seems that Slick is loading all the results into memory (it does not fail if the query is limited to a few rows). Why is this the case, and what…
David Portabella
  • 12,390
  • 27
  • 101
  • 182
2
votes
1 answer

Alpakka : how to deserialize xml to objects - Is there a more concise pattern for deserializers?

I have complex XML data (it might contain a lot of data and could have more than 15GB) which has a sophisticated nature with a deep structure. We need stream processing for our huge XML. Using the new Alpakka library is our first choice since it…
sergiusz.kierat
  • 139
  • 1
  • 10
2
votes
0 answers

Scala alpaakka s3 connect with DO Spaces multipart upload result 403

i am trying to setup akka-http file upload to Digital Ocean Spaces. And i have got problem with multipart upload. I use alpaakka to stream file directly to s3. However i get 403 error SignatureDoesNotMatch on upload. But if i try to upload same…
Anton M.
  • 472
  • 4
  • 14
2
votes
1 answer

Issues uploading/downloading files in akka-http/akka-streams

I'm trying to use akka-streams and akka-http and the alpakka library to download/upload files to Amazon S3. I am seeing two issues which might be related... I can only download very small files, the largest one 8kb. I can't upload larger files. It…
zerayaqob
  • 426
  • 1
  • 6
  • 12
2
votes
1 answer

Why don't Akka Streams application terminate normally?

I wrote this simple application using Alpakka Cassandra Library package com.abhi import akka.actor.ActorSystem import akka.stream.{ActorMaterializer, ClosedShape} import akka.stream.alpakka.cassandra.scaladsl.CassandraSource import…
Knows Not Much
  • 30,395
  • 60
  • 197
  • 373
2
votes
1 answer

Akka Streams, source items as another source?

I am using Alpakka-FTP, but maybe I'm looking for a general akka-stream pattern. The FTP connector can list files or retrieve them: def ls(host: String): Source[FtpFile, NotUsed] def fromPath(host: String, path: Path): Source[ByteString,…
ticofab
  • 7,551
  • 13
  • 49
  • 90
1
vote
1 answer

Batch inserts and LAST_INSERT_ID with Slick and MariaDB

I'm trying to insert some data into a MariaDB database. I got two tables and I have to insert the rows (using a batch insert) into the first table and use the IDs of the newly-inserted rows to perform a second batch insert into the second table. I'm…
madipi
  • 355
  • 2
  • 11
1
vote
0 answers

How to configure Alpakka Kafka Consumers depending on Partition Number and Akka Nodes

I have a question about Alpakka Kafka behaviour by handling partitions for consumers. For ex, how Alpakka Kafka will act in the following configuration. Let's say, I have a topicA with 4 partitions and I have 6 Akka Nodes. Normal behaviour of Kafka…
posthumecaver
  • 1,584
  • 4
  • 16
  • 29
1
vote
0 answers

Kafka does not reconsume messages after group-id has changed and offset.reset is set to earliest

I am trying to test a feature that requires reconsuming the kafka messages from an earlier offset. I changed the group id into a new one and the auto.offset.reset = "earliest", but that didn't solve the problem at all. On top of this I…
Stela Leon
  • 11
  • 2
1
vote
1 answer

How to log Kafka Producer Config on Startup

I currently have an application with inbound and outbound of kafka, however upon launch I can see the Consumer config being logged as so: INFO [o.a.k.c.c.ConsumerConfig] - ConsumerConfig values: allow.auto.create.topics = true …
Blawless
  • 1,229
  • 2
  • 16
  • 26
1
vote
0 answers

RestartFlow.onFailuresWithBackoff not restarting on failure

We are encountering an unexpected behavior from: RestartFlow.onFailuresWithBackoff(restartSetting)(flowFactory: () => Flow[In, Out, _]) See RestartFlow.onFailuresWithBackoff. We're using: akka:2.6.15 akka-stream-kafka_2.12:2.1.1…
Desanth pv
  • 351
  • 1
  • 4
  • 13
1
vote
1 answer

Akka stream best practice for dynamic Source and Flow controlled by websocket messages

I'm trying to understand what is the best way to implement with akka stream and alpakka the following scenario (simplified): The frontend opens a websocket connection with the backend Backend should wait an initialization message with some…
rivamarco
  • 719
  • 8
  • 23
1
vote
0 answers

How to distribute workload to many compute and do scatter-gather scenarios with Kafka Steam?

I am new to Kafka Stream and Alpakka Kafka. Problem: I have been using Java Executor Service to run parallel jobs and when ALL of them are done, marking the entire process done. The issue is fault tolerance, High Availability and Not utilizing all…
Bmis13
  • 550
  • 1
  • 8
  • 27
1
vote
1 answer

How to map parquet record (json) to case class using alpakka

I have the data that saved in parquet file as json e.g {"name":"john", "age":23.5} I want to convert it to case class Person(name: String, age: Double) so I can use pattern matching in my actor. This is what I got so far: val reader:…
igx
  • 4,101
  • 11
  • 43
  • 88
1
vote
1 answer

Can Throttle rate of consuming incoming messages using Akka Kafka stream

Is there a way to read only X number of message in particular time period suppose 1 minutes using Akka kafka Stream Consumer https://doc.akka.io/docs/alpakka-kafka/0.15/consumer.html via some configuration. Need to handle a situation where there is…
Bigdatas
  • 13
  • 2
1 2
3
14 15