Questions tagged [alpakka]

Alpakka is the collective name for various Akka Streams connectors, integration patterns, and data transformations.

Alpakka is a community-driven initiative that provides connectors, integration patterns, and data transformations that are built with Akka Streams. This toolkit is meant to be a "modern alternative to Apache Camel" (hence its name, which is a homophone of "alpaca," a relative of the camel, and was first used as a code name for an old akka-camel module).

From an introductory blog post:

Akka Streams already has a lot that are useful for integrations. Defining processing pipelines is what the Akka Streams DSL is all about and that is exactly what you need for operating on streaming data that cannot fit in memory as a whole. It handles backpressure in an efficient non-blocking way that prevents out-of-memory errors, which is a typical problem when using unbounded buffering with producers that are faster than consumers.

Connectors:

  • AMQP
  • Apache Geode
  • AWS DynamoDB
  • AWS Kinesis
  • AWS Lambda
  • AWS S3
  • AWS SNS
  • AWS SQS
  • Azure Storage Queue
  • Cassandra
  • Elasticsearch
  • File
  • FTP
  • Google Cloud Pub/Sub
  • HBase
  • IronMq
  • JMS
  • MongoDB
  • MQTT
  • Server-sent Events (SSE)
  • Slick (JDBC)
  • Spring Web

Integration Patterns:

  • Splitter

Data Transformations:

  • Parsing Lines
  • JSON
  • Compressing/decompressing
  • Comma-Separated Values (CSV)
  • RecordIO Framing
  • Extensible Markup Language (XML)

Additional Links:

217 questions
3
votes
1 answer

Play Framework 2.6 Alpakka S3 File Upload

I use Play Framework 2.6 (Scala) and Alpakka AWS S3 Connector to upload files asynchronously to S3 bucket. My code looks like this: def richUpload(extension: String, checkFunction: (String, Option[String]) => Boolean, cannedAcl: CannedAcl, bucket:…
cutoffurmind
  • 497
  • 8
  • 19
2
votes
0 answers

Alpakka search in elasticsearch without scrolling

can someone help me how can I search without using the scroll feature inside the elastic with alpakka. I want to use alpakka or akka-stream to reactively send a query to elastic without scrolling, But the only thing I found was "ElasticsearchSource"…
sadegh
  • 21
  • 3
2
votes
0 answers

alpakka kafka auto offset reset config latest does not read messages in test case

I have a java play application that leverages akka streams to read from kafka and output to a websocket. my application creates a kafka source using the…
mradey
  • 202
  • 1
  • 12
2
votes
0 answers

alpakka s3 source cause akka stream to crash with TcpIdleTimeoutException: TCP idle-timeout encountered on connection

I am reading a file from S3 using alpakka s3 client import akka.stream.alpakka.s3.scaladsl.S3 S3.download(bckt, bcktKey).flatMapConcat{ case Some((file, _)) => Foo }.mapAsync(1){ f=> Future{ Thread.sleep(1000) }.recover{ e => …
igx
  • 4,101
  • 11
  • 43
  • 88
2
votes
0 answers

presigned url for aws s3 in akka java application

I am using Akka framework and writing a Java application that talks to AWS S3. Is there any example of generating a pre-signed get request for s3 in java using Akka framework? Alpakka S3Client does not have any method in its interface to create a…
2
votes
1 answer

alpakka s3 file download problems with Play Framework 2.8

I have a simple alpakka s3 file downloading on top of Play framework 2.8, the code is like this: in S3 service: def download(link: String): Source[Option[(Source[ByteString, NotUsed], ObjectMetadata)], NotUsed] = { …
cutoffurmind
  • 497
  • 8
  • 19
2
votes
2 answers

Alpakka S3 connection issue

I'm trying to use Alpakka S3 to connect to a minio instance in order to store files, but I'm running into problems since I upgraded the library version from 1.1.2 to 2.0.0. Here is a simple service class with just two methods that try to create a…
schub
  • 912
  • 1
  • 8
  • 26
2
votes
0 answers

How to read values from kafka topic using consumer through akka-streams/alpakka-kafka?

Running with a Consumer.plainSource, nothing happens. Isn't this one way of reading from kafka topic through streams . val consumerSettings2 = ConsumerSettings(system,new StringDeserializer,new StringDeserializer) …
supernatural
  • 1,107
  • 11
  • 34
2
votes
1 answer

What is the passthrough used for in alpakka-kafka connector while producing messages?

This is the code for producing a single message to Kafka given in the doc https://doc.akka.io/docs/alpakka-kafka/current/producer.html val single: ProducerMessage.Envelope[KeyType, ValueType, PassThroughType] = ProducerMessage.single( new…
supernatural
  • 1,107
  • 11
  • 34
2
votes
0 answers

Performing efficient external system queries during kafka stream processing

I would like to consume an event from a Kafka topic, inject it into a database, perform some queries, remove the event from the database and produce the results of the query back to a topic. This looks like what was asked in External system queries…
2
votes
0 answers

Akka-stream-kafka Increasing consumers in consumer group trigger a rebalance which causes a CommitFailedException because of revoked partition

I think this issue is related to #539 but I don't know if it is a bug, or the user is supposed to handle it himself. So I have a consumer group, whenever i increase the number of consumer in that group, the revoking of partition is causing the…
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127
2
votes
2 answers

Change a materialized value in a source using the contents of the stream

Alpakka provides a great way to access dozens of different data sources. File oriented sources such as HDFS and FTP sources are delivered as Source[ByteString, Future[IOResult]. However, HTTP requests via Akka HTTP are delivered as entity streams of…
David Weber
  • 1,965
  • 1
  • 22
  • 32
2
votes
0 answers

Alpakka - Akka-Http failure while downloading file with latest S3 Connector

we are using alpakka latest S3 connector for downloading stream but it fails with max-length error as exceeded content length limit (8388608 bytes)! You can configure this by setting `akka.http.[server|client].parsing.max-content-length` or calling…
Learner
  • 45
  • 1
  • 6
2
votes
1 answer

Alpakka S3 connector stream won't handle the load, throwing akka.stream.BufferOverflowException

I have an akka-http service and I am trying out the alpakka s3 connector for uploading files. Previously I was using a temporary file and then uploading with Amazon SDK. This approach required some adjustments for Amazon SDK to make it more scala…
Michał Kreft
  • 548
  • 3
  • 16
2
votes
4 answers

Akka Stream Kafka: No configuration setting found for key 'kafka-clients'

I'm trying to create a simple prototype using Alpakka Kafka connector (Akka Stream Kafka). When running the application I get the following error: com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'kafka-clients' I…
Davide Icardi
  • 11,919
  • 8
  • 56
  • 77
1
2
3
14 15