Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-agent]

32 questions
1
vote
0 answers

No need to Dynamodb client while consuming data from Kinesis using kcl

I am using kcl api version 2 ,and dont want to use Dynamodb Client for storing the records . private static final Logger LOG = LoggerFactory.getLogger(DisplayConsumerApplication.class); public static void main(String... args) { …
Chintamani
  • 1,076
  • 7
  • 23
  • 47
1
vote
2 answers

Kinesis agent not sending .log files through firehose

I've setup a Kinesis firehose and the installed the Kinesis agent as described in the AWS docs. I can get test data through to the S3 bucket, but the Kinesis agent won't send any .log files through. I suspect a problem connecting the agent to the…
Dean Schulze
  • 9,633
  • 24
  • 100
  • 165
1
vote
0 answers

Kinesis Java consumer failing to post or get messages using sdk 2.0

I have followed https://docs.amazonaws.cn/en_us/streams/latest/dev/kcl2-standard-consumer-java-example.html to write a simple java kinesis client.The credentials used have full admin access. When executing the code getting the below…
sallu
  • 376
  • 1
  • 6
1
vote
0 answers

How to send data to aws kinesis firehose using kinesis agent

I am trying to send below data to aws kinesis firehose using kinesis agent: 3985199||"ER1AA"|"2017"|"BMC PUBLIC HEALTH"|"17"||"2017-04-09-00.53.55.353280 -0400"|0|"1471-2458"|"13941J"|"C ,CC,D ,ES,JS,Q ,XC"|"A"|"N"|"APR 4 17" I want to use 15…
Prakash
  • 45
  • 4
  • 11
1
vote
0 answers

Sent APACHEERRORLOG to Elasticsearch via amazon-kinesis-agent

I managed to send my Apache access_log (httpd / amazon-linux-ami 2016.09 / php5.6) to elasticsearch via kinesis-agent and firehose. Now I'm struggling to forward the error_log. My agent.json: { "cloudwatch.emitMetrics": true, …
Camillo
  • 544
  • 1
  • 6
  • 24
0
votes
1 answer

What determines the speed at which the aws-kinesis-agent processes and sends data to Kinesis?

There is information on the Kinesis Firehose configuration in AWS - however not finding anything on the data process speed for the aws-kinesis-agent Reference: AWS docs GitHub
Leon Africa
  • 509
  • 6
  • 11
0
votes
1 answer

AWS-Kinesis-Agent in custom AMI and on-premise: The security token included in the request is invalid

I hope someone can help me on this as i am rather new to kinesis firehose and the firehose agent. I have compiled kinesis-agent for my on-premise debian server and ec2 debian instance (in a test aws account). In a seperate aws account I have…
yoyomonkey
  • 1
0
votes
1 answer

Java KCL SPark Streaming can't import org.apache.spark.streaming.kinesis

Hello guys I'm trying to perform a KCL within a Streaming Spark job, I have created a first draft project and along with my dependencies I might be able to import few packages. However there is the package org.apache.spark.streaming.kinesis that…
Andres Urrego Angel
  • 1,842
  • 7
  • 29
  • 55
0
votes
1 answer

What is the difference between AWS Transcribe > Streaming Transcription feature and Kinesis Video Streams(For Audio Input) for live streaming audio

Hi My requirement is I have live audio stream as input, say a call between 2 people, now to convert that audio to text on live and pick certain keywords from that extracted text and insert in Database. As per architecture in…
sudhir tataraju
  • 1,159
  • 1
  • 14
  • 30
0
votes
0 answers

How KCL internally spawns new RecordProcessor

I have a java consumer (KCL implementation) to consume messages from 1 Kinesis stream with 2 shards. The KCL creates 2 RecordProcessors correlates to each shard. Then I put my machine to sleep, so JVM suspended. When wake up, the KCL library…
ethan
  • 1,881
  • 2
  • 17
  • 31
0
votes
3 answers

Load from S3 into Redshift using Kinesis

I receive my source files in S3, i need to load this data into redshift. The source S3 files are populated via Kinesis Firehose. Source : via Kinesis Webstreams -------------- > S3 Requirement : via Kinesis …
Srivignesh KN
  • 452
  • 8
  • 22
0
votes
1 answer

Kinesis Agent Input - Log file created for each day

New Log file is created for each day with date appended in the name. For example "data_log-2017-05-14" log file is created for 14th May. I need to set up Kinesis Agent to read data from the logs generated. But since each day a new log file is…
shruti
  • 587
  • 4
  • 6
0
votes
1 answer

Kinesis agent install in docker fails on missing mirrors

I am trying to run a dockerfile that has worked for the past few months and since yesterday it started giving errors when using it on a new version of the same product. This is my dockerfile: FROM centos:7 MAINTAINER email@example.com # Install…
Alfalfamale
  • 63
  • 5
0
votes
1 answer

Kinesis Agent not sending records to Stream

I have built a Kinesis Firehose stream to push data into redshift and am trying to push data from an EC2 instance using kinesis agent. Firehose Stream is able to parse the records but not identify the firehose streams am getting the following java…
Srivignesh KN
  • 452
  • 8
  • 22
0
votes
1 answer

Kinesis Firehose load csv data into Redshift

I am using Kinesis firehose to process data into redshift and i am trying both Json and Csv formats. The Json format works fine for me and data is getting loaded into redshift table. COPY COMMAND USED FOR JSON : COPY products_json FROM…
Srivignesh KN
  • 452
  • 8
  • 22
1
2
3