I am using AWS Kinesis Streaming Application to process streaming data.
The application is running but I found nothing was sank to my Redis. Related code:
public class SpyProcessor {
private static final Logger log = LogManager.getLogger(SpyProcessor.class);
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Process logic
......
//Redis sink
FlinkJedisPoolConfig conf = new FlinkJedisPoolConfig.Builder().setHost("my-redis-host").setPort(6379).build();
aggStream.addSink(new RedisSink<>(conf, new RedisSinkExample()));
log.info("Redis sink added");
env.execute("Kinesis Data Analytics Flink Application with Session Window and Aggregate Function");
public class RedisSinkExample implements RedisMapper<SpyViewCount> {
private static final Logger log = LogManager.getLogger(RedisSinkExample.class);
// private Jedis jedis;
@Override
public RedisCommandDescription getCommandDescription() {
return new RedisCommandDescription(RedisCommand.SET, null);
}
@Override
public String getKeyFromData(SpyViewCount spyViewCount) {
log.info("sinking key: " + spyViewCount.getUserId() + "_" + spyViewCount.getElement());
return spyViewCount.getUserId() + "_" + spyViewCount.getElement();
}
@Override
public String getValueFromData(SpyViewCount spyViewCount) {
log.info("sinking value: " + spyViewCount.getCount());
return spyViewCount.getCount().toString();
}
}
@Data
@AllArgsConstructor
public class SpyViewCount {
private String userId;
private Long element;
private Long count;
private Long windowEnd;
}
My Pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<java.version>1.11</java.version>
<kda.version>2.0.0</kda.version>
<kda.runtime.version>1.2.0</kda.runtime.version>
<flink.version>1.15.2</flink.version>
<seel-common-version>1.1.54</seel-common-version>
<kinesis.analytics.flink.version>2.0.0</kinesis.analytics.flink.version>
<kinesis.analytics.runtime.version>1.2.0</kinesis.analytics.runtime.version>
<maven.build.timestamp.format>yyyy_MM_dd_HH_mm_ss</maven.build.timestamp.format>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bom</artifactId>
<version>1.11.903</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-redis_2.10</artifactId>
<version>1.1.5</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kinesis</artifactId>
<version>1.15.2</version>
<exclusions>
<exclusion>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.17.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.17.1</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-kinesisanalytics-runtime</artifactId>
<version>${kinesis.analytics.runtime.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-kinesisanalytics-flink</artifactId>
<version>${kinesis.analytics.flink.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-logs</artifactId>
</dependency>
<dependency>
<groupId>com.seel</groupId>
<artifactId>merchant-service-api</artifactId>
<version>1.2.80</version>
</dependency>
<dependency>
<groupId>com.seel</groupId>
<artifactId>seel-common</artifactId>
<version>${seel-common-version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-kinesisanalytics-runtime</artifactId>
<version>${kda.runtime.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-kinesisanalytics-flink</artifactId>
<version>${kda.version}</version>
</dependency>
<dependency>
<groupId>org.example</groupId>
<artifactId>instant-variable</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>compile</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<release>11</release>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.flink:force-shading</exclude>
<exclude>com.google.code.findbugs:jsr305</exclude>
<exclude>org.slf4j:*</exclude>
<exclude>log4j:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>
com.seel.quasi.spy.SpyProcessor
</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<finalName>spy_${maven.build.timestamp}</finalName>
</build>
</project>
I can see in Cloudwatch Logs that the application is processing streaming data. However, no a single data sank to my redis server.
Hope someone can help..
Data successfully sink to redis server
