1

I have flink running with RabbitMQ as source, something in my configuration could be causing unacked messages in rabbitmq and then everything is going to crash eventually. Here is my code:

Checkpoint config:

Configuration conf = new Configuration();
    conf.setInteger("rest.port", PropertyFileReader.getRestPort());
    conf.setString("web.log.path", PropertyFileReader.getFlinkDashboardLogPath());
    conf.setString("taskmanager.memory.flink.size", PropertyFileReader.getFlinkMemory());
    conf.setString("taskmanager.memory.process.size", PropertyFileReader.getFlinkMemory());
    conf.setInteger("taskmanager.data.port", PropertyFileReader.getFlinkTaskmanagerDataPort());

    StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(8, conf);
    //Environment configurations
    env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
    env.getConfig().setAutoWatermarkInterval(500);
    env.setMaxParallelism(8);
    env.setBufferTimeout(1000);
    env.getConfig().setUseSnapshotCompression(true);
    env.getConfig().disableSysoutLogging();
    env.getConfig().enableObjectReuse();
    if (PropertyFileReader.isCheckpointing_enabled()) {
        env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
        env.setStateBackend(new FsStateBackend(new Path(PropertyFileReader.getCheckpointing_path()).toUri(), true));
        env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2000);
        env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
        env.enableCheckpointing(3000, CheckpointingMode.EXACTLY_ONCE);
        env.getCheckpointConfig().setCheckpointTimeout(6000);
        env.getCheckpointConfig().setFailOnCheckpointingErrors(false);
    }
    env.setRestartStrategy(RestartStrategies.fixedDelayRestart(
            10, // number of restart attempts
            Time.of(5, TimeUnit.SECONDS) // delay
    ));

Consumer code:

connectionConfig = new RMQConnectionConfig.Builder()
            .setHost(PropertyFileReader.getRbt_host())
            .setPort(PropertyFileReader.getRbt_port())
            .setUserName(PropertyFileReader.getRbt_username())
            .setPassword(PropertyFileReader.getRbt_password())
            .setVirtualHost(PropertyFileReader.getRbt_virtualhost())
            .setTopologyRecoveryEnabled(true)
            .setAutomaticRecovery(true)
            .setRequestedChannelMax(PropertyFileReader.getMax_channel())
            .setRequestedChannelMax(PropertyFileReader.getRequested_max_channel())
            .setNetworkRecoveryInterval(1)
            .build();

 public static DataStream<String> eventStreamObject(StreamExecutionEnvironment env) {
    return env.addSource(new RMQSource<>(
            connectionConfig,
            PropertyFileReader.getRbt_queue(),
            true,
            new SimpleStringSchema()))
            .setParallelism(1);
}

and everytime i start flink this message is there:

INFO  org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase  - No state to restore for the RMQSource.

What can I try to fix this?

halfer
  • 19,824
  • 17
  • 99
  • 186
Alter
  • 903
  • 1
  • 11
  • 27
  • Please read [Under what circumstances may I add “urgent” or other similar phrases to my question, in order to obtain faster answers?](//meta.stackoverflow.com/q/326569) - the summary is that this is not an ideal way to address volunteers, and is probably counterproductive to obtaining answers. Please refrain from adding this to your questions. – halfer May 18 '20 at 22:28
  • Thanks a lot, already found the cause of the unacked messages, was a back pressure in the source and the mapping, which means that the sink is having issues to send the information out from Flink. Kind regards. – Alter May 20 '20 at 13:02

0 Answers0