Retry stream in akka on failure of any stage in flow

Question

I am using akka stream to process my data. In which I have 1 Source which consists of element UUID.

The flow is as follows :

1. is fetching the Element from some third party HTTP service which returns complete Element with its properties.
2. Then I retrieve required data from that element and convert it to object that my application understands.
3. Then i write the data from that object to DB.
4. Finally, I update the DB with status of all the elements in the stream.

Now I want to add retry mechanism to this flow so that if any of the stages in flow fails it should retry the stage for some no of times say 3 and if after that it fails then the only failure of the stream should happen. For example, if there are some issues with third-party service like HTTP 504 error then most of the time after retrying this element success. So is there any way in akka to achieve this.

Currently, I am maintaining 1 list to store all the failed element ids like below.

Code :

List<UUID> failedId = new CopyOnWriteArrayList<>();
Source.from(elementIdToProcess).map(f -> {
            failedId.add(f);
            return f;
        }).via(featureFlow()).via(filterNullFeaturesFlow())
            .via(mapToFeatureFlow()).via(setFeaturesAdditionalInfo())
            .runForeach(features -> {
                features.parallelStream().forEach(feature -> {
                    try {
                        featureCommitter.save(feature);
                        failedId.remove(UUID.fromString(feature.getFeatureId()));
                    } catch (Exception e) {
                        throw new RuntimeException(e);
                    }
                });
            }, materializer);

Answer 1

You can retry like this

source.recoverWithRetries(attempts = 2, {
  case _: RuntimeException ⇒ source
})

Or you can have a back off strategy using either RestartSource, RestartSink or RestartFlow. All of this is documented in the Akka docs

Answer 2

I had a similar issue and solved it with Future feature named retry

Base on your code I would implemente it this way:

  @BeforeClass
  public static void setUpClass() {
      system = ActorSystem.create();
      mat = ActorMaterializer.create(system);
  }

  @AfterClass
  public static void tearDown() {
      TestKit.shutdownActorSystem(system);
      system = null;
  }

  @Test
  public void shouldRetryArbitraryNumberOfTimes() {
    doThrow(new RuntimeException()).when(featureCommitter).process(anyString(), anyString(), anyString());
    TestSubscriber.Probe<Message> probe = getJmsSource().runWith(TestSink.probe(system), mat);
    Throwable throwable = probe.request(1).expectError();
    verify(featureCommitter, timeout(5000).times(4)).save(any(Feature.class));
  }

  private Source<Feature> getJmsSource() {
    List<UUID> failedId = new CopyOnWriteArrayList<>();
      return Source.from(elementIdToProcess)
              .via(featureFlow())
              .via(filterNullFeaturesFlow())
              .via(mapToFeatureFlow())
              .via(setFeaturesAdditionalInfo())
              .mapAsync(1, features -> {
                      features.parallelStream().forEach(feature -> retry(getCompletionStageCallable(feature),
                                                                         3, 
                                                                         Duration.ofMillis(200),
                                                                         system.scheduler(),
                                                                         system.dispatcher());
                  });
  }

  private Callable<CompletionStage<Feature>> getCompletionStageCallable(Feature feature) {
    () -> {
      return return CompletableFuture.supplyAsync(() -> {
          try {
              featureCommitter.save(feature);
          } catch (Exception e) {
              throw new RuntimeException(e);
          }
          return feature;
      }
  }

The main thing to take into account here is that we are not handling the retry as part of the stream but rather as a way to handle the future. As you can see I moved the saving out of the Sink and into a mapAsync in which I set the parallelism to 1. This is so that I can use a TestSubscriber.Probe to validate the stream behavior.

I hope this helps.

Regards