Kafka stream: Monthly time windows

Question

Based on this example (https://github.com/confluentinc/kafka-streams-examples/blob/5.5.0-post/src/test/java/io/confluent/examples/streams/window/DailyTimeWindows.java), I would like to create a Monthly time windows. The problem is the size method which I don't know the size since every month have a different size.

For more context, I want to count each unique user who made a transaction over a month based on userId.

Actual implementation for windowsFor method:

public Map<Long, TimeWindow> windowsFor(final long timestamp) {
    final Instant instant = Instant.ofEpochMilli(timestamp);

    final ZonedDateTime zonedDateTime = instant.atZone(this.zoneId);

    final ZonedDateTime startTime = zonedDateTime.truncatedTo(ChronoUnit.DAYS).withDayOfMonth(1);
    final ZonedDateTime endTime = startTime.plusMonths(1);

    final Map<Long, TimeWindow> windows = new LinkedHashMap<>();
    windows.put(toEpochMilli(startTime), new TimeWindow(toEpochMilli(startTime), toEpochMilli(endTime)));
    return windows;
}

Is someone have an idea?

score 1 · Accepted Answer · answered Dec 09 '20 at 09:37

1

Unfortunately, calendar-based windows are currently not supported in Kafka Streams. There is a ticket requesting it.

The main issue is due to how Kafka Streams serializes time windows. In the test for the example you linked, there is a test with an explanation regarding this limitation.

answered Dec 09 '20 at 09:37

Bruno Cadonna

1,348
7
11

A rather hacky solution would be to always return 31 days in `size()` and compute the correct monthly windows in `windowsFor()`. If you have aggregates over months as results, you then need to just consider the start date of the window downstream and compute the end date manually if needed since the one provided by Kafka Streams in the time windowed key will always be 31 days after the start date. However, I do not know if this works for all cases. So, bear with me if it doesn't work. You have to try it out and be aware that it is hacky. – Bruno Cadonna Dec 09 '20 at 12:01

score 0 · Answer 2 · answered Dec 08 '20 at 18:25

The problem is the size method which I don't know the size since every month have a different size.

You can convert months to days and then add it. You will also need to take care of checking leap year.

public Map<Long, TimeWindow> windowsFor(final long timestamp) {
    final Instant instant = Instant.ofEpochMilli(timestamp);

    final ZonedDateTime zonedDateTime = instant.atZone(zoneId);
    final ZonedDateTime startTime = zonedDateTime.truncatedTo(ChronoUnit.DAYS).withDayOfMonth(1);
    final ZonedDateTime endTime = startTime.plusDays(getDays(startTime.getYear(), startTime.getMonthValue()));

    final Map<Long, TimeWindow> windows = new LinkedHashMap<>();
    windows.put(toEpochMilli(startTime), new TimeWindow(toEpochMilli(startTime), toEpochMilli(endTime)));
    return windows;
}

public static int getDays(int year, int months) {
   return YearMonth.of(year, months).lengthOfMonth();
}

The problem is with this method @Override public long size() { return Duration.ofDays(this.yearMonth.lengthOfMonth()).toMillis(); } Size doesn't take parameter, so I'm unable to know with windows / month is — Miloune, Dec 09 '20 at 08:22

Kafka stream: Monthly time windows

2 Answers2