2

In Apache Druid configuration you can select the granuality of the segments (hour/day/week/etc.). What will happen if you change the granuality later? Will the new settings be applied just to the new data and old segments will be left as it is, or it will regenerated old segments too? for example, if we decide to change from day granuality to week..

Olaf Kock
  • 46,930
  • 8
  • 59
  • 90
user1258683
  • 59
  • 1
  • 7

3 Answers3

3

What will happen if you change the granuality later? Will the new settings be applied just to the new data and old segments will be left as it is

Segments are immutable, so changing the granularity will only apply to new data.

[Will] old segments will be left as it is, or it will regenerated old segments too? for example, if we decide to change from day granuality to week

Old segments will retain the granularity with which they were ingested, while new segments will be committed and published to deep storage with the updated granularity.

In other words, with your example, old segments would retain their day granularity, while new segments would be committed and published with week granularity.

user17936657
  • 131
  • 4
1

Changing the granularity does not affect the data which was stored previously. If you want you can do this with a reindex task.

Please note that there is a difference between segment granularity and query granularity.

In a nutshell, segment granularity describes how much data will be stored in 1 segment. Query granularity describes the size of the "grouped" data which is returned. These 2 can be different from each other.

For example, if you can have a segment granularity size "week", and your query granularity set to "hour". In this situation, all your data is stored in a file "per week", with as smallest dataset hour data.

If you are using PHP, you can use this package which allows you to easily compact or re-index your segments to different granularity sizes.

58k723f1
  • 579
  • 2
  • 15
0

In short, you can change segment granularity for newly created segments going forward, but other cluster features can work differently when changing segment granularity, so really, possible breakage could affect whether you "can" change it in the future.

  • Existing segments are immutable, so they retain whatever segment granularity in effect during segment creation.

    • (Of course you could overshadow those segments by replacing them with new segments of a different granularity such as through compaction, but the original segments usually stick around as unused segments, just not loaded for serving queries)

  • You are free to change the new segment granularity in the future (for creating new segments)

    • But if you want to maintain uninterrupted ingestion, you can run a test beforehand to anticipate possible ingest problems with different segment granularities.

These are the types of ingest errors that can occur when changing the segment granularity

But it is less likely you would see other problems like broken queries with different/mixed segment granularities.

  • however you could see waiting or failed compaction or other batch ingest jobs depending on the change in segment granularity and how aggressively you launch your tasks
  • Also with a very large cluster you could see problems from having too many segments (or too many tasks to process the segments) -- this is unlikely but possible
  • However, if you had slow/failing compaction tasks, they might run better with a different segment granularity
OurNewestMember
  • 41
  • 5