0

TLDR; Is there any way to prioritize the kafka-engine import threads over any other CH threads OR can i reserve CPUs for the kafka consumers?

In my setup, the kafkalag increases too much when issuing a big query. I guess, this is because the import thread doesn't receive enough CPU time when there is too much CPU load. I tried to set a maximum thread cap for users as well as setting nice values. Nothing seems to work, so any advice is welcome.

the_dude
  • 11
  • 2

1 Answers1

0

upgrade to 20.9.7.11

recreate kafka engine tables with settings kafka_num_consumers=5(10), kafka_thread_per_consumer=1

add to default profile (users.xml) background_schedule_pool_size=30

Denny Crane
  • 11,574
  • 2
  • 19
  • 30
  • Can you provide any documentation to the option background_processing_pool_thread=30? i could not find anything related to that except a russian telegram channel. – the_dude Feb 01 '21 at 09:50
  • yeah, i was wrong. *background_schedule_pool_size* . here https://t.me/clickhouse_ru/182198 https://t.me/clickhouse_ru/182200 mikhail is the maintainer of kafka engine – Denny Crane Feb 01 '21 at 13:38