Resource limit exceeded in druid groupBy Query

Question

I am trying to run groupBy Query above the limit of 500k data. I am getting this error.

{
    "error": "Resource limit exceeded",
    "errorMessage": "Not enough dictionary space to execute this query. Try increasing druid.query.groupBy.maxMergingDictionarySize or enable disk spilling by setting druid.query.groupBy.maxOnDiskStorage to a positive number.",
    "errorClass": "io.druid.query.ResourceLimitExceededException",
    "host": "ubuntu:8083"
}

How can i use groupBy Query above the limit of 500k data? I am using druid 0.11.0 version and groupBy v2 engine. I have 1 million row data that i am running query on it.

I have tried increasing the druid.query.groupBy.maxMergingDictionarySize and druid.query.groupBy.maxOnDiskStorage but this does not solve my problem.

EDIT

When i tried increasing druid.processing.buffer.sizeBytes this error occurs in linux terminal.

5 errors
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050) ~[guice-4.1.0.jar:?]
        at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:154) ~[druid-api-0.11.0.jar:0.11.0]
        at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:101) [druid-services-0.11.0.jar:0.11.0]
        at io.druid.cli.ServerRunnable.run(ServerRunnable.java:50) [druid-services-0.11.0.jar:0.11.0]
        at io.druid.cli.Main.main(Main.java:108) [druid-services-0.11.0.jar:0.11.0]

There must be more errors in the log files of druid – mdeora Jul 12 '18 at 13:19 — mdeora, Jul 12 '18 at 13:19

score 1 · Answer 1 · answered Jul 10 '18 at 13:25

1

For this you need to increase the buffer sizes on all historical and realtime nodes and broker nodes.

druid.processing.buffer.sizeBytes

Also increase the druid.query.groupBy.maxMergingDictionarySize and druid.query.groupBy.maxOnDiskStorage

And double check that your machines are not running out of disk.

answered Jul 10 '18 at 13:25

mdeora

4,152
2
19
29

when i increase maxOnDiskStorage and when i run query on postman it crashes. – Salman S Jul 11 '18 at 05:09
What is your Druid version ? – mdeora Jul 11 '18 at 07:55
version that i am using is 0.11.0 – Salman S Jul 11 '18 at 08:51
Can you see the druid logs and see whats there after increasing the above mentioned sizes. – mdeora Jul 11 '18 at 13:36
2

Where do i have to change the configuration in conf-quickstart or conf ??? I am still confused about this – Salman S Jul 16 '18 at 06:51
If you are running druid in cluster mode than it should be in conf, if in single machine mode than in conf-quickstart. – mdeora Jul 16 '18 at 16:01
i got to run the query but its too slow. Suggest any performance tuning – Salman S Jul 17 '18 at 06:19
share your query and configuration and data details, may be than I can help. You can check some benchmark here - http://druid.io/blog/2014/03/17/benchmarking-druid.html – mdeora Jul 17 '18 at 12:56
sorry my bad i was using postman to get response directly, when i saved the response to a file then its speed is good. thanks for your valuable time – Salman S Jul 19 '18 at 06:09

Resource limit exceeded in druid groupBy Query

1 Answers1