4

I have a dataset that has 1 billion rows. The data is stored in Hive. Also, I put Impala as a layer between Hive and Superset. The queries that are run in Superset have row limit max. 100.000. I need to change it with no row limit. Furthermore, I need to make a visualization from what the queries return from SQL lab, but it cannot be done because there is a timeout cache limit also. Therefore, if I change/increase the row limit in SQL lab and timeout cache in visualization, then I guess, there will be no problem.

TylerH
  • 20,799
  • 66
  • 75
  • 101
ufukyılmaz
  • 51
  • 1
  • 6
  • so you need to increase limit in `superset` or `impala` ? in case of impala, if you are trying to access using hue, you can increase it for other connections/tools, probably there is no limit to # of rows. – Koushik Roy Jan 03 '22 at 05:18
  • When I run any query in impala, there is no problem. But I need to run queries in superset to visualize the output. ındeed, when using superset, it returns results that just have max. 100000 rows but when I use impala the same queries give me more than 100000 rows. So, superset do not give me the whole result that I need. Furthermore, for visualization in superset, there is also another row limit. It uses only 1000 rows of tables and if the run time of query is above 60sec, it also return error. – ufukyılmaz Jan 03 '22 at 15:09

5 Answers5

0

i am trying my best to answer below. Pls backup all config files before changing.

For SQL row limit issue -

  1. modify config.py file inside the 'anaconda3/lib/python3.7/site-packages' and set
DEFAULT_SQLLAB_LIMIT  to 1000000000
QUERY_SEARCH_LIMIT to 1000000000
  1. modify viz.py and set-
filter_row_limit  to 1000000000

For timeout issue, increase below parameter values -

For synchronous queries - change superset_config.py

SUPERSET_WEBSERVER_TIMEOUT 
SQLLAB_TIMEOUT 
SUPERSET_TIMEOUT  --This value should be >=SQLLAB_TIMEOUT

For async queries -

SQLLAB_ASYNC_TIME_LIMIT_SEC
Koushik Roy
  • 6,868
  • 2
  • 12
  • 33
  • Thank you for your response, but we tried what you said and there is nothing changed in the superset. After changing the config file, it looks like a change has been made, I mean it shows a 1billion row limit, but when I run the query, it considers the row limit as 100000 in query history. Therefore, we still have the same problem. – ufukyılmaz Jan 10 '22 at 06:39
0

I've been able to solve the problem by modifying SQL_MAX_ROW in /superset/config.py to the new maximum desired.

# Maximum number of rows returned for any analytical database query
SQL_MAX_ROW = 100000
dbaltor
  • 2,737
  • 3
  • 24
  • 36
maximilian
  • 11
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 30 '22 at 01:36
0

There must be a config parameter to change the max row limit in site-packages/superset, DEFAULT_SQLLAB_LIMIT to set the default and SQL_MAX_ROW to set the max in SQL Lab.

Tannishtha Mahindru
  • 1
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jan 20 '23 at 22:34
0

I guess we have to run superset_init again to make the change appear on Superset.

Yunnosch
  • 26,130
  • 9
  • 42
  • 54
Leo VN
  • 1
  • 1
    As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Feb 21 '23 at 23:47
0

find your superset_config.py, then add those words will solve the row limit question

SQL_MAX_ROW = your_dedired_row_number

e.g. SQL_MAX_ROW = 10000000 enter image description here

Sean.H
  • 640
  • 1
  • 6
  • 18