1

I have a query on a large table that lasts from 2 seconds to 3 minutes depending on a parameter. Is there a way to tell to Postgres (9.6) that its query plan is sometimes very bad ?

Explanations

I have two tables:

      Table "public.ItemUpdates"
  Column    |            Type             | Modifiers 
------------+-----------------------------+----------
item_id     | integer                     | not null
type        | integer                     | not null
next_update | timestamp without time zone | not null
Indexes:
    "item_updates_item_id_unique" UNIQUE CONSTRAINT, btree (item_id)
    "ItemUpdates_type_next_update_idx" btree (type, next_update)
Foreign-key constraints:
    "fk_itemupdates_item_id" FOREIGN KEY (item_id) REFERENCES "ItemInfo"(id)

        Table "public.ItemInfo"
     Column     |      Type       |   Modifiers                      
----------------+-----------------+-----------------
id              | integer         | not null default nextval('items_id_seq'::regclass)
status          | smallint        | not null
...
and 10 more columns with text and ts_vectors (74 GB without indexes)

There are 4 ItemUpdates.types: 1, 2, 3 and 4 and the repartition is:

SELECT type, count(*) FROM "ItemUpdates" GROUP BY type

 type |  count
------+----------
    1 |  1859706
    2 |  5061595
    3 | 61973086
    4 |  2018415

The query I use is:

SELECT iu.item_id
FROM "ItemUpdates" iu
INNER JOIN "ItemInfo" i ON i.id = iu.item_id
WHERE iu.type = $1
AND iu.next_update < now()
AND i.status = 1
LIMIT 200000

With types 1, 3 and 4, the query is fast enough (less than 3 seconds). The problem is when I query on the type 2: about 4 minutes.

The query plan is different according to the type.

For 1 and 4 (2,000,000 items):

QUERY PLAN                                                                               
 Limit  (cost=28410.75..1679381.70 rows=200000 width=4) (actual time=88.930..2696.893 rows=35 loops=1)
   ->  Nested Loop  (cost=28410.75..8170462.90 rows=986335 width=4) (actual time=88.929..2696.860 rows=35 loops=1)
         ->  Bitmap Heap Scan on "ItemUpdates" iu  (cost=28410.18..459949.42 rows=993328 width=4) (actual time=79.411..2266.702 rows=78312 loops=1)
               Recheck Cond: ((type = 1) AND (next_update < now()))
               Rows Removed by Index Recheck: 14458426
               Heap Blocks: exact=30028 lossy=80441
               ->  Bitmap Index Scan on "ItemUpdates_type_next_update_idx"  (cost=0.00..28161.85 rows=993328 width=0) (actual time=73.388..73.388 rows=262838 loops=1)
                     Index Cond: ((type = 1) AND (next_update < now()))
         ->  Index Scan using "Items_pkey" on "ItemInfo" i  (cost=0.57..7.75 rows=1 width=4) (actual time=0.005..0.005 rows=0 loops=78312)
               Index Cond: (id = iu.item_id)
               Filter: (status = 1)
               Rows Removed by Filter: 1
 Planning time: 0.316 ms
 Execution time: 2696.948 ms

For 3 (62,000,000 items):

 QUERY PLAN                                                                 
 Limit  (cost=0.57..1481409.86 rows=200000 width=4) (actual time=0.202..27676.311 rows=200000 loops=1)
   ->  Nested Loop  (cost=0.57..244077225.62 rows=32952031 width=4) (actual time=0.201..27600.918 rows=200000 loops=1)
         ->  Seq Scan on "ItemUpdates" iu  (cost=0.00..1729530.37 rows=33185659 width=4) (actual time=0.015..237.416 rows=200123 loops=1)
               Filter: ((type = 3) AND (next_update < now()))
               Rows Removed by Filter: 205436
         ->  Index Scan using "Items_pkey" on "ItemInfo" i  (cost=0.57..7.29 rows=1 width=4) (actual time=0.135..0.136 rows=1 loops=200123)
               Index Cond: (id = iu.item_id)
               Filter: (status = 1)
               Rows Removed by Filter: 0
 Planning time: 0.352 ms
 Execution time: 27716.495 ms

It's longer but still OK for our use case.

And with the type 2 (5,000,000 items):

QUERY PLAN                                                                                   
Limit  (cost=583733.44..1446607.01 rows=200000 width=4) (actual time=47634.435..130378.407 rows=461 loops=1)
  ->  Hash Join  (cost=583733.44..12198399.97 rows=2692090 width=4) (actual time=47634.434..130378.148 rows=461 loops=1)
        Hash Cond: (i.id = iu.item_id)
        ->  Seq Scan on "ItemInfo" i  (cost=0.00..10591334.10 rows=70216021 width=4) (actual time=0.007..99254.593 rows=70607997 loops=1)
              Filter: (status = 1)
              Rows Removed by Filter: 497449
        ->  Hash  (cost=539252.73..539252.73 rows=2711177 width=4) (actual time=3723.732..3723.732 rows=262938 loops=1)
              Buckets: 131072  Batches: 64  Memory Usage: 1170kB
              ->  Bitmap Heap Scan on "ItemUpdates" iu  (cost=77558.13..539252.73 rows=2711177 width=4) (actual time=147.440..3642.451 rows=262938 loops=1)
                    Recheck Cond: ((type = 2) AND (next_update < now()))
                    Rows Removed by Index Recheck: 9698516
                    Heap Blocks: exact=36509 lossy=54070
                    ->  Bitmap Index Scan on "ItemUpdates_type_next_update_idx"  (cost=0.00..76880.34 rows=2711177 width=0) (actual time=140.141..140.141 rows=358860 loops=1)
                          Index Cond: ((type = 2) AND (next_update < now()))
Planning time: 0.264 ms
Execution time: 130378.583 ms

The Seq Scan on the 74 GB table is very long and costs a lot of resources.

The tables are AUTOVACUUMed and a VACUUM ANALYZE changes nothing.

To resume :

  • 2M rows to parse => OK
  • 5M rows to parse => not OK
  • 62M rows to parse => OK

Is there a way to pimp, force or hack the query plan ?

Amoki
  • 31
  • 4
  • `item_id ` is both a PK for itemUpdates and a FK into items. Which implies that there can only be one single row in itemUpdates for every item. Do you really want that? – wildplasser Aug 10 '17 at 13:01
  • Yes. The next_update is frequently updated and lot of updates on big rows implies many problems (10 hours long vacuum on table and related toast table by exemple). – Amoki Aug 10 '17 at 13:43

0 Answers0