Optimize getting the latest events by type per room

[S7evinK]

Original EXPLAIN ANALYZE

Nested Loop  (cost=24248.63..24751.93 rows=1196 width=431) (actual time=47.760..58.901 rows=548 loops=1)
  ->  HashAggregate  (cost=24248.21..24250.21 rows=200 width=8) (actual time=47.738..57.919 rows=548 loops=1)
        Group Key: max(syncv3_events_1.event_nid)
        Batches: 1  Memory Usage: 89kB
        ->  Finalize GroupAggregate  (cost=23924.27..24233.26 rows=1196 width=53) (actual time=47.159..57.736 rows=548 loops=1)
"              Group Key: syncv3_events_1.room_id, syncv3_events_1.event_type"
              ->  Gather Merge  (cost=23924.27..24203.36 rows=2392 width=53) (actual time=47.155..57.560 rows=771 loops=1)
                    Workers Planned: 2
                    Workers Launched: 2
                    ->  Sort  (cost=22924.25..22927.24 rows=1196 width=53) (actual time=43.275..43.285 rows=257 loops=3)
"                          Sort Key: syncv3_events_1.room_id, syncv3_events_1.event_type"
                          Sort Method: quicksort  Memory: 57kB
                          Worker 0:  Sort Method: quicksort  Memory: 41kB
                          Worker 1:  Sort Method: quicksort  Memory: 42kB
                          ->  Partial HashAggregate  (cost=22851.15..22863.11 rows=1196 width=53) (actual time=42.897..42.931 rows=257 loops=3)
"                                Group Key: syncv3_events_1.room_id, syncv3_events_1.event_type"
                                Batches: 1  Memory Usage: 129kB
                                Worker 0:  Batches: 1  Memory Usage: 97kB
                                Worker 1:  Batches: 1  Memory Usage: 97kB
                                ->  Parallel Seq Scan on syncv3_events syncv3_events_1  (cost=0.00..21962.23 rows=118523 width=53) (actual time=0.007..19.345 rows=94820 loops=3)
  ->  Index Scan using syncv3_events_pkey on syncv3_events  (cost=0.42..2.52 rows=1 width=431) (actual time=0.002..0.002 rows=1 loops=548)
        Index Cond: (event_nid = (max(syncv3_events_1.event_nid)))
Planning Time: 0.188 ms
Execution Time: 59.088 ms

Optimzied EXPLAIN ANALYZE

Sort  (cost=6614.15..6626.65 rows=5000 width=431) (actual time=10.170..10.191 rows=548 loops=1)
  Sort Key: syncv3_events.event_nid DESC
  Sort Method: quicksort  Memory: 255kB
  ->  Nested Loop  (cost=66.83..6306.96 rows=5000 width=431) (actual time=0.237..9.928 rows=548 loops=1)
        ->  Nested Loop  (cost=66.40..6176.68 rows=5000 width=8) (actual time=0.076..8.016 rows=2250 loops=1)
              ->  CTE Scan on t  (cost=65.26..67.28 rows=100 width=32) (actual time=0.037..0.345 rows=45 loops=1)
                    Filter: (event_type IS NOT NULL)
                    Rows Removed by Filter: 1
                    CTE t
                      ->  Recursive Union  (cost=0.42..65.26 rows=101 width=32) (actual time=0.035..0.333 rows=46 loops=1)
                            ->  Limit  (cost=0.42..0.52 rows=1 width=14) (actual time=0.034..0.034 rows=1 loops=1)
                                  ->  Index Only Scan using syncv3_events_type_sk_idx on syncv3_events syncv3_events_2  (cost=0.42..27471.67 rows=284455 width=14) (actual time=0.033..0.033 rows=1 loops=1)
                                        Heap Fetches: 1
                            ->  WorkTable Scan on t t_1  (cost=0.00..6.37 rows=10 width=32) (actual time=0.006..0.006 rows=1 loops=46)
                                  Filter: (event_type IS NOT NULL)
                                  Rows Removed by Filter: 0
                                  SubPlan 1
                                    ->  Limit  (cost=0.42..0.62 rows=1 width=14) (actual time=0.006..0.006 rows=1 loops=45)
                                          ->  Index Only Scan using syncv3_events_type_sk_idx on syncv3_events syncv3_events_1  (cost=0.42..18467.73 rows=94818 width=14) (actual time=0.005..0.005 rows=1 loops=45)
                                                Index Cond: (event_type > t_1.event_type)
                                                Heap Fetches: 44
              ->  Nested Loop  (cost=1.14..60.59 rows=50 width=8) (actual time=0.005..0.167 rows=50 loops=45)
                    ->  Seq Scan on syncv3_rooms  (cost=0.00..2.50 rows=50 width=34) (actual time=0.001..0.004 rows=50 loops=45)
                    ->  Result  (cost=1.14..1.15 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=2250)
                          InitPlan 3 (returns $5)
                            ->  Limit  (cost=0.42..1.14 rows=1 width=8) (actual time=0.003..0.003 rows=0 loops=2250)
                                  ->  Index Only Scan Backward using syncv3_events_type_room_nid_idx on syncv3_events e  (cost=0.42..171.64 rows=238 width=8) (actual time=0.003..0.003 rows=0 loops=2250)
                                        Index Cond: ((event_type = t.event_type) AND (room_id = syncv3_rooms.room_id) AND (event_nid IS NOT NULL))
                                        Heap Fetches: 548
        ->  Memoize  (cost=0.43..2.65 rows=1 width=431) (actual time=0.001..0.001 rows=0 loops=2250)
              Cache Key: ($5)
              Cache Mode: logical
              Hits: 1701  Misses: 549  Evictions: 0  Overflows: 0  Memory Usage: 272kB
              ->  Index Scan using syncv3_events_room_event_nid_type_skey_idx on syncv3_events  (cost=0.42..2.64 rows=1 width=431) (actual time=0.002..0.002 rows=1 loops=549)
                    Index Cond: (event_nid = ($5))
Planning Time: 0.311 ms
Execution Time: 10.433 ms

Note: 280k events in syncv3_events
diff query.optimized query.orig doesn't show any difference between the results.

[kegsay]

This reduces the query time to 4m from 8m on matrix.org.