Improve "IFDB Recommends" performance, adding $count_all_possible_rows parameter to doSearch

[dfabulich]

This is rebased from @alice-blue's PR #1261, with a couple of added commits from me to improve "IFDB Recommends" performance on the home page.

Note that if you click "More recommendations" and search for played:no willplay:no wontplay:no reviewed:no rated:no, that is still doing a full scan, because it's using sql_calc_found_rows to count the total number of search results for pagination.

This PR is purely about home page queries (which are more frequent and therefore more important)

Logged out

Before, full scan

MariaDB [ifdb]> explain select
    ->     distinct games.id as id,
    ->             games.title as title,
    ->             games.author as author,
    ->             games.desc as description,
    ->             games.tags as tags,
    ->             games.moddate as moddate,
    ->             games.system as devsys,
    ->             if (time(games.published) = '00:00:00',
    ->                 date_format(games.published, '%Y'),
    ->                 date_format(games.published, '%M %e, %Y'))
    ->                 as pubfmt,
    ->             if (time(games.published) = '00:00:00',
    ->                 date_format(games.published, '%Y'),
    ->                 date_format(games.published, '%Y-%m-%d'))
    ->                 as published,
    ->             date_format(games.published, '%Y') as pubyear,
    ->             (games.coverart is not null) as hasart,
    ->             avgRating as avgrating,
    ->             numRatingsInAvg as ratingcnt,
    ->             stdDevRating as ratingdev,
    ->             numRatingsTotal,
    ->             numMemberReviews,
    ->             starsort,
    ->             games.sort_title as sort_title,
    ->             games.sort_author as sort_author,
    ->             ifnull(games.published, '9999-12-31') as sort_pub,
    ->             games.pagevsn,
    ->             games.flags
    ->
    -> from
    ->     games
    ->                 left join gameRatingsSandbox0_mv on games.id = gameid
    ->
    -> where
    ->     1
    ->
    ->
    ->
    -> order by
    ->     starsort desc
    ->
    -> limit 0, 12;
+------+-------------+------------------------+--------+---------------+---------+---------+---------------+-------+---------------------------------+
| id   | select_type | table                  | type   | possible_keys | key     | key_len | ref           | rows  | Extra                           |
+------+-------------+------------------------+--------+---------------+---------+---------+---------------+-------+---------------------------------+
|    1 | SIMPLE      | games                  | ALL    | NULL          | NULL    | NULL    | NULL          | 13984 | Using temporary; Using filesort |
|    1 | SIMPLE      | gameRatingsSandbox0_mv | eq_ref | PRIMARY       | PRIMARY | 130     | ifdb.games.id | 1     |                                 |
+------+-------------+------------------------+--------+---------------+---------+---------+---------------+-------+---------------------------------+

After, fast index lookup

MariaDB [ifdb]> explain select
    ->     distinct games.id as id,
    ->             games.title as title,
    ->             games.author as author,
    ->             games.desc as description,
    ->             games.tags as tags,
    ->             games.moddate as moddate,
    ->             games.system as devsys,
    ->             if (time(games.published) = '00:00:00',
    ->                 date_format(games.published, '%Y'),
    ->                 date_format(games.published, '%M %e, %Y'))
    ->                 as pubfmt,
    ->             if (time(games.published) = '00:00:00',
    ->                 date_format(games.published, '%Y'),
    ->                 date_format(games.published, '%Y-%m-%d'))
    ->                 as published,
    ->             date_format(games.published, '%Y') as pubyear,
    ->             (games.coverart is not null) as hasart,
    ->             avgRating as avgrating,
    ->             numRatingsInAvg as ratingcnt,
    ->             stdDevRating as ratingdev,
    ->             numRatingsTotal,
    ->             numMemberReviews,
    ->             starsort,
    ->             games.sort_title as sort_title,
    ->             games.sort_author as sort_author,
    ->             ifnull(games.published, '9999-12-31') as sort_pub,
    ->             games.pagevsn,
    ->             games.flags
    ->
    -> from
    ->     games
    ->                 join gameRatingsSandbox0_mv on games.id = gameid
    ->
    -> where
    ->     1
    ->
    ->
    ->
    -> order by
    ->     starsort desc
    ->
    -> limit 0, 12;
+------+-------------+------------------------+--------+---------------+----------+---------+------------------------------------+------+-----------------+
| id   | select_type | table                  | type   | possible_keys | key      | key_len | ref                                | rows | Extra           |
+------+-------------+------------------------+--------+---------------+----------+---------+------------------------------------+------+-----------------+
|    1 | SIMPLE      | gameRatingsSandbox0_mv | index  | PRIMARY       | starsort | 9       | NULL                               | 12   | Using temporary |
|    1 | SIMPLE      | games                  | eq_ref | PRIMARY       | PRIMARY  | 130     | ifdb.gameRatingsSandbox0_mv.gameid | 1    |                 |
+------+-------------+------------------------+--------+---------------+----------+---------+------------------------------------+------+-----------------+

Logged in

Before, full scan

MariaDB [ifdb]> explain select sql_calc_found_rows
    ->     distinct games.id as id,
    ->             games.title as title,
    ->             games.author as author,
    ->             games.desc as description,
    ->             games.tags as tags,
    ->             games.moddate as moddate,
    ->             games.system as devsys,
    ->             if (time(games.published) = '00:00:00',
    ->                 date_format(games.published, '%Y'),
    ->                 date_format(games.published, '%M %e, %Y'))
    ->                 as pubfmt,
    ->             if (time(games.published) = '00:00:00',
    ->                 date_format(games.published, '%Y'),
    ->                 date_format(games.published, '%Y-%m-%d'))
    ->                 as published,
    ->             date_format(games.published, '%Y') as pubyear,
    ->             (games.coverart is not null) as hasart,
    ->             avgRating as avgrating,
    ->             numRatingsInAvg as ratingcnt,
    ->             stdDevRating as ratingdev,
    ->             numRatingsTotal,
    ->             numMemberReviews,
    ->             starsort,
    ->             games.sort_title as sort_title,
    ->             games.sort_author as sort_author,
    ->             ifnull(games.published, '9999-12-31') as sort_pub,
    ->             games.pagevsn,
    ->             games.flags
    ->
    -> from
    ->     games
    ->                 left join gameRatingsSandbox0_mv on games.id = gameid left join playedgames as pg on games.id = pg.gameid and pg.userid = 'oyrrw74upu8n2dds' left join wishlists as wl on games.id = wl.gameid and wl.userid = 'oyrrw74upu8n2dds' left join unwishlists as ul on games.id = ul.gameid and ul.userid = 'oyrrw74upu8n2dds' left join reviews as reviewed on games.id = reviewed.gameid and reviewed.review is not null and reviewed.userid = 'oyrrw74upu8n2dds' left join reviews as rated on games.id = rated.gameid and rated.rating is not null and rated.userid = 'oyrrw74upu8n2dds'
    ->
    -> where
    ->     pg.gameid is null AND wl.gameid is null AND ul.gameid is null AND reviewed.gameid is null AND rated.gameid is null
    ->
    ->
    ->
    -> order by
    ->     starsort desc
    ->
    -> limit 0, 12;
+------+-------------+------------------------+------------+-------------------------+---------------+---------+---------------------+--------+-------------------------------------------------------+
| id   | select_type | table                  | type       | possible_keys           | key           | key_len | ref                 | rows   | Extra                                                 |
+------+-------------+------------------------+------------+-------------------------+---------------+---------+---------------------+--------+-------------------------------------------------------+
|    1 | SIMPLE      | games                  | ALL        | NULL                    | NULL          | NULL    | NULL                | 13984  | Using temporary; Using filesort                       |
|    1 | SIMPLE      | gameRatingsSandbox0_mv | eq_ref     | PRIMARY                 | PRIMARY       | 130     | ifdb.games.id       | 1      |                                                       |
|    1 | SIMPLE      | pg                     | ref|filter | userid,gameid           | gameid|userid | 130|130 | ifdb.games.id       | 7 (0%) | Using where; Not exists; Distinct; Using rowid filter |
|    1 | SIMPLE      | wl                     | ref        | gameid,userid           | userid        | 130     | const               | 1      | Using where; Not exists; Distinct                     |
|    1 | SIMPLE      | ul                     | ref|filter | gameid,userid           | gameid|userid | 130|130 | ifdb.games.id       | 1 (0%) | Using where; Not exists; Distinct; Using rowid filter |
|    1 | SIMPLE      | reviewed               | ref        | userid,gameid,user_game | user_game     | 260     | const,ifdb.games.id | 1      | Using where; Not exists; Distinct                     |
|    1 | SIMPLE      | rated                  | ref        | userid,gameid,user_game | user_game     | 260     | const,ifdb.games.id | 1      | Using where; Not exists; Distinct                     |
+------+-------------+------------------------+------------+-------------------------+---------------+---------+---------------------+--------+-------------------------------------------------------+

After, fast index lookup

MariaDB [ifdb]> explain select
    ->     distinct games.id as id,
    ->             games.title as title,
    ->             games.author as author,
    ->             games.desc as description,
    ->             games.tags as tags,
    ->             games.moddate as moddate,
    ->             games.system as devsys,
    ->             if (time(games.published) = '00:00:00',
    ->                 date_format(games.published, '%Y'),
    ->                 date_format(games.published, '%M %e, %Y'))
    ->                 as pubfmt,
    ->             if (time(games.published) = '00:00:00',
    ->                 date_format(games.published, '%Y'),
    ->                 date_format(games.published, '%Y-%m-%d'))
    ->                 as published,
    ->             date_format(games.published, '%Y') as pubyear,
    ->             (games.coverart is not null) as hasart,
    ->             avgRating as avgrating,
    ->             numRatingsInAvg as ratingcnt,
    ->             stdDevRating as ratingdev,
    ->             numRatingsTotal,
    ->             numMemberReviews,
    ->             starsort,
    ->             games.sort_title as sort_title,
    ->             games.sort_author as sort_author,
    ->             ifnull(games.published, '9999-12-31') as sort_pub,
    ->             games.pagevsn,
    ->             games.flags
    ->
    -> from
    ->     games
    ->                 join gameRatingsSandbox0_mv on games.id = gameid
    ->
    -> where
    ->     gameid not in (select gameid from playedgames where userid = 'oyrrw74upu8n2dds') AND gameid not in (select gameid from wishlists where userid = 'oyrrw74upu8n2dds') AND gameid not in (select gameid from unwishlists where userid = 'oyrrw74upu8n2dds') AND gameid not in (select gameid from reviews where review is not null and userid = 'oyrrw74upu8n2dds') AND gameid not in (select gameid from reviews where rating is not null and userid = 'oyrrw74upu8n2dds')
    ->
    ->
    ->
    -> order by
    ->     starsort desc
    ->
    -> limit 0, 12;
+------+--------------+------------------------+--------+-------------------------+----------+---------+------------------------------------+------+------------------------------------+
| id   | select_type  | table                  | type   | possible_keys           | key      | key_len | ref                                | rows | Extra                              |
+------+--------------+------------------------+--------+-------------------------+----------+---------+------------------------------------+------+------------------------------------+
|    1 | PRIMARY      | gameRatingsSandbox0_mv | index  | PRIMARY                 | starsort | 9       | NULL                               | 12   | Using where; Using temporary       |
|    1 | PRIMARY      | games                  | eq_ref | PRIMARY                 | PRIMARY  | 130     | ifdb.gameRatingsSandbox0_mv.gameid | 1    |                                    |
|    6 | MATERIALIZED | reviews                | ref    | userid,gameid,user_game | userid   | 130     | const                              | 89   | Using index condition; Using where |
|    5 | MATERIALIZED | reviews                | ref    | userid,gameid,user_game | userid   | 130     | const                              | 89   | Using index condition; Using where |
|    4 | MATERIALIZED | unwishlists            | ref    | gameid,userid           | userid   | 130     | const                              | 1    | Using index condition              |
|    3 | MATERIALIZED | wishlists              | ref    | gameid,userid           | userid   | 130     | const                              | 1    | Using index condition              |
|    2 | MATERIALIZED | playedgames            | ref    | userid,gameid           | userid   | 130     | const                              | 1    | Using index condition              |
+------+--------------+------------------------+--------+-------------------------+----------+---------+------------------------------------+------+------------------------------------+

[alice-blue]

I didn't confirm the details or speed of the queries, but i did test briefly to see if anything had obviously broken, and I didn't notice any problems.