0

How can I reduce the percent of sorts that require temporary tables? What does this warning mean exactly?

A comment here suggests it's due to using ORDER BY on a non-indexed column. I think I've added the indexes that I can.

The full mysqltuner output is below.

 >>  MySQLTuner 2.0.1
         * Jean-Marie Renouard <jmrenouard@gmail.com>
         * Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.pl/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[OK] Currently running supported MySQL version 8.0.29-0ubuntu0.20.04.3
[OK] Operating on 64-bit architecture
 
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log (0B)
[--] Log file /var/log/mysql/error.log is empty. Assuming log-rotation. Use --server-log={file} for explicit file
 
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA 
[--] Data in MyISAM tables: 2.4M (Tables: 6)
[--] Data in InnoDB tables: 418.7M (Tables: 48)
[OK] Total fragmented tables: 0
 
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
 
-------- Views Metrics -----------------------------------------------------------------------------
 
-------- Triggers Metrics --------------------------------------------------------------------------
 
-------- Routines Metrics --------------------------------------------------------------------------
 
-------- Security Recommendations ------------------------------------------------------------------
[--] Skipped due to unsupported feature for MySQL 8
 
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
 
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 2d 9h 7m 26s (15M q [75.676 qps], 1M conn, TX: 10G, RX: 3G)
[--] Reads / Writes: 49% / 51%
[--] Binary logging is enabled (GTID MODE: OFF)
[--] Physical Memory     : 7.8G
[--] Max MySQL memory    : 11.8G
[--] Other process memory: 0B
[--] Total buffers: 2.0G global + 65.9M per thread (151 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 2.5G (32.14% of installed RAM)
[!!] Maximum possible memory usage: 11.8G (151.35% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (2K/15M)
[OK] Highest usage of available connections: 4% (7/151)
[OK] Aborted connections: 0.00%  (3/1783474)
[--] Skipped name resolution test due to skip_networking=ON in system variables.
[--] Query cache have been removed in MySQL 8
[!!] Sorts requiring temporary tables: 73% (75K temp sorts / 103K sorts)
[!!] Joins performed without indexes: 697
[OK] Temporary tables created on disk: 0% (16 on disk / 13K total)
[OK] Thread cache hit rate: 99% (7 created / 1M connections)
[OK] Table cache hit rate: 99% (14M hits / 14M requests)
[OK] table_definition_cache (2000) is greater than number of tables (393)
[OK] Open file limit used: 1% (108/10K)
[OK] Table locks acquired immediately: 99% (20K immediate / 20K locks)
[OK] Binlog cache memory access: 100.00% (6897276 Memory / 6897283 Total)
 
-------- Performance schema ------------------------------------------------------------------------
[--] Performance_schema is activated.
[--] Memory used by P_S: 72B
[--] Sys schema is installed.
 
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
 
-------- MyISAM Metrics ----------------------------------------------------------------------------
[--] MyISAM Metrics are disabled on last MySQL versions.
 
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 2.0G/418.7M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (4.6875%): 48.0M * 2/2.0G should be equal to 25%
[!!] InnoDB buffer pool instances: 8
[--] Number of InnoDB Buffer Pool Chunk: 16 for 8 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (4763088942 hits/ 4763111006 total)
[!!] InnoDB Write Log efficiency: 66.41% (39425182 hits/ 59368351 total)
[OK] InnoDB log waits: 0.00% (0 waits / 19943169 writes)
 
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine not available.
 
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
 
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
 
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
 
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
 
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    Buffer Key MyISAM set to 0, no MyISAM table detected
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: <redacted>
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    sort_buffer_size (> 256K)
    read_rnd_buffer_size (> 256K)
    join_buffer_size (> 256.0K, or always use indexes with JOINs)
    key_buffer_size=0
    innodb_log_file_size should be (=256M) if possible, so InnoDB total log files size equals 25% of buffer pool size.
    innodb_buffer_pool_instances(=2)

What is the cause of sorting using temporary tables? How can I avoid this?

mgiuffrida
  • 3,299
  • 1
  • 26
  • 27
  • Why exactly can't you have a composite index? – PM 77-1 Jun 15 '22 at 22:10
  • Oh, looks like composite indexes *can* be used in mixed ASC-DESC ORDER BY queries in MySQL 8 [my version]. They couldn't in 5.6: https://dev.mysql.com/doc/refman/5.6/en/order-by-optimization.html – mgiuffrida Jun 15 '22 at 22:21
  • You can also change the buffer pool size dynamically on MySQL 8.0. You don't need to restart. – Bill Karwin Jun 15 '22 at 23:45
  • @mgiuffrida From MySQL Command Prompt, SET GLOBAL innodb_buffer_pool_size=500*1024*1024; will enable room for all tables and indexes with some room for growth. Stop/Start not required. Change your my.cnf for next stop/start. You may need to ADD a line to your configuration file in [mysqld] section. – Wilson Hauck Jun 16 '22 at 01:07
  • 1
    Please post your QUERY that has ORDER BY with multiple columns for us to consider and also SHOW CREATE TABLE tbl_name; for each table used n the query for analysis. – Wilson Hauck Jun 16 '22 at 01:09
  • @WilsonHauck I've updated the question with my query, tables and indices. Thanks! – mgiuffrida Jun 16 '22 at 14:07
  • What gives your select using EXPLAIN before? – JoelCrypto Jun 20 '22 at 15:07
  • Please qualify each column in a query with JOINs -- it is hard for us to know what columns came from where. And that is critical to understanding the `WHERE` and `ORDER BY`. – Rick James Jun 20 '22 at 20:10
  • 1
    Posted info about scores, users, and user_settings did not come from SHOW CREATE TABLE table_name; We have NO idea what your tables REALLY look like at this time. There could have been many changes. Additional information requested, Text results of A) EXPLAIN SELECT (rest of your slow query); SHOW WARNINGS; of the query to be analyzed in the session. B) SHOW CREATE TABLE scores: C) SHOW CREATE TABLE users; D) SHOW CREATE TABLE user_settings; E) SHOW TABLE STATUS WHERE name LIKE 'scores'; F) SHOW TABLE STATUS WHERE name LIKE 'users'; and G) SHOW TABLE STATUS WHERE name LIKE 'user_settings'; – Wilson Hauck Jun 20 '22 at 23:33

1 Answers1

2

Mysqltuner has a number of scary output statements. I suggest that this one is not too critical.

Sorting comes in many flavors; I will try to explain what the Tuner is seeing and why there is some concern.

  • 103K / 15M queries need a sort -- Probably not bad.
  • A sort is caused by ORDER BY and a few other things. ORDER BY is probably 'required' by your application.
  • If MySQL can do the sort, it is usually better to let it. That is, don't simply remove the ORDER BY and add code to your app to do the work.
  • Sometimes the results can be sorted without actually sorting. This happens when the INDEX being used provides the necessary ordering. This is not always possible.
  • Run EXPLAIN SELECT ... on any query that you want to investigate. If it says "Using temporary" in the "Extra" column, then the Optimizer decided to collect some intermediate results before proceeding. Subqueries and GROUP BY and ORDER BY may cause such. This is normal. (Again, don't try to move such actions into your code.) This is not an issue since only "16 on disk".
  • If you see "Using filesort", that may be a very fast sort in RAM, or it may be involve sorting via a file. The difference is no clearly indicated.

As for that query, here is a slightly modified version:

SELECT  s.user_id, u.name,
        ( SELECT us.is_premium
               FROM user_settings AS us
                 ON us.user_id = s.user_id
            ) AS is_premium,
        s.score, s.word_count,
        s.obscure_count, s.non_word_count, s.ms,
        IF(s.word_count = 0, NULL, s.word_count /
             (s.word_count + s.non_word_count)) AS accuracy,
        s.last_updated_word
    FROM  scores AS s
    INNER JOIN  users AS u  ON u.id = s.user_id
    WHERE  s.puzzle_id = $puzzleId # e.g. 123
      AND  s.is_beta_puzzle = 0
      AND  u.is_test = 0
    ORDER BY  s.score DESC,
              s.word_count DESC,
              $tiebreaker,
              s.last_updated_word ASC
    LIMIT  500

On scores, replace KEY puzzle_id (puzzle_id) with INDEX(puzzle_id, is_beta_puzzle)

is_premium, is_test, and the varying directions in the ORDER BY are hard to get around.

If you don't have to have 500 results, it might be possible to move the test for is_test out of the query, then filter them later. For example:

SELECT ...as above, including is_premium...
    FROM (...most of above formulation...) AS x
    WHERE is_test = 0
    ORDER BY ...  # repeated, but without LIMIT
Rick James
  • 135,179
  • 13
  • 127
  • 222
  • Thanks! When I run `EXPLAIN` on my original query, the select_types are all `SIMPLE`. Using your updated query (the subquery instead of the `LEFT JOIN`), the select types are `PRIMARY` (on scores and users) and `DEPENDENT SUBQUERY` (on user_settings). Does this matter? Updating the index changes the `filtered` value on `scores` from 10 to 100 and removes the `Using where` flag -- are these improvements? – mgiuffrida Jun 21 '22 at 04:05
  • @mgiuffrida - Sorry, but I need to see the new query and the new `EXPLAIN` (all of it) in order to discuss your questions. A simpler answer... If my reformulation is faster, then who cares about the `EXPLAIN`. (Yes, my subquery is a "Dependent Subquery. And it _may_ be faster than what you had before.) – Rick James Jun 21 '22 at 04:30
  • I'll mark this as accepted since you explained a fair amount about the tuner's output and how sorting works. I ultimately sped up my query by adding columns from the SELECT field to the index, since otherwise the engine preferred to filesort the table. Thanks! – mgiuffrida Jun 22 '22 at 15:10