MariaDB 10.3.10 Galera Cluster not syncing InnoDB table definition

Question

I have a 3 node multimaster MariaDB Galera Cluster that is not syncing a modification to an InnoDB table definition.

Server version: 10.3.10-MariaDB MariaDB Server

wsrep_provider_version: 25.3.24(r3825)

The table is mysql.innodb_table_stats and the differnece is the data_type/length of the field table_name.

The DDL Operation executed in galera-01 member using mysql_upgrade utility:

ALTER TABLE innodb_index_stats MODIFY `table_name` varchar(199) COLLATE utf8_bin NOT NULL;

Table definition in galera-01 member:

CREATE TABLE `innodb_table_stats` (
  `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `table_name` varchar(199) COLLATE utf8_bin NOT NULL,
  `last_update` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
  `n_rows` bigint(20) unsigned NOT NULL,
  `clustered_index_size` bigint(20) unsigned NOT NULL,
  `sum_of_other_index_sizes` bigint(20) unsigned NOT NULL,
  PRIMARY KEY (`database_name`,`table_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0

Table definition in galera-02 and galera-03 members:

CREATE TABLE `innodb_table_stats` (
  `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `table_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `last_update` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
  `n_rows` bigint(20) unsigned NOT NULL,
  `clustered_index_size` bigint(20) unsigned NOT NULL,
  `sum_of_other_index_sizes` bigint(20) unsigned NOT NULL,
  PRIMARY KEY (`database_name`,`table_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0

The cluster seem to be in healthy shape, I can't understand why it won't sync that specific DDL operation. I tried with other DDL operations on some innodb test table and it works like a charm.

+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_state_uuid | 213a9d66-d207-11e8-b7bb-733e94f6ddc5 |
| wsrep_cluster_conf_id | 294   |
| wsrep_cluster_size | 3     |
| wsrep_cluster_status | Primary |
| wsrep_ready   | ON    |
| wsrep_connected | ON    |
| wsrep_local_state_comment | Synced |
| wsrep_local_recv_queue_avg | 0.038462 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_cert_deps_distance | 18.686275 |
| wsrep_local_send_queue_avg | 0.200000 |

Yes, the ddl operation was done with mysql_update on galera-01 member. — Alexandre Juma, Mar 05 '19 at 00:09
Have you upgraded the other nodes? Check the release notes; it seems like you must run mysql_upgrade on each node. — Rick James, Mar 05 '19 at 03:58
I've totally got it wrong. I had the idea that mysql_upgrade was run in only one cluster node and all changes would be propagated to the other nodes via Galera replication. Once I ran mysql_upgrade on all nodes, the table definition was updated on the missing members. — Alexandre Juma, Mar 06 '19 at 10:12
@RickJames can you post an answer about running mysql_upgrade on all nodes — Alexandre Juma, Mar 06 '19 at 11:52

score 1 · Accepted Answer · answered Mar 06 '19 at 15:42

MyISAM tables are not replicated by Galera.
Some system tables, (most of the tables in the mysql database) are MyISAM.
When upgrading, mysql_upgrade must be run.
mysql_upgrade often makes changes to system tables.

So, unless the upgrade instructions somehow do the task for you, you must run mysql_upgrade on each node.

MariaDB 10.3.10 Galera Cluster not syncing InnoDB table definition

1 Answers1