How do you remove/undo/delete a pending modification on Amazon Aurora RDS?

Question

I was pegging the 90 total connections allowed on my Aurora RDS instance, now that I've got a few more heavy duty sites than I had before. I decided to upgrade the instance and I did so, but put the upgrade as pending until the next maintenance window, without considering adding a read-only replica into the cluster. I've added the replica and all is well. I want to cancel that pending upgrade from a db.t2.medium Aurora to a db.r3.large Aurora instance. I can't find a way to clear out the pending modifications queue.

I have the AWS CLI installed and config'd. I just can't find documentation on flushing out the pending modifications queue. Thanks to all in advance.

Answer 1

As @gileri described, there is now a way to undo a pending modification.

I tested it multiple times with RDS/Aurora instances and it works as expected, even without the --apply-immediately parameter (at least for the instance class modifications).

Full Example

Let's change the instance class for an Aurora instance called database-2-instance-1 from db.t3.medium to db.r4.large and undo it again afterwards. None of these commands do impact the availability of the database.

Note: Using jq here to only output the important parts.

Check instance class

$ aws rds describe-db-instances --db-instance-identifier database-2-instance-1 | jq '.DBInstances[].DBInstanceClass'
"db.t3.medium"

Validate that there is no pending modification

$ aws rds describe-db-instances --db-instance-identifier database-2-instance-1 | jq '.DBInstances[].PendingModifiedValues'
{}

Modify instance class

This modification will result in a change of the instance class in the next maintenance window.

$ aws rds modify-db-instance --db-instance-identifier database-2-instance-1 --db-instance-class db.r4.large | jq '.DBInstance.PendingModifiedValues'
{
  "DBInstanceClass": "db.r4.large"
}

Validate again

Just to be sure, check if everything looks as expected.

$ aws rds describe-db-instances --db-instance-identifier database-2-instance-1 | jq '.DBInstances[].PendingModifiedValues'
{
  "DBInstanceClass": "db.r4.large"
}

Undo modify of instance class

This is the important part which modifies the instance class back to the old value. The documentation describes that a --apply-immediately is required, but it turns out that that is not the case. At least in this example.

$ aws rds modify-db-instance --db-instance-identifier database-2-instance-1 --db-instance-class db.t3.medium | jq '.DBInstance.PendingModifiedValues'
{}

Validate the removed pending modification

$ aws rds describe-db-instances --db-instance-identifier database-2-instance-1 | jq '.DBInstances[].PendingModifiedValues'
{}

At this point the pending modification is gone.

Answer 2

Since February 2019, it is documented that you can cancel such pending modifications. I have tested this with a pending DB instance class modification.

If you don't want a pending change to be applied in the next maintenance window, you can modify the DB instance to revert the change using the AWS CLI and specify the --apply-immediately option.

Answer 3

There is no documented way to cancel pending modifications.

Note, though, that on an Aurora cluster, the master does not play a role that is as important as you might assume. It differs from other setups, because your data does not actually live on the master instance -- it lives on the Aurora Cluster Volume, which is not part of any instance -- and replicas have a different relationship to the physical data store than in conventional replication.

When you create an Amazon Aurora instance, you create a DB cluster. A DB cluster consists of one or more DB instances, and a cluster volume that manages the data for those instances. An Aurora cluster volume is a virtual database storage volume that spans multiple Availability Zones, with each Availability Zone having a copy of the DB cluster data.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.html

So, the nodes don't hold the data -- it's separate, which is one of the powerful features of Aurora's internals. In fact, Aurora replicas don't have their own copy of the data, so in some strict/literal/pedantic sense, they technically are not even properly "replicas" (though that terminology is still used) -- they're just read-only nodes in the cluster.

So, you could probably just do a Reboot with Failover, which would switch the other instance (your new replica) to become the master. Then create a new replica and destroy the original instance.

Note that if you have replicas in an aurora cluster, you need to be using the cluster endpoints to connect to the writer node or one of the reader nodes, because if your writer ("master") fails, Aurora will usually swap the roles (or possibly always swap the roles, though there might be cases where it doesn't). You might already be doing that.

Answer 4

Check your pending modifications:

# aws --profile example_profile rds describe-db-clusters --db-cluster-identifier example_globalcluster --region example_region | jq '.DBClusters[].PendingModifiedValues'
{
  "EngineVersion": "12.7"
}

Next, stop the global database (which consequently stops the writer node or whatever setup you have) and check again for pending modifications:

# aws --profile example_profile rds describe-db-clusters --db-cluster-identifier example_globalcluster --region example_region | jq '.DBClusters[].PendingModifiedValues'
  null

When you start the global database again, you'll see that those pending modifications are gone. I managed to solve this issue for two times now.

Answer 5

It is mentioned in the documentation that,

If you don't want a pending change to be applied in the next maintenance window, you can modify the DB instance to revert the change using the AWS CLI and specify the --apply-immediately option.

However, do note that --apply-immediately will result in a reboot of the instance even if you are reverting the changes and rds may be down for about a minute.