Your best option is likely to revoke access to that certificate and set up individual server users. You can also use a firewall to limit access from specific IPs.
I have a small tutorial on SSH/SFTP setup for Amazon Linux on my website you may find useful. They key parts are.
Firewall / Security Groups
On AWS you can use Security Groups or Network Access Control Lists to specify what IPs can connect on what protocols. You can set this up so only your corporate IP can connect on SSH. This is very quick and easy to do. See instructions here.
I suggest using security groups rather than NACLs, just because I find it easier. You could use either or both.
Revoke Existing Key Pair
To revoke the key pair (documentation, under "Adding or Replacing a Key Pair for Your Instance")
Retrieve the public key from your new key pair. For more information, see Retrieving the Public Key for Your Key Pair on Linux or Retrieving the Public Key for Your Key Pair on Windows.
Connect to your instance using your existing private key file.
Using a text editor of your choice, open the .ssh/authorized_keys file on the instance. Paste the public key information from your new key pair underneath the existing public key information. Save the file.
Disconnect from your instance, and test that you can connect to your instance using the new private key file.
(Optional) If you're replacing an existing key pair, connect to your instance and delete the public key information for the original key pair from the .ssh/authorized_keys file.
Create the new users
sudo su
sudo useradd fred
passwd fred
Create the keys
su fred
ssh-keygen -f rsa
mkdir .ssh
touch .ssh/authorized_keys
chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
cat fred.pub >> /home/fred/.ssh/authorized_keys
Allow user login
vi /etc/ssh/sshd_config
PasswordAuthentication no
AllowUsers ec2-user fred