No, if you are managing really large number of servers, configuring your SSH key into each and every server is not a good way. If we are talking about server on cloud, which are highly dynamic in nature, ie, they are started/terminated as and when needed.
You can always configure which "remote user" to use for SSH connections on Ansible master configuration.
Apart from that, you can configure the user anywhere playbook or roles or pass as command line parameter.
For connecting to remote server, using SSH key, same methods can be used.
eg : from command line :
ansible-playbook <playbook yml> -u <user name on remote host> --key-file <SSH key file name with path on master host>
ansible-playbook abc.yml -u "user1" --key-file "/u01/ansible_keys/user1_key.pem"
You can setup these keys in inventory file as well, as below :
myHost ansible_ssh_private_key_file=~/.ssh/mykey1.pem
myOtherHost ansible_ssh_private_key_file=~/.ssh/mykey2.pem
Reference : Specifying ssh key in ansible playbook file