On the destination server you could restrict SSH access by username in the /etc/ssh/sshd_config file, but I don't think that it what you are looking for here.
On the source machine you can try to use the iptables "owner" module to do this. It might be some maintenance work but it would do the trick. It checks the UID of the user and than allows (or rejects) the connection.
Lets say that the UID of your users are 1, 2 and 3. user A is allowed to SSH to 1.1.1.1, user B to 2.2.2.2 and user C to 3.3.3.3
iptables -A OUTPUT -m owner --owner-uid 1 -d 1.1.1.1/32 -p tcp --dport 22 -j ACCEPT
iptables -A OUTPUT -m owner --owner-uid 1 -p tcp --dport 22 -j DROP
iptables -A OUTPUT -m owner --owner-uid 2 -d 2.2.2.2/32 -p tcp --dport 22 -j ACCEPT
iptables -A OUTPUT -m owner --owner-uid 2 -p tcp --dport 22 -j DROP
iptables -A OUTPUT -m owner --owner-uid 3 -d 3.3.3.3/32 -p tcp --dport 22 -j ACCEPT
iptables -A OUTPUT -m owner --owner-uid 3 -p tcp --dport 22 -j DROP
to keep it a bit more "manageable" you could create a "chain" per user, and define all his allowed accesses in there.
Create the chains:
iptables -N USER1
iptables -N USER2
iptables -N USER2
Forward the outgoing traffic from the users to the chain:
iptables -A output -m owner --owner-uid 1 -J USER1
iptables -A output -m owner --owner-uid 2 -J USER2
iptables -A output -m owner --owner-uid 3 -J USER3
Add rules to each user his chain:
iptables -A USER1 -d 1.1.1.1/32 -p tcp --dport 22 -m comment --comment "allow ssh to 1.1.1.1" -j ACCEPT
iptables -A USER1 -p tcp --dport 22 -j DROP
iptables -A USER1 -j ACCEPT
iptables -A USER2 -d 2.2.2.2/32 -p tcp --dport 22 -j ACCEPT
iptables -A USER2 -p tcp --dport 22 -j DROP
iptables -A USER3 -d 3.3.3.3/32 -p tcp --dport 22 -j ACCEPT
iptables -A USER3 -p tcp --dport 22 -j DROP
check access of a user (display chain)
iptables -L USER1
The owner module also works with gid's:
iptables -A USER3 -m owner --owner-gid 3 -p tcp --dport 22 -j DROP