In this situation, the secure authentication does not rely on the actual certificate, but the key pair covered.
Asymmetric cryptography relies on a private and a public key. The private key must remain confidential (so keep it on the server which has to be authenticated, the one you described as the correct server). The public key can be arbitrarily spread, the only thing somebody could do is encrypting information for the private key's holder, or verifying a signature issued by him.
The (very) basic idea behind the actual authentication process works by either the authenticating client sending an encrypted random number, which only the correct server can decrypt and return, or by having the server sign something defined by the client. The real process implemented for authentication is a little bit more complicated to prevent man-in-the-middle-attacks, but far beyond scope of this answer.
So, can I implement some method to verify the server even when the server binary is available for everyone, since it is the client too?
Rely on something already there, and do not implement crypto code on your own. Maybe consider using TLS for encrypted communication anyway if you're already distributing keys, authentication will be done during the handshake phase.