ECDH doesn't rely on the public keys alone; those are just the only components that are required to be send. Instead, it depends on two public / private key pairs, generated by both parties. The trick in Diffie-Hellman key agreement (DH) is that a calculates the shared-secret given the private key and the public key of the other party. This shared secret is identical on both sides if and only if the correct private and public keys are used.
The public and private keys of a pair are linked during key pair generation; the DH public key is calculated from the base point of the curve and the private key. This specific bond between the keys is required to calculate the same shared secret. For this calculation to succeed it is also required that both keys use the same domain parameters; in other words, the public keys need to be on the same curve.
A third party / adversary can of course copy the public key of either party. That will however not help the adversary as it doesn't have access to either of the accompanying private keys. So no other party than the ones involved in the key agreement will be able to calculate the same shared secret; you need one of the private keys to do that.
Taking it further, it is possible for an adversary to create a different key pair. If the public key of that key pair is accepted by the other parties then it is possible to create one or two different shared secrets.
SSL / TLS for instance mainly uses ephemeral (temporary) keys; any public ECDH key is accepted. That means that such a form of DH does not offer authentication of the parties involved. So a man-in-the-middle (MitM) attack is possible unless other authentication measures are used. The TLS for used in browsers uses server certificates / server signing for that.
But this part is giving an answer to a question that you didn't ask (yet).
Sometimes the word "secret key" is incorrectly replaced with "private key", even in books on crypto. This is very confusing, as it is obviously impossible to have a shared private key: "shared" and "private" are two opposites. Diffie-Hellman doesn't calculate a shared private key, it produces a shared secret, which is then used to calculate one or more session keys.