The client asks for something. The server responds with something.
For it to be idempotent, you should expect to get the same response if you ask for the same thing. Only the server can decide what is in the response, so that has to be implemented by the server.
For it to be safe, the request must not alter the server. Only the server can decide to alter something or not based on the request.
Caching takes place on the client (it decides if it should make a request or not based on what data it already has). Only the client can make that decision. (Of course, the server is responsible for giving the client the information they need to make that decision).