Anyone who needs to verify an encrypted token, needs to first decrypt it. So if you want to use encryption, you would have to provide the decryption key to the client or API. You could also implement some kind of introspection endpoint, which would accept the encrypted JWT, verify it and return either claims or just confirmation that the JWT is valid. Any client or API could then call this endpoint to verify the token, and you would only have to provide the decryption key to one party.
Any APIs which are accessed with this JWT can also have a gateway in front of them, which would decrypt the JWE and pass just the signed JWS.
Also remember that usually the client shouldn't be concerned about the validity of the token. The client just sends the token to the API, and either gets a correct response or not.
Normally I wouldn't go with encrypted tokens, unless you have a strong need for them. If you want to hide contents of tokens from any onlookers I would go with using opaque access tokens and implementing the Phantom Token pattern.