How to fix the 'invalid_grant' error when using Google Photos API

Question

I'm trying to use the Google Photos API (quite a newbie on this). I've managed to go through the authentication/consent process. I stored the authorization code that was returned by Google. On a later moment, I'm trying to use this authorization code to use the Google Photos API. Below a code snippet.

// $clientId and $clientSecret are read from my .json file
// $state was randomly generated, used in the registration process and stored to/read from the database
// $code was stored to the database at the end of te registration process and read from the database to pass in subjoined call
// $redirectUrl and $scope are hard coded for now
$oauth = new OAuth2( [
    'clientId'           => $clientId,
    'clientSecret'       => $clientSecret,
    'authorizationUri'   => 'https://accounts.google.com/o/oauth2/v2/auth',
    'state'              => $state,
    'redirectUri'        => $redirectUrl,
    'tokenCredentialUri' => 'https://www.googleapis.com/oauth2/v4/token',
    'scope'              => $scope,
] );
$oauth->setCode( $code );

$credentials = new UserRefreshCredentials(
    $this->getScope(),
    [
        'client_id'     => $oauth->getClientId(),
        'client_secret' => $oauth->getClientSecret(),
        'refresh_token' => $oauth->getRefreshToken()
    ]
);

$client = new PhotosLibraryClient( [ 'credentials' => $credentials ] );

For the refresh_token, I've tried both $oauth->fetchToken()['access_token'] as well as $oauth->getRefreshToken().

Unfortunately, I get an exception:

guzzlehttp/guzzle/src/Exception/RequestException.php @ 113 : Client error: `POST https://www.googleapis.com/oauth2/v4/token` resulted in a `400 Bad Request` response: { "error": "invalid_grant", "error_description": "Bad Request" }

Anyone any idea what could be wrong here?