Without including client secret, you cannot refresh the access tokens
I tried to reproduce the same in my environment and got below results:
I got refresh token by giving offline_access
in scope like below:
POST https://login.microsoftonline.com/common/oauth2/v2.0/token
client_id:appID
client_secret:secret
grant_type:authorization_code
scope:offline_access user.read
code:code
redirect_uri:https://jwt.ms

When I tried to get access token
using above refresh token without giving client secret, I got error like below:
POST https://login.microsoftonline.com/common/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded
redirect_uri: https://jwt.ms
client_id:appID
grant_type:refresh_token
refresh_token: <refresh token>
scope: https://graph.microsoft.com/.default
Response:

To get access token using refresh token, you must include client_secret
like below:
POST https://login.microsoftonline.com/common/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded
redirect_uri:https://jwt.ms
client_id:appID
client_secret: secret
grant_type:refresh_token
refresh_token: <refresh token>
scope: https://graph.microsoft.com/.default

Alternatively, you can make use of below PowerShell script to create token lifetime policy that can keep access token alive for 24 hrs.
$policy = New-AzureADPolicy -Definition @('{"TokenLifetimePolicy":{"Version":1,"AccessTokenLifetime":"23:59:59"}}') -DisplayName "WebPolicyScenario" -IsOrganizationDefault $true -Type "TokenLifetimePolicy"
$sp = Get-AzureADServicePrincipal -Filter "DisplayName eq '<service principal display name>'"
Add-AzureADServicePrincipalPolicy -Id $sp.ObjectId -RefObjectId $policy.Id
Response:

When I generated access token again, token lifetime increased like below:
POST https://login.microsoftonline.com/common/oauth2/v2.0/token
client_id:appID
client_secret:secret
grant_type:authorization_code
scope:offline_access user.read
code:code
redirect_uri:https://jwt.ms
Response:

Reference:
Configurable token lifetimes - Microsoft