It depends on the type of client and the authentication used. Considering cookie authentication first.
For a client that uses a cookie, authentication is based on the existence of a valid cookie. In order to signout a user, the cookie has to be expired or removed. Perhaps use sessions or otherwise extend the lifetime or keep track of the cookie on activity.
The hard part is the existence of JWT tokens. Now it doesn't matter whether the app uses them to access other resources (API) or for the backend (SPA).
The nature of the JWT is that it can't be revoked and remains active untill it expires. The good news is that you can set the lifetime to more or less 20 minutes, taking clock skew (default 5 minutes) into account. The token will expire automatically, no further action necessary.
To keep things working you may want to use refresh tokens. A refresh token is used to request a new access token on behalf of the user without having the user to login again. The refresh token is passed to the authorization endpoint of IdentityServer in order to obtain a new access token.
An important difference between a refresh token and an access token is that the refresh token can be any string value without any meaning, while the access token is self-contained (JWT). The refresh token is stored in IdentityServer (PersistedGrants table) and has to match the one received. This means that if the token is not found in the store, the request will be denied. In other words, a refresh token can be revoked unlike a JWT access token.
The alternative of a JWT access token is a reference access token. For more information about tokens, read my answer here.
The reference access token is also stored in the PersistedGrants table and can be revoked.
The problem with inactivity is that the client can't trigger actions on the server. So any action needed server-side has to be initiated there.
The client can only determine inactivity of the user after the user becomes active again. But that's easy, without valid cookie the user has to login again.
One remark: with single sign-on (SSO) the cookie on the IdentityServer website will automatically authenticate the user again, skipping the login form. In order to bypass this, make sure the client redirects to IdentityServer with the parameter: prompt=login.
Having set the lifetime of access tokens to 20 minutes, everything is in place.
However, the refresh token is still valid. Luckily you can also set an absolute expiration on the refresh token. In this case you can think of setting it to 20 minutes as well (normally this would be a much longer lifetime). Just make sure you refresh the access token within 5 minutes before expiration. By the way, this does not really refresh the token, it replaces the stored token. The current token remains valid (and useable) until it expires.
One remark: the refresh token has to be set to one-time-use. When requesting a new access token, a new refresh token is returned as well, replacing both. Make sure you refresh the tokens always in time, regardless whether you need it for the request.
Another scenario is that the user logs out. In that case you can signout the user from all apps (that are part of the IdentityServer session) and IdentityServer itself. Make sure logout also removes the refresh token from the store.