Hi kan and welcome to stackoverflow!
You are totally correct that you can version an API with both Azure API Management (APIM) and directly inside your app with ASP.NET core.
Sometimes it can even make sense to version in both.
I would say that if you think you need versioning but are not sure where to start, use the versioning in ASP.NET Core.
If you later find that your use case was infact much more complicated you will probably want to use API Management aswell.
Version your API in ASP.NET Core app
Suitable when you are building a simple API app that you control and will be called by other services. Say that you are building this app for your workplace and another team will consume your API, but it will not be used outside your workplace organisation. Then versioning in the app code makes sense since it allows you to evolve your API without breaking it for your consumers and depending on another cloud service that will add complexity, management overhead and potentially cost (SKU dependent).
Your urls could look something like this as an example:
myapiapp.azurewebsites.net/v1/domain
myapiapp.azurewebsites.net/v2/domain
myapiapp.azurewebsites.net/v3/domain
Version your API with API Management and ASP.NET Core
API Management is a feature rich service and can do more than just version you API.
Now let's add on to the example above, say that you get an external organisation that want's to consume your API. Your internal API has 3 versions, but you only want the external consumers to be able to reach version 3 of your service. API management allows you to expose a subset of endpoints and versions of an API. Making it easy to restrict the external access. In this example we now practically have "internal versions" and "external versions" since both the service and APIM versions the API.
API Management is also a good choice if you are planning on exposing many of your API's to external customers, since it provides a unified point of entry.
Instead of exposing separate urls for each API to your customers as:
myapiapp.azurewebsites.net/v3/domain
otherteamapiapp.azurewebsites.net/v1/someotherdomain
It could look something like this exposed through APIM:
api.kan-company/v1/domain
api.kan-company/v1/someotherdomain
When to only version through API Management
I would say, like @Dai that you would wan't to version through API Management if you have an old legacy application that you can't change or the risk of change is too high.
Another scenario when to only version through API Management is if you have many API services that doesn't need changes that you would like to compose torgether or call through a unified endpoint.
Hope this helps with deciding which approach to take.