It looks like Azure artifacts requires using a CI/CD pipeline
That's not correct. Anything that can authenticate to Azure Artifacts can push packages to it, including doing it from your own machine. Similarly, if you want to have a private feed, that's a simple configuration, whereas hosting your own NuGet server means you'll need to implement authentication yourself.
I strongly recommend you use Azure Artifacts instead of hosting your own server, if you already have access to Azure DevOps. They handle scalability/performance, whereas I believe that NuGet.Server has performance problems when there are a large number of packages. Azure DevOps also handles authentication, as previously discussed. I wouldn't be surprised if Azure Artifacts is also cheaper than Azure App Service, although this depends on a lot of things so will depend on your specific usage, making it impossible to make a blanket statement.
If you really want to host your own nuget feed, I can't find docs explicitly saying it, but Azure App Service on a Windows host is IIS. ASP.NET (before Core) only run in IIS, and here are Azure App Service's docs for running ASP.NET (not Core) apps: https://learn.microsoft.com/en-us/azure/app-service/configure-language-dotnet-framework
