Blazor server and API in same project, 404 not found when app.UserAuth is activate

Question

I have a Blazor server project with some API controllers in same project. In my Program.cs I have this code :

   builder.Services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
            .AddMicrosoftIdentityWebApp(builder.Configuration.GetSection("AzureAd"));
    builder.Services.AddControllersWithViews()
            .AddMicrosoftIdentityUI();
    builder.Services.AddAuthorization(options =>
    {
        options.FallbackPolicy = options.DefaultPolicy;
    });    
    
    ..
    app.UseAuthentication();
    app.UseAuthorization();

When I call my API from my blazor component I got 404 not found response.

If I comment out app.UseAuthentication() and app.UseAuthorization my component can call my API and it works.

I'm a newbie on auth and API and don't know where to start.

My API has no [Auth] tags in it. I can reach the API with Swagger without problems.

My code in component (it works without "UseAuth" but not when it's activate):

    string filnamn = WebUtility.UrlEncode(fil.Namn);

    string reqUri = $"delete/{filnamn}";

    Http.BaseAddress = new Uri("https://localhost:7285/");
    Http.DefaultRequestHeaders.Accept.Clear();

    HttpResponseMessage response = await Http.DeleteAsync(reqUri);

My API controller :

[ApiController]
public class UploadController : ControllerBase
{

    private readonly string grundPath = @"G:\Testfolder";
    private readonly string ulPath = "Upload";


    [HttpDelete("delete/{filename}")]
    public IActionResult Delete(string filename)
    {
        
        try
        {
            var filePath = Path.Combine(grundPath, ulPath, filename);
            if (System.IO.File.Exists(filePath))
            {
                System.IO.File.Delete(filePath);
                return StatusCode(200);
            }
        }
        catch (Exception ex)
        {
            return StatusCode(500, ex.Message);
        }
        return StatusCode(500);
    }

Do you see some obvious wrong/missing part I do or could give me some direction on what I should google for?