Can't get Source Link to work in a .NET 5 library hosted in Github

Question

I'm trying to get Source Link set up for a .NET 5 C# library hosted in Github.

Currently inside my library project, the .csproj file looks like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <AssemblyName>Cubit.Zones</AssemblyName>
    <RootNamespace>Cubit.Zones</RootNamespace>
    <PackageProjectUrl>https://github.com/Cubit-AS/zones</PackageProjectUrl>
    <RepositoryUrl>https://github.com/Cubit-AS/zones</RepositoryUrl>
    <PublishRepositoryUrl>true</PublishRepositoryUrl>
    <RepositoryType>git</RepositoryType>
    <EmbedUntrackedSources>true</EmbedUntrackedSources>
    <IncludeSymbols>true</IncludeSymbols>
    <SymbolPackageFormat>snupkg</SymbolPackageFormat>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.1.1" PrivateAssets="All" />
    <PackageReference Include="MongoDB.Driver" Version="2.19.0" />
  </ItemGroup>

</Project>

The project currently just have a testclass with a method inside it:

public class TestClass
{
    public int ReturnAnotherNumber(int number)
    {
        // This is a comment inside the method in the library to explain stuff
        var result = number * 5 / 2;
        return result;
    }
}

This project also have a workflow file that ends up creating a new version of the nuget package whenever we create a new release inside Github:

name: CI

on:
  release:
    types: 
      - published

jobs:
  Publish:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - name: Setup .NET Core
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: '5.0.x'
          include-prerelease: true
      - name: Add github source
        run: dotnet nuget add source https://nuget.pkg.github.com/Cubit-AS/index.json -n github -u ${{ secrets.PUBLISHER_ID }} -p ${{ secrets.PUBLISHER_TOKEN }} --store-password-in-clear-text
      - name: Create NuGet Package
        run: dotnet pack -c Release /p:Version=${{ github.event.release.tag_name }} /p:PackageReleaseNotes="See https://github.com/Cubit-AS/identity/releases/tag/${{ github.event.release.tag_name }}"
      - name: Publish package
        run: dotnet nuget push **/*.nupkg --source "github"

So the package is pushed using the username and password for our Github source control.

I now create a release and the package is created successfully.

I then import this into another project as a NuGet package.

Here I make a instance of TestClass, and call ReturnAnotherNumber and sets a breakpoint there. When debugging I can't step into this function just as if for other packages that don't use source link.

So I guess I'm missing something in the setup here. My gut feeling is that it's related to the package is inside a private repo that has a username and password, and therefore my project can't access the source code from the repo since it does not have access to it.

But I can't see anything inside the Source Link docs on how this is supposed to be set up.

Or it might be a complete other issue of course.

Any help here is appreciated!