How to automatically bump NPM package version with semantic-release?

Question

I have an NPM package that up until now, I have been manually bumping the version in package.json and then my GitHub actions CI publishes.

I have now added semantic release with the following config but it keeps trying to update the package version to 1.0.0 because I guess it can't find any previous releases?

{
  "branches": ["main", "master"],
  "plugins": [
    [
      "@semantic-release/commit-analyzer",
      {
        "preset": "conventionalcommits"
      }
    ],
    "@semantic-release/release-notes-generator",
    "@semantic-release/changelog",
    [
      "@semantic-release/npm",
      {
        "npmPublish": false
      }
    ],
    [
      "@semantic-release/github",
      {
        "assets": [
          "CHANGELOG.md",
          "dist",
          "package.json"
        ]
      }
    ],
    [
      "@semantic-release/git",
      {
        "message": "chore(release): ${nextRelease.version} [skip ci]\n\n${nextRelease.notes}"
      }
    ]
  ]
}

I then trigger my publish and version update with GitHub actions:

- run: npm ci
- run: npm run test:unit
# - run: npm run test:integration
- run: npm run build

- name: Install semantic-release plugins
  run: npm i --no-save @semantic-release/changelog @semantic-release/git conventional-changelog-conventionalcommits

- name: Create Release
  run: npx semantic-release
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- run: npm publish --access public
  env:
    NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

To make matters more complicated, I recently modified my base branch from master to main so I don't know if that mucks things up with trying to find release history but I have tried to remedy that with the branches config of "branches": ["main", "master"],.

How can I make semantic release aware of package versions that were handled manually / without the semantic release library?