Getting Permission Error Deploying a Cloud Function with Private Dependency using Cloud Build

Question

I'm trying to deploy a Cloud Function (Node.js + TypeScript) using Cloud Build.
What's unique is that my Cloud Function uses a private dependency, so that my package.json looks like this:

"dependencies": {
  "@foo/my_private_repo": "git+ssh://git@github.com:foo/my_private_repo.git", // this is the tricky part
  "@google-cloud/functions-framework": "^3.1.2",
  "axios": "^0.27.2",
  "dotenv": "^16.0.2"
},

This is working totally fine on my local computer, but I'm having a hard time deploying.
I have followed the Cloud Build document, yet I'm getting the following error.

Step #1: npm ERR! git@github.com: Permission denied (publickey).
Step #1: npm ERR! fatal: Could not read from remote repository.
Step #1: npm ERR!
Step #1: npm ERR! Please make sure you have the correct access rights
Step #1: npm ERR! and the repository exists.

My cloudbuild.yaml looks like:

steps:
# reads the deploy key of the private repo from Secret Manager, and setup ssh
- name: gcr.io/cloud-builders/git
  secretEnv: ['SSH_KEY']
  entrypoint: bash
  args:
  - -c
  - |
    echo "$$SSH_KEY" >> /root/.ssh/id_rsa
    chmod 400 /root/.ssh/id_rsa
    cp known_hosts.github /root/.ssh/known_hosts
  volumes:
  - name: ssh
    path: /root/.ssh

# deploy
- name: gcr.io/google.com/cloudsdktool/cloud-sdk
  args:
  - gcloud
  - functions
  - deploy
  - my-function
  - --gen2
  - --region=asia-northeast1
  - --trigger-http
  - --runtime=nodejs16
  - --entry-point=myFunction
  - --env-vars-file=.env.staging.yml
  - --allow-unauthenticated
  volumes:
  - name: ssh
    path: /root/.ssh

availableSecrets:
  secretManager:
  - versionName: projects/<PROJECT_ID>/secrets/github_deploy_key/versions/latest
    env: SSH_KEY

I have made sure that there isn't any problem with the key because I can access the repo using this key from my CLI.

$ ssh -T -i id_github git@github.com
Hi foo/my_private_repo! You've successfully authenticated, but GitHub does not provide shell access.

Funny thing is that steps other than deploying works successfully, only the deploying part seems to have a problem.

steps:
# reads the deploy key of the private repo from Secret Manager, and setup ssh
- name: gcr.io/cloud-builders/git
  secretEnv: ['SSH_KEY']
  # ...same

# npm install
# this works fine
- name: node:16
  entrypoint: npm
  args:
  - install
  volumes:
  - name: ssh
    path: /root/.ssh

# clone
# this works fine too
- name: 'gcr.io/cloud-builders/git'
  args:
  - clone
  - git@github.com:foo/my_private_repo
  volumes:
  - name: 'ssh'
    path: /root/.ssh

Any ideas that might solve the problem?