Authenticate private Git repo in a swift package, How to refer the package specific .ssh directory?

Question

I have implemented a package which has Private SPM dependencies, I am trying to authenticate the private Git repo with GitHub Deploy Key (SSH key), added dependency looks like below,

     dependencies: [
        .package(url: "git@github.com:xxxx/FrameworkX.git", exact:"1.0.0")
    ],

I have followed the https://medium.com/@ianpartridge/swift-package-manager-github-ssh-e8256c374e16,

1. Added .ssh directory in the root of SPM package
2. Generated public and private keys named frameworkAKey
3. On GitHub, for a private repository, have added a key in a “Deploy Keys” (pasted public frameworkAKey value).

Now When I open the package.swift, SPM tries to resolves the dependencies, but it fails with the `Authentication failed error``

However if I move the frameworkAKey in ~/.ssh directory of the macOS user, SPM is able to resolve the dependencies.

How can I make sure that SPM refers to the .ssh directory which is at the root of SPM Package, not the directory of the macOS user?

Let me know if I should do something else here.