2

I'm using GitHub Actions to run some tests on every push and I need DVC. I'm trying to make this work with the runs-on: ubuntu-latest option but when I try to run it, the action get's stuck because it requires manual authentication. Is there a way to go around this and automate the authentication ?

Shcheklein
  • 5,979
  • 7
  • 44
  • 53
WholesomeGhost
  • 1,101
  • 2
  • 17
  • 31
  • 1
    What kind of DVC remote are you using? For many types, you could set up the authentication via environment variables (i.e. `AWS_ACCESS_KEY_ID` & `AWS_SECRET_ACCESS_KEY` for s3) that you can combine with https://docs.github.com/en/actions/security-guides/encrypted-secrets – David de la Iglesia Oct 10 '22 at 16:26
  • gdrive, sorry I forgot to mention that. – WholesomeGhost Oct 11 '22 at 08:06

2 Answers2

2

I figured out that in order for this to work, I had to switch to using a service account.

WholesomeGhost
  • 1,101
  • 2
  • 17
  • 31
0

I had the same issue. Here are the steps I used to setup a Google Service Account with DVC, Google Drive and Github Actions.

Setup a google cloud platform service account

A google cloud platform service account "is a special kind of account used by an application or compute workload, such as a Compute Engine virtual machine (VM) instance, rather than a person. A service account is identified by its email address, which is unique to the account." In this case, the service account for your app to access Google Drive.

  1. Go to the Google API Console
  2. Select or Create a project for DVC remote connections.
  3. Enable the Google Drive API by clicking the ENABLE button
  4. Go to the Google API Crediential page
  5. Click on the + Create Credentials button and then select Service Account.
  6. Enter a service account name (such as gdrive-api) and a some description (such as "Allows access to the google drive api") then click on the Create and continue button.
  7. Click continue at the "Grant this service account access to the project (optional)" section: No need to add any accesses.
  8. Click Done at the "Grant users access to this service account (optional)". Again, no need to grant any additional accesses.
  9. Go to Service Accounts. You'll see a list of all service accounts for the your GCP project and click on the service account you just created.
  10. Click on key --> add key --> Create new key and
  11. In the popup window that appears, select JSON as key type, click the Create button. A JSON file of the key you created will download.

Share access to the Google Drive folder where the DVC repo will live

Still on Google Cloud Platform,

  1. Go in IAM --> Service Accounts and click on the service account name you created in the previous section
  2. Copy and paste the email associated to this service account. It should have a form service_account_name@project_id.iam.gserviceaccount.com
  3. Move to Google Drive and share the drive folder where the DVC repo will live with the service account email defined in the last step.

Set DVC config to use a google service account

  1. On your local machine, modify the DVC project config file by typing the following commands. Change my_remote with your actual DVC remote name.

    dvc remote add --default my_remote gdrive://your_folder_hash

    dvc remote modify my_remote gdrive_use_service_account true

    dvc remote modify my_remote --local gdrive_service_account_json_file_path path/to/file.json

  2. This will modify the .dvc/config and .dvc/config.local files in your project.

  3. Run the command dvc push. If you see no error message, you are done with this step.

  4. Push your code to origin on Github

Setup a Repository Secret

  1. In your Github repo, go to Settings -> Secrets and Variables -> Actions
  2. Click on the New repository secret button.
  3. Name the secret to whatever is meaningful to you.
  4. Copy and paste the JSON file content you downloaded in the previous section to the Secret* text box.
  5. Click the Add Secret button

Setup a Github action

To your Github action yaml file, add the following (make sure to change the YOUR_REPO_SECRET_NAME to the name of the repo secret you created in the previous step.

  - uses: iterative/setup-dvc@v1
  
  - name: Pull data with DVC
    env:
      GDRIVE_CREDENTIALS_DATA: ${{ secrets.YOUR_REPO_SECRET_NAME }}
    run: |
      dvc pull

Commit this change and run the Github action!

References

Francis
  • 663
  • 1
  • 8
  • 13