Creating a virtualenv on circleci can't be done in separate step in a job/workflow. Why is this?

Question

I'm trying to install requirements in a build step and run tests on circleci. I run into a permission denied problem when running the pip install because it tries to install the requirements globally. I could install the requirements on a user level, but using virtualenv also works:

version: 2
defaults: &defaults
  docker:
    - image: circleci/python:3.6
jobs:
  build_dataloader:
    <<: *defaults
    working_directory: ~/app/dataloader
    steps:
      - checkout:
          path: ~/app
      - run:
          name: Install requirements
          command: |
            virtualenv env
            source env/bin/activate
            pip install -r requirements.txt
  dataloader_tests:
    <<: *defaults
    parallelism: 2
    steps:
      - checkout:
          path: ~/app
      - run:
          name: Running dataloader tests
          command: |
            cd ~/app/dataloader
            python3 -m unittest discover tests/unit/
      - store_artifacts:
          path: test-reports/
          destination: app_tests


workflows:
  version: 2
  run_tests:
    jobs:
      - build_dataloader
      - dataloader_tests:
          requires:
            - build_dataloader

Is there any way to put the virtualenv in a separate step? This does not work when I put the virtualenv part in a separate step:

version: 2
defaults: &defaults
  docker:
    - image: circleci/python:3.6
jobs:
  build_dataloader:
    <<: *defaults
    working_directory: ~/app/dataloader
    steps:
      - checkout:
          path: ~/app
      - run:
          name: Setup virtualenv
          command: |
            virtualenv env
            source env/bin/activate
      - run:
          name: Install requirements
          command: |
            pip install -r requirements.txt
  dataloader_tests:
    <<: *defaults
    parallelism: 2
    steps:
      - checkout:
          path: ~/app
      - run:
          name: Running dataloader tests
          command: |
            cd ~/app/dataloader
            python3 -m unittest discover tests/unit/
      - store_artifacts:
          path: test-reports/
          destination: app_tests


workflows:
  version: 2
  run_tests:
    jobs:
      - build_dataloader
      - dataloader_tests:
          requires:
            - build_dataloader

Why does this fail? Shouldn't it create the virtual env if I put it in a separate step?

score 1 · Accepted Answer · answered Nov 06 '19 at 16:32

1

I think the problem with this approach is that every step runs with its own shell hence the result of source env/bin/activate is lost when the step is finished; pip install runs in a different shell where the virtualenv is not activated. Try this instead:

 - run:
      name: Setup virtualenv
      command: |
        virtualenv env
  - run:
      name: Install requirements
      command: |
        source env/bin/activate
        pip install -r requirements.txt

Or simply env/bin/pip install -r requirements.txt without any activation.

answered Nov 06 '19 at 16:32

phd

82,685
13
120
165

Where does it say in the docs that every step runs in its own shell? – Jwan622 Nov 06 '19 at 16:45
Thanks so much for this answer! – Jwan622 Nov 06 '19 at 16:45

Creating a virtualenv on circleci can't be done in separate step in a job/workflow. Why is this?

1 Answers1