0

I need to compile a CMake based C++ project using GCC. It depends on MKL, and for successful Cmake configuration, compilation and test execution, I need run the following comands beforehand

        source /opt/intel/bin/compilervars.sh -arch intel64
        export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

In order to run this with Azure pipelines I have a container which I am able to run based on the documentation from https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops.

Normally the above mentioned setup scripts would be called during container startup (https://hub.docker.com/layers/vvtk/vvcoreazurelinuxdockeragent/latest/images/sha256-c5e3775546ee90a111c9ae700306eb4cd1ebc710686bda5011633c4e5e883e13?context=repo) however it seems (as also described in https://stackoverflow.com/a/63643979/15128314 this startup CMD command is not execute since Azure pipelines do not actually call docker run

As a result I am forced to replicate this into multiple steps of my pipeline job (basically each of config, build and test), since these env vars are also not persistent across the different steps. How can I solve this more efficiently? The pipeline looks horrible ..

  - script: |
      source /opt/intel/bin/compilervars.sh -arch intel64
      export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
      (more cmds here)
    displayName: config_Linux_x64_Release

  - script: |
      source /opt/intel/bin/compilervars.sh -arch intel64
      export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
      (more cmds here)
    displayName: build_Linux_x64_Release
  
  - script: | 
        source /opt/intel/bin/compilervars.sh -arch intel64
        export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
        (more cmds here)
    displayName: test_Linux_x64_Release
iv1
  • 45
  • 5

1 Answers1

0

You can try using the logging command(ie. ##vso[task.setvariable]) to set the system variable and avoid the replication for export command. See below:

You can define environment variable in the pipeline Variable section like below:

variables:
  LD_LIBRARY_PATH: /usr/local/lib:$LD_LIBRARY_PATH

Or you can try adding a script task in the top of your pipeline to run below command:

- script: |
  
    #below script will only take effect in the following tasks. 
      
    source /opt/intel/bin/compilervars.sh -arch intel64
    echo "##vso[task.setvariable variable=LD_LIBRARY_PATH]/usr/local/lib:$LD_LIBRARY_PATH" 

  displayName: 'SetVariable'

Note: the variable set up in above script task only takes effect in the following task.

Please refer to this thread.

Update:

For source command only applies for current shell, and each script step will launch a new shell. So you will still need to repeat source commands in each script step.

Or you can run config, build, test in a single script step to avoid replication.

Levi Lu-MSFT
  • 27,483
  • 2
  • 31
  • 43
  • this would only work for `LD_LIBRARY_PATH` but does not persist whatever `source /opt/intel/bin/compilervars.sh -arch intel64` actually sets, so no it does not completely solve this issue. Similarly for Virtual environments with python, see also https://medium.com/faun/azure-pipelines-and-python-virtual-environments-fdc1089656db – iv1 Feb 14 '21 at 21:51
  • @multipitch according to https://stackoverflow.com/a/60664697/15128314 you were able to use virtual environments across different steps. I cannot really make this work, see also https://medium.com/faun/azure-pipelines-and-python-virtual-environments-fdc1089656db - did this really solve your issue? – iv1 Feb 14 '21 at 22:08
  • @iv1 I am afraid you have to repeat `source` command in each script step. This is because each script step will launch a new shell, and previous shell will be closed. – Levi Lu-MSFT Feb 16 '21 at 07:20
  • ok I see - thanks anyways! I wanted to avoid clubbing everything into the same step, as I need to check success for each of them and abort execution if necessary – iv1 Feb 17 '21 at 15:45