Using the Pulumi Automation API without the Pulumi Service

Question

I am trying to build a CLI utilizing the Pulumi Automation API and would like to store the required Pulumi state in a GCP Storage Bucket (using the Pulumi CLI I can do this using pulumi login gs://some-bucket).

In the documentation I could not find information about this, the only login-related topic mentioned is the use of a Pulumi Access Token used to talk to the Pulumi Service - which I'm trying to avoid.

Does the option to use the Automation API without using the Pulumi Service even exist?

score 2 · Answer 1 · answered Nov 29 '22 at 09:29

There is a direct way to set backend configuration in the LocalWorkspaceOptions inside ProjectSettings.

from pulumi.automation import create_or_select_stack
from pulumi.automation import LocalWorkspace
from pulumi.automation import LocalWorkspaceOptions
from pulumi.automation import ProjectBackend
from pulumi.automation import ProjectSettings

# Option for local backend
backend_dir = 'path to your local backend directory'
backend_url = f'file://{backend_dir}'

# Option for remote backend in GCS
backend_url = 'gs://your-gcs-bucket'

stack = create_or_select_stack(
    stack_name=STACK_NAME,
    project_name=PROJECT_NAME,
    program=func_that_creates_resources,
    opts=LocalWorkspaceOptions(
        secrets_provider=SECRET_PROVIDER,
        work_dir=WORK_DIR,
        project_settings=ProjectSettings(
            name=PROJECT_NAME,
            runtime='python',
            backend=ProjectBackend(backend_url)
    )))


stack.up()

jaxxstorm · Answer 2 · 2022-09-14T18:18:39.430

1

The automation API doesn't manage the mechanism by which you login to Pulumi. You're expected to have selected your backend and logged into it before you start using it.

Once you've logged in, set a project name that doesn't use an organization features, and it should just work. Here's an example using the Python SDK


# create_pulumi_program is a function that has a Pulumi program in it
def pulumi_program():
  return create_pulumi_program(stack_name, image, int(port))


# create a workspace
stack = auto.create_stack(
  stack_name=str("dev"),
  project_name="my-project", # with the OSS backends, you cannot do "my-org/my-project"
  program=pulumi_program,
)

edited Sep 14 '22 at 18:18

answered Sep 14 '22 at 14:43

jaxxstorm

12,422
5
57
67

I think there's two issues: [1] Using the automation api with a file backend does not work for relative paths (it tries to locate the relative path in some temporary folders it creates), only for ~ and even then it still requires some configuration via env variables. I tried this approach earlier but gave up on it. [2] This approach does not help with bootstrapping: The usual chicken and egg problem - how to use the tool (pulumi/terraform) to bootstrap the backend. In the context of automation, this is very much something you'd like to do. – Gerrit Begher Sep 14 '22 at 22:14
Is there a way to switch the current backend via the automation API? – Gerrit Begher Sep 14 '22 at 22:17
1

@GerritBegher - yes, using the `PULUMI_BACKEND_URL` environment variable: https://github.com/pulumi/pulumi/pull/5789 Also see the official doco with an example here: https://www.pulumi.com/docs/reference/cli/environment-variables/#environment-variables – Thomas Anderson Nov 21 '22 at 00:54

Gerrit Begher · Answer 3 · 2022-09-14T23:18:06.603

I've had a similar problem and used the following but I don't understand it yet (first day with pulumi). Maybe it helps you to figure it out:

// Not sure if these need to be the same or if they have a different role
const STACK_PROJECT = "not-sure-what-this-projects-role-is"
const WORKSPACE_PROJECT= "not-sure-what-this-projects-role-is"

const stack = await LocalWorkspace.createOrSelectStack(
    {
      stackName: "some-stack-name",
      projectName: STACK_PROJECT,
      program: someInlineProgram,
    },
    {
      projectSettings: {
        name: WORKSPACE_PROJECT,
        // Does this need to be nodejs? Not sure...
        runtime: "nodejs",
        backend: {
          // Modify this for your purposes.
          // It seems to have the same effect as a prior `pulumi login {url}`
          url: "file://~",
        },
      },
    }
)

score 0 · Answer 4 · answered Nov 21 '22 at 01:09

As per my comment above, you can reference different Pulumi backends by setting the environment variable PULUMI_BACKEND_URL.

See the example in the Pulumi documentation here: https://www.pulumi.com/docs/reference/cli/environment-variables/#environment-variables

For your example, you could either:

have the PULUMI_BACKEND_URL environment variable already set, or
set the environment variable at runtime with:

import os
os.environ["PULUMI_BACKEND_URL"] = "gs://some-bucket"

Using the Pulumi Automation API without the Pulumi Service

4 Answers4