How to use relative dependencies when deploying to App Engine with Cloud Build

Question

I have a full stack project that have some common modules, mostly type definitions. I want to keep them in the same repository to keep a consistent state of my application. However, the deploying doesn't work as I expect it. During my first two build steps yarn install and yarn build there are no apparent issues with my relative dependency.

Gcloud, however, doesn't seem to find it. At the 6th build step it shows the follwing error message:

Gcloud building steps

Already have image (with digest): eu.gcr.io/gae-runtimes/buildpacks/nodejs12/builder:nodejs12_20210308_12_21_0_RC00
=== Node.js - Yarn (google.nodejs.yarn@0.9.0) ===
--------------------------------------------------------------------------------
Running "bash -c command -v yarn || true"
/usr/bin/yarn
Done "bash -c command -v yarn || true" (53.639626ms)
DEBUG: Yarn is already installed, skipping installation.
--------------------------------------------------------------------------------
Running "node -v"
v12.21.0
Done "node -v" (35.463448ms)
DEBUG: Current dependency hash: "8d626f89a4d8d27dba2aab32f998ebdf2b45531f6abae7f55eabb00d85cff016"
DEBUG:   Cache dependency hash: "e6c8baadd1eb6d1fd69fc3255ba5936ea43d709f6502d54719fdb17c9d9865af"
Installing application dependencies.
DEBUG: ***** CACHE MISS: "prod dependencies"
--------------------------------------------------------------------------------
Running "node -v"
v12.21.0
Done "node -v" (5.746529ms)
--------------------------------------------------------------------------------
Running "yarn install --non-interactive --frozen-lockfile (NODE_ENV=production)"
yarn install v1.22.4
[1/4] Resolving packages...
error Package "common" refers to a non-existing file '"/common"'.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Done "yarn install --non-interactive --frozen-lockfile (NODE_ENV=p..." (528.261919ms)
Failure: (ID: 98e389b6) yarn install v1.22.4
[1/4] Resolving packages...
error Package "common" refers to a non-existing file '"/common"'.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
--------------------------------------------------------------------------------
Running "mv -f /builder/outputs/output-5577006791947779410 /builder/outputs/output"
Done "mv -f /builder/outputs/output-5577006791947779410 /builder/o..." (15.651484ms)
ERROR: failed to build: exit status 1

Folder structure:

client/
    package.json
    cloudbuild.json
common/
    package.json
server/
    package.json
    cloudbuild.json

client/package.json

{
    "name": "client",
    "dependencies": {
        "common": "file:../common"
    }
}

client/cloudbuild.json

{
    "steps": [
        {
            "name": "node",
            "entrypoint": "yarn",
            "args": ["--cwd", "client", "install"]
        },
        {
            "name": "node",
            "dir": "client",
            "entrypoint": "yarn",
            "args": ["run", "build"]
        },
        {
            "name": "gcr.io/google.com/cloudsdktool/cloud-sdk",
            "entrypoint": "bash",
            "dir": "client",
            "args": ["gcloud", "app", "deploy"]
        }
    ],
    "timeout": "1600s"
}

How do I need to adjust my cloudbuild file so the builder will find my relative dependency?

Cloud Build runs on a Source Repository (Github) commit trigger. In the trigger I specify the location of client/cloudbuild.json — Maroben, Mar 19 '21 at 08:34

score 2 · Accepted Answer · answered Mar 19 '21 at 20:18

You should combine everything (client, server and common) into a single Cloud Build config.

When you gcloud build submit, the source directory is transferred to the Cloud Build service (VM) and placed in a directory called /workspace. If you want to be able to make references to common from client or server then this directory must be copied as part of their build process. To save energy, it's convenient to build common once and then build client and server referencing it.

You would then able to build the client and server by referencing common because it's been transferred too.

You are able to produce multiple images from one Cloud Build job and so you should produce one for the client and one for the server.

Folder structure:

cloudbuild.json
client/
    package.json

common/
    package.json
server/
    package.json

Alternatively and I'm not as familiar with Node.JS package management, you could (Cloud) Build common first and make this accessible to the (Cloud) Builds for client and server. Customarily this would be through some (network-accessible) private package management solution that client and server would reference as part of their Build.

Could you give me an example when you mean "must be copied as part of their build process"? I don't think I have any means to change gclouds app deploy process and anything that happens before the deploy doesn't seem to affect later parts of the build. — Maroben, Mar 23 '21 at 18:43
If you want to build e.g. `client` and it depends upon `common`, then you must bundle `common` with `client` (both directories must be present) in the Cloud Build step. — DazWilkin, Mar 23 '21 at 19:52
It is possible to reuse the file-system across steps. By default, each Cloud Build job (e.g. VM) has a default `/workspace` directory that is mounted into each Build step as a volume mount and becomes the step's working directory. — DazWilkin, Mar 23 '21 at 19:54

How to use relative dependencies when deploying to App Engine with Cloud Build

1 Answers1