Is there any way to use compose v2 in bitbucket pipeline?

Question

First of all - no, I cannot switch from Bitbucket pipelines to something appropriate, unfortunately, it is direct requirement.

[x] I have searched other SO questions and google, the following two questions are related:

• Bitbucket Pipeline - docker compose error (no answer)
• How to use docker compose V2 in Bitbucket Pipelines (answer not working even when literally copied to pipeline definition for one of reasons below)

Working v1 main pipeline (only significant step and job, of course, it is larger)

image: python:3.10

definitions:
  steps:
    - step:  &run-tests
        name: Test
        image: docker/compose:debian-1.29.2
        caches:
          - docker
        services:
          - docker
        script:
          - COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose --project-name reporting --env-file .env.ci -f docker-compose.ci.yaml up -d --build
          # - ... (wait until ready and run tests, ignored, because error happens earlier)

pipelines:
  default:
    - parallel:
      - step: *run-tests

Encountered errors

I'll to refer to them multiple times, so let's define short aliases:

403

+ COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose --project-name reporting --env-file .env.ci -f docker-compose.ci.yaml up -d --build
listing workers for Build: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing unable to upgrade to h2c, received 403"

priviliged

+ COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose --project-name reporting --env-file .env.ci -f docker-compose.ci.yaml up -d --build
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 2.8s done
#1 creating container buildx_buildkit_default 0.0s done
#1 ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed
------
 > [internal] booting buildkit:
------
Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

Unfortunately, there is no docker/compose v2 image, and our deployment uses v2, so some inconsistencies happen. I'm trying to use v2 in pipeline now. I replaced docker-compose references with docker compose and try to prevent this command from crashing. Important thing to note: I need docker buildkit and cannot go without it, because I'm using Dockerfile.name.dockerignore files which are separate for prod and dev, and docker without buildkit does not support it (builds will simply fail).

Things I tried (debug smts like docker version and docker compose version were always working OK in these cases):

• using image: linuxserver/docker-compose:2.10.2-v2. Result: 403.
• using image: library/docker:20.10.18.
  • No more changes. Result: privileged.
  • Add docker buildx create --driver-opt image=moby/buildkit:v0.10.4-rootless --use as a step. Result: privileged (logs show that this image is actually used: pulling image moby/buildkit:v0.10.4-rootless 6.3s done).
• using no explicit image (relying on bitbucket docker installation).
  • with official compose installation method (result: 403):

    - mkdir -p /usr/local/lib/docker/cli-plugins/
    - wget -O /usr/local/lib/docker/cli-plugins/docker-compose https://github.com/docker/compose/releases/download/v2.10.2/docker-compose-linux-x86_64
    - chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
    

  • with solution from 2nd link above (result: 403, but with some portion of success: downloaded two services that do not require building - postgres and redis - and failed only then)

If it is important, compose file for CI (only healthchecks trimmed, everything else not touched):

# We need this file without volumes due to bitbucket limitations.

version: '3.9'

services:
  db:
    image: mariadb:10.8.3-jammy
    env_file: .env.ci
    volumes:
      - ./tests/db_init/:/docker-entrypoint-initdb.d
    networks:
      - app_network

  redis:
    image: redis:alpine
    environment:
      - REDIS_REPLICATION_MODE=master
    networks:
      - app_network

  app:
    build:
      context: .
      args:
        - APP_USER=reporting
        - APP_PORT
    env_file: .env.ci
    depends_on:
      - db
      - redis
    networks:
      - app_network

  nginx:
    build:
      context: .
      dockerfile: configs/Dockerfile.nginx
    env_file: .env.ci
    environment:
        - APP_HOST=app
    ports:
      - 80:80
    depends_on:
      - app
    networks:
      - app_network


networks:
  app_network:
    driver: bridge

For now I reverted everything and keep using v1. The limitations of bitbucket pipelines drive me mad, I can easily run the same stuff in github actions, but now have to remove one service (that uses docker directory mounting, so cannot run on bitbucket) and spend whole day trying to upgrade compose. Sorry for this tone, this really makes me desire to quit bitbucket forever and never touch it again.