Vertex AI "Bigquery resource does not exist error" for custom training job

Question

Currently running custom training job in Vertex AI with dataset in BigQuery in the same project. This was working last week but is now coming up with "Bigquery resource does not exist error" with no changes to configuration/permissions.

This seems to be happening when Vertex AI creates the train, test and validation tables. The dataset for the tables is being created but not the tables themselves. Since the dataset is empty, Vertex AI cannot find the tables and comes back with "Bigquery resource does not exist error. It's not clear why this has started to occur.

Pipeline being used to run custom job:

import kfp
from kfp.v2 import compiler, dsl
from kfp.v2.dsl import pipeline

from google.cloud import aiplatform
from google_cloud_pipeline_components import aiplatform as gcc_aip

PIPELINE_ROOT = gs://gcs_bucket/custom_pipelines

@kfp.dsl.pipeline(name=PIPELINE_NAME,
                  pipeline_root=PIPELINE_ROOT)
def pipeline(
    bq_source: str = BQ_SOURCE,
    bucket: str = BUCKET_NAME,
    project: str = PROJECT_ID,
    gcp_region: str = REGION,
    bq_dest: str = "",
    container_uri: str = "",
):
    dataset_create_op = gcc_aip.TabularDatasetCreateOp(
        display_name=display_name,
        bq_source=bq_source,
        project=project,
        location=gcp_region
    )

    training_op = gcc_aip.CustomContainerTrainingJobRunOp(
        display_name=PIPELINE_NAME,
        container_uri=container_uri,
        project=project,
        location=gcp_region,
        dataset=dataset_create_op.outputs["dataset"],
        base_output_dir=PIPELINE_ROOT,
        staging_bucket=gs://gcs_bucket,
        training_fraction_split=0.8,
        validation_fraction_split=0.1,
        test_fraction_split=0.1,
        timestamp_split_column_name="timestamp",
        bigquery_destination=f"bq://{PROJECT_ID}",
        machine_type="n1-highmem-4",
    )