Problem Statement

I'm encountering a 400 (Bad Request) error when attempting to upload a Kubeflow pipeline to Google Artifact Registry.

Environment Setup

kfp==2.10.1
kfp-pipeline-spec==0.5.0

Configuration

Repository Format: Kubeflow Pipeline IAM: artifactregistry.writer role and artifactregistry.admin roleconfigured Region: europe-west1

Reproducible Code Example

1. Initial Setup

(a repo need to be created first in artifact registry in kfp format)

from kfp import dsl, compiler
from kfp.registry import RegistryClient

ARTIFACT_REGISTRY_URL = "/my_project/ml-pipeline"
PIPELINE_OUTPUT_PATH = "pipeline.yml"

2. Pipeline Definition

@dsl.pipeline(
name="example_pipeline",
description="Example pipeline to demonstrate upload to Artifact Registry"
)
def my_pipeline():
\# Pipeline definition here
pass

3. Compilation and Upload

python

Compilation

pipeline_func=my_pipeline,
package_path=PIPELINE_OUTPUT_PATH
)

Upload

client = RegistryClient(host=ARTIFACT_REGISTRY_URL)
client.upload_pipeline(file_name=PIPELINE_OUTPUT_PATH)

Error Message

HTTPError: 400 Client Error: Bad Request for url: /my_project/ml-pipeline

Specific Questions

1. Are there any missing steps in the repository configuration?
2. How can I better debug this HTTP 400 error?

Troubleshooting Steps Taken

1. ✅ Verified pipeline.yml file generation

   • File generates without errors
   • YAML structure appears valid
   • No compilation errors reported

2. ✅ Checked IAM permissions

   • Confirmed artifactregistry.writer role
   • User account has necessary access
   • Service account permissions verified

3. ✅ Validated repository URL

   • URL format matches documentation
   • Repository exists and is accessible
   • Region configuration is correct

End Goal

Automate the upload of pipeline updates to Artifact Registry for automatic synchronization with Vertex AI.

Additional Context

I'm using the @pipeline decorator (versus @dsl.pipeline seen in some examples) The Artifact Registry repository is configured in Kubeflow Pipeline format All required IAM permissions are in place The YAML file generation completes successfully

thanks you !

Problem Statement

I'm encountering a 400 (Bad Request) error when attempting to upload a Kubeflow pipeline to Google Artifact Registry.

Environment Setup

kfp==2.10.1
kfp-pipeline-spec==0.5.0

Configuration

Repository Format: Kubeflow Pipeline IAM: artifactregistry.writer role and artifactregistry.admin roleconfigured Region: europe-west1

Reproducible Code Example

1. Initial Setup

(a repo need to be created first in artifact registry in kfp format)

from kfp import dsl, compiler
from kfp.registry import RegistryClient

ARTIFACT_REGISTRY_URL = "https://europe-west1-kfp.pkg.dev/my_project/ml-pipeline"
PIPELINE_OUTPUT_PATH = "pipeline.yml"

2. Pipeline Definition

@dsl.pipeline(
name="example_pipeline",
description="Example pipeline to demonstrate upload to Artifact Registry"
)
def my_pipeline():
\# Pipeline definition here
pass

3. Compilation and Upload

python

Compilation

pipeline_func=my_pipeline,
package_path=PIPELINE_OUTPUT_PATH
)

Upload

client = RegistryClient(host=ARTIFACT_REGISTRY_URL)
client.upload_pipeline(file_name=PIPELINE_OUTPUT_PATH)

Error Message

HTTPError: 400 Client Error: Bad Request for url: https://europe-west1-kfp.pkg.dev/my_project/ml-pipeline

Specific Questions

1. Are there any missing steps in the repository configuration?
2. How can I better debug this HTTP 400 error?

Troubleshooting Steps Taken

1. ✅ Verified pipeline.yml file generation

   • File generates without errors
   • YAML structure appears valid
   • No compilation errors reported

2. ✅ Checked IAM permissions

   • Confirmed artifactregistry.writer role
   • User account has necessary access
   • Service account permissions verified

3. ✅ Validated repository URL

   • URL format matches documentation
   • Repository exists and is accessible
   • Region configuration is correct

End Goal

Automate the upload of pipeline updates to Artifact Registry for automatic synchronization with Vertex AI.

Additional Context

I'm using the @pipeline decorator (versus @dsl.pipeline seen in some examples) The Artifact Registry repository is configured in Kubeflow Pipeline format All required IAM permissions are in place The YAML file generation completes successfully

thanks you !

• google-cloud-platform
• kubeflow-pipelines
• google-artifact-registry
• 400-bad-request

• Unfamiliar with KFP but familiar with Artifact Registry. The URL appears correct (I assume you have a Project my_project containing a KFP repository in europe-west1?). One debugging step you could try is to upload your artifact using a tool like curl, this will eliminate your Python code from the configuration. See example curl request. You'll need to impersonate the service account – DazWilkin Commented Nov 17, 2024 at 17:43
• My immediate reaction is that you're not (evidently) authenticating the request but you're also not receiving a 403. It's possible that the error is being obfuscated. I noticed that there is an auth property on kfp.registry.RegistryClient which isn't being set (unless it's leveraging ADC?) – DazWilkin Commented Nov 17, 2024 at 19:48
• I've configured my authentication using ADC by running gcloud auth application-default login and – Eliot Commented Nov 18, 2024 at 1:15

1 Answer 1

Use .yaml instead of .yml for the extension of the compiled file.

PIPELINE_OUTPUT_PATH = "pipeline.yaml"