[backend] Cannot create new pipeline or upload new version to existing pipelines

[yangyang919]

Environment

• How did you deploy Kubeflow Pipelines (KFP)? Full deployment
• KFP version: 1.6
• KFP SDK version: 1.6.0

Steps to reproduce

Previously, everything looks fine. But all of a sudden, couldn't upload pipelines or create pipelines either from UI or from cli. The error messages are like follows:
Upload version to existing pipeliens:

I0415 03:29:46.118884       7 pipeline_upload_server.go:144] Upload pipeline version called
E0415 03:29:46.142425       7 pipeline_upload_server.go:240] Failed to upload pipelines. Error: Error creating pipeline version: Create pipeline version failed: InternalServerError: Failed to add version to pipeline version table: Error 1364: Field 'Description' doesn't have a default value: Error 1364: Field 'Description' doesn't have a default value

Create new pipelines:

E0415 03:30:50.340822 7 pipeline_upload_server.go:240] Failed to upload pipelines. Error: Error creating pipeline: Create pipeline failed: InternalServerError: Failed to add pipeline version to pipeline_versions table: Error 1364: Field 'Description' doesn't have a default value: Error 1364: Field 'Description' doesn't have a default value

No matter which yaml I use, the one generated by kfp cli, or the one in sample code, all get same errors. It looks like a global issue instead of the format in one single yaml file. No clue at why it happened...thanks for checking...

[yangyang919]

The issue is solved with a workaround.

First kubectl exec mysql-pod -it -- mysql -h mysql

Then change the column of Description in Table pipeline_versions to accept NULL values

ALTER table pipeline_versions MODIFY Description longtext NULL;

Problem solved. Still don't know why issue happened why it's working for quite some time. I suspect there are some mismatch between frontend and backend codes, which makes the description value does not get passed correctly from UI to db

[Linchin]

Maybe you could check your mysql config and see if there's any recent change? In kfp the description should be optional.

[gioargyr]

Hello,

I am reopening this issue as I had a similar one.
In my case, when I tried to create a Kubeflow pipeline, I go the error:

HTTP response body: {"error_message":"Error creating pipeline: Create pipeline failed: InternalServerError:
 Failed to add pipeline version to pipeline_versions table: Error 1364:
 Field 'PipelineSpec' doesn't have a default value

In my case the installation of Kubeflow was done by following your manifests's version 1.7.0
I found in namespace kubeflow, a pod mysql.
From inside the pod, I acted on database mlpipeline on table pipeline_versions:
First, I executed this command (mysql command line):
ALTER table pipeline_versions MODIFY PipelineSpec longtext NULL;

but, after I got a new error for column PipelineSpecURI

HTTP response body: {"error_message":"Error creating pipeline: Create pipeline failed: InternalServerError: Failed to add pipeline version to pipeline_versions table: Error 1364: Field 'PipelineSpecURI' doesn't have a default value

I also executed the same command for column PipelineSpecURI:
ALTER table pipeline_versions MODIFY PipelineSpecURI longtext NULL;

After those 2 modification on table pipeline_versions on database mlpipeline, my pipelines now run successfully.

Do you have any comments on this?