chore: Seaweedfs as Minio replacement

[juliusvonkohout]

Description of your changes:

closes kubeflow/manifests#3119

Thank you everyone!
I am pushing this for 4 years or so and even had google and redhat employees involved, back then even Amazon. It is fundamental for CVEs, maintainabiliy (minio is now stuck for 5 years or so) and hard multi-tenancy as basic requirement for an enterprise platform. We also had approaches there for several years with minio. It started all in 2020 here #4649 and went via #7725 (2022) and kubeflow/manifests#2826 (October 2024) to kubeflow/manifests#3051 (2025). Without that experimental and extended tests it would have been very hard to pull of and coordinate. I want to especially highlight @pschoen-itsc who spent his effort here for the public health sector in Germany where many insurances need hard multi-tenancy to process data.

We evaluated many alternatives and now we have something S3 and IAM policy compatible, scalable and with hard multi-tenancy.

@akagami-harsh you can create branches against this PR.

• use seaweedfs
• replace the old sync.py etc.
• refactor our tests to be usable within KFP
• Remove Minio (including /env/azure and other legacy stuff)
• Explain how to use the seaweedfs gateway for AWS/GCP/Azure S3

[google-oss-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[juliusvonkohout]

closes #7725

[juliusvonkohout]

We are still missing
Explain how to use the seaweedfs gateway for AW3/GCP/Azure S3 (Harshvir)
add an architectural diagram here for minio and in general for kubeflow/manifest as from my kubecon presentations and blogs (Julius)

but it is at least ready for a first review @HumairAK

[juliusvonkohout]

/retest

[droctothorpe]

This is incredibly comprehensive and impressive, @juliusvonkohout and @akagami-harsh. What obstacles do you anticipate for end users upgrading from Minio to Seawead? Does it make sense to provide migration documentation or automation?

[juliusvonkohout]

This is incredibly comprehensive and impressive, @juliusvonkohout and @akagami-harsh. What obstacles do you anticipate for end users upgrading from Minio to Seawead? Does it make sense to provide migration documentation or automation?

If users really need the old data then the cluster administrator needs to handle it anyway by copying via boto3 from minio to seaweedfs. So that is something we could deal with in follow up PRs. We could provide a job/cronjob that does this automatically. i think even LLMS can write this.

More interesting is probaly the seaweedfs gateway documentation link that shows how users can connect seaweefdfs to AWS/GCP/Azure/S3 compatible object storage. But also there i prefer a follow up PR. Lets /approve and merge what we have @HumairAK @hbelmiro @droctothorpe and continue in follow up PRs.

[juliusvonkohout]

/lgtm
@HumairAK for approval

[akagami-harsh]

/lgtm
@HumairAK for approval

Hopefully everything adressed

[akagami-harsh]

@akagami-harsh I noticed this flakiness occur in this CI failure https://github.com/kubeflow/pipelines/actions/runs/16993993913/job/48180413365?pr=11965

I noticed the seaweedfs pod was filled with:

2025-08-15T16:40:38.8534567Z I0815 16:39:12.797939 master_grpc_server_volume.go:141 volume grow &{Option:{"collection":"mlpipeline","replication":{},"ttl":{"Count":0,"Unit":0},"preallocate":1073741824,"version":3} Count:0 Force:false Reason:grpc assign}
2025-08-15T16:40:38.8535444Z E0815 16:39:12.801327 volume_grpc_admin.go:59 assign volume volume_id:82  collection:"mlpipeline"  preallocate:1073741824  replication:"000"  version:3: No more free space left
2025-08-15T16:40:38.8536338Z W0815 16:39:12.802934 volume_growth.go:273 Failed to assign volume 82 on topo:DefaultDataCenter:DefaultRack:10.244.0.26:8080: rpc error: code = Unknown desc = No more free space left
2025-08-15T16:40:38.8537464Z I0815 16:39:12.803731 volume_growth.go:120 create 7 volume, created 0: failed to assign volume 82 on topo:DefaultDataCenter:DefaultRack:10.244.0.26:8080: rpc error: code = Unknown desc = No more free space left
2025-08-15T16:40:38.8538277Z I0815 16:39:12.815178 master_grpc_server_volume.go:141 volume grow &{Option:{"collection":"mlpipeline","replication":{},"ttl":{"Count":0,"Unit":0},"preallocate":1073741824,"version":3} Count:0 Force:false Reason:grpc assign}

I also notice that the "Free up Disk Space" step took 8 min 39s to complete, which is odd given the others complete much faster. Still, the final disk usage reports the same - so I'm not sure it's related but it was the only one that stood out.

Not sure if this is seaweedFS specific, but looking at the history of this workflow, I couldn't find us encountering this behavior with minio in recent history. Thoughts?

I think lots of files were deleted in the free-disk space step, creating fragmented free space, there is a preallocation feature in seaweedfs. Preallocation needs large contiguous blocks, not just total free space. logs show preallocate:1073741824 (1GB) - it needs 1GB of contiguous space

2025-08-15T16:40:38.8534567Z I0815 16:39:12.797939 master_grpc_server_volume.go:141 volume grow &{Option:{"collection":"mlpipeline","replication":{},"ttl":{"Count":0,"Unit":0},"preallocate":1073741824,"version":3} Count:0 Force:false Reason:grpc assign}
2025-08-15T16:40:38.8535444Z E0815 16:39:12.801327 volume_grpc_admin.go:59 assign volume volume_id:82  collection:"mlpipeline"  preallocate:1073741824  replication:"000"  version:3: No more free space left
2025-08-15T16:40:38.8536338Z W0815 16:39:12.802934 volume_growth.go:273 Failed to assign volume 82 on topo:DefaultDataCenter:DefaultRack:10.244.0.26:8080: rpc error: code = Unknown desc = No more free space left
2025-08-15T16:40:38.8537464Z I0815 16:39:12.803731 volume_growth.go:120 create 7 volume, created 0: failed to assign volume 82 on topo:DefaultDataCenter:DefaultRack:10.244.0.26:8080: rpc error: code = Unknown desc = No more free space left
2025-08-15T16:40:38.8538277Z I0815 16:39:12.815178 master_grpc_server_volume.go:141 volume grow &{Option:{"collection":"mlpipeline","replication":{},"ttl":{"Count":0,"Unit":0},"preallocate":1073741824,"version":3} Count:0 Force:false Reason:grpc assign}

After disk cleanup, the disk likely has plenty of total free space but it's fragmented, i think this would explain why MinIO don't hit this issue

[HumairAK]

@akagami-harsh I'm skeptical that this is due to defragmentation, and it's likely due to the storage size we are setting in the pvc for seaweedfs, see here. Can we set it to the same size as the minio pvc at 20Gi here?

[akagami-harsh]

@akagami-harsh I'm skeptical that this is due to defragmentation, and it's likely due to the storage size we are setting in the pvc for seaweedfs, see here. Can we set it to the same size as the minio pvc at 20Gi here?

@HumairAK, Opened a pr to update pvc volume size #12156

[google-oss-prow]

New changes are detected. LGTM label has been removed.

[HumairAK]

thanks @akagami-harsh, this lgtm, can you squash your commits and add a meaningful clean commit message?

[juliusvonkohout]

thanks @akagami-harsh, this lgtm, can you squash your commits and add a meaningful clean commit message?

The commits will be automatically squashed on merge with the PR title as commit message, so i would like to avoid any further changes.

also given multiple authors it is a bit more complicated.

[HumairAK]

I updated the commit message to a more meaningful message - in general I prefer to have the PR authors curate their message as they have more domain knowledge.

Thank you everyone for all your hard work on this - a much needed change, well done all around !

[droctothorpe]

Congrats, everyone! Phenomenal (and overdue) enhancement! You should consider submitting a talk on this to the next Kubeflow Summit in Europe.