Segment-Anything to remove noise

[Ashoka74]

I am not familiar with 3D splats and the feasibility of this proposition but would it be possible to use segment-anything to recognise the main object on 2D projection with a click and remove all noise outside coordinates of the contour area?

[slimbuck]

Yeah it would be great to have something like this.

We did a quick test with an open source implementation of dbscan (a clustering algorithm) which looked promising, but ran extremely slowly.

We'll be trying a few things when we have time, but feel free to have a go yourself too. This is probably the functionality we need most.

[TonyDua]

Yeah it would be great to have something like this.

We did a quick test with an open source implementation of dbscan (a clustering algorithm) which looked promising, but ran extremely slowly.

We'll be trying a few things when we have time, but feel free to have a go yourself too. This is probably the functionality we need most.

Yes! I really need clustering algorithms like DBSCAN to remove outliers in 3DGS. Manual cleaning is very time-consuming and meticulous. I once implemented a DBSCAN-like clustering algorithm in Metashape to remove outliers, but encountered the problem of slow computation. I tried adding KDTree, but the speed was still slow. Later, I tried a filtering function where I selected a reference point in the viewport, specified a distance, created a box with sides of that distance, selected it in the viewport, and then ran DBSCAN. This approach was much faster. External noise points can be directly removed using features like rectangular selection boxes, but it's difficult to quickly select outliers on the main surface. By only computing points within the box I set (i.e., the main body range), I solved the problem of slow speed. This script runs very well, and I also added a feature to filter points by color.

Unfortunately, due to the algorithmic characteristics of 3DGS, it only requires a sparse point cloud for Gaussian model initialization. Even if I clean the alignment points in Metashape exceptionally well, 3DGS training will still learn these points back, which is its robustness. It's really frustrating. So, I started using your super-splat tool to clean up 3DGS, but it would be great if you could incorporate DBSCAN to quickly select outliers!

Oh, thank you again for developing super-splat. It's truly an amazing tool! If you have any ideas about using DBSCAN to clean outliers, I would be very happy to discuss with you! Thanks again!

[slimbuck]

Ideally floaters would be removed at the training stage so we don't have deal with this at all.

BTW have you tried using the new "rings" mode for removing floaters? I find it's easier and faster than before.

[TonyDua]

Ideally floaters would be removed at the training stage so we don't have deal with this at all.

BTW have you tried using the new "rings" mode for removing floaters? I find it's easier and faster than before.

I just tested the rings stage! It turns out that I can select those abnormal splatter points, and now the selection is very fast. I originally thought this stage was just for some artistic style rendering display, but I didn't expect it to be used like this. Honestly, I had a dataset of rocks before, with many points splattered outside the main body of the rocks. I tried filtering by size or transparency, but couldn't find them. Now this method works! Thanks.

Also, you mentioned that those unwanted points can be removed from the training stage. Actually, I did the same. Because I found that 3DGS and Metashape can accept images with alpha channels. So I removed the sky from my outdoor scenes in advance, using PNG image format. This way, I don't need to remove those annoying white skies in super-splat. However, there are some points that cannot be removed during the training stage, such as those floating points between the contact edges of the main body of the rocks and the background, as well as the gaps between the edges of the rocks. These points generate floating outliers on the main body of the rocks during the alignment process. After entering the 3DGS training, these points cause ghosting. I've tested it, and deleting these points in Metashape doesn't work. I even tried the most extreme case, deleting half of the point cloud of the aligned scene, and it didn't affect the 3DGS training. The scene was completely restored, and the ghosting was still there. So I feel that there is a great demand to include a DBSCAN-like algorithm in super-splat to clean up points floating on the main body of the rocks.

[RobLinkA]

I've tried with SAMURAI which is based on SAM2 and it really worked!



No need to edit or remove noise. All SH tightly adhere to the model surface, with no strange shadows caused by large-radius ellipsoids, especially the base surface is very smooth and flat.
You can check the result here: https://3d.explorerglobal.cn/model/6l7vdmlj

[RobLinkA]

I first convert consecutive photos into a video using ffmpeg, then run SAMURAI, which helps achieve better mask continuity.