feat(ui): SAM2 Node & Integration #8526
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hipsterusername
requested review from
blessedcoolant,
lstein,
maryhipp and
psychedelicious
as code owners
github-actions
bot
added
python
There was a really confusing aspect of the SAM pipeline classes where they accepted deeply nested lists of different dimensions (bbox, points, and labels). The lengths of the lists are related; each point must have a corresponding label, and if bboxes are provided with points, they must be same length. I've refactored the backend API to take a single list of SAMInput objects. This class has a bbox and/or a list of points, making it much simpler to provide the right shape of inputs. Internally, the pipeline classes take rejigger these input classes to have the correct nesting. The Nodes still have an awkward API where you can provide both bboxes and points of different lengths, so I added a pydantic validator that enforces correct lenghts.
Revised the Select Object feature to support two input modes: - Visual mode: Combined points and bounding box input for paired SAM inputs - Prompt mode: Text-based object selection (unchanged) Key changes: - Replaced three input types (points, prompt, bbox) with two (visual, prompt) - Visual mode supports both point and bbox inputs simultaneously - Click to add include points, Shift+click for exclude points - Click and drag to draw bounding box - Fixed bbox visibility issues when adding points - Fixed coordinate system issues for proper bbox positioning - Added proper event handling and interaction controls 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Fixed an issue where bounding boxes could grow exponentially when created at small sizes. The problem occurred because Konva Transformer modifies scaleX/scaleY rather than width/height directly, and the scale values weren't consistently reset after being applied to dimensions. Changes: - Ensure scale values are always reset to 1 after applying to dimensions - Add minimum size constraints to prevent zero/negative dimensions - Fix scale handling in transformend, dragend, and initial bbox creation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…dule When middle mouse button is used for canvas panning, the pointerup event was still creating points in the segmentation module. Added button check to onBboxDragEnd handler to only process left clicks. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Added button checks to bbox rect and transformer mousedown/touchstart handlers to only process left clicks. Also added stage dragging check in onBboxDragMove to clear bbox drag state when middle mouse panning is active. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
need to add translation strigns for new functionality
psychedelicious
force-pushed
the
SAM2
branch
from
e2fafc2 to
a97ee7a
Compare
This actually works fine for SAM.
psychedelicious
approved these changes
Sep 11, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This pull request introduces support for the Segment Anything 2 (SAM2) model in the backend, enabling advanced image segmentation capabilities. It adds a new invocation for SAM2, a pipeline wrapper for model management, updates frontend integration to use the new model, and bumps the
transformersdependency to ensure SAM2 availability.Backend: Segment Anything 2 (SAM2) integration & Node
Frontend: Canvas module update
segment-anything-large) and changed mask filtering to include all detected masks (this seemed to produce better results in the testing I did)Dependency update
transformerslibrary version to>=4.56.0inpyproject.tomlRelated Issues / Discussions
Requests.
QA Instructions
Test out the SAM2 Node, ensure both bounding box inputs and points work correctly.
Merge Plan
Merge when ready. Ensure we have any/all necessary model changes in place downstream
Checklist
What's Newcopy (if doing a release after this PR)