SAHI Bottleneck Issue

[bagikazi]

I have high-end pc to run predictions with Cam. Since I'm getting max 3 fps at %15GPU load and 7-8%CPU load. Without SAHI at test runs I can able to get around 90fps. Is there anything I'm missing by building my structure with SAHI. Gemini says whole process waiting for SAHI to complete slices meanwhile GPU is not used. Is there any way to express to load on GPU.

[bagikazi]

PROBLEM SOLVED!

After investigating and optimizing this issue, I've developed a solution that provides 5x performance improvement:

Before vs After Results:

• FPS: 3 → 14+ FPS
• GPU Utilization: 15% → 80%+
• Processing Method: Sequential → Batched

Root Cause Analysis:
SAHI's default get_sliced_prediction() function was processing slices sequentially:

Old method

for slice in slices:
result = model(slice) # Separate GPU transfer per slice

 **Solution: Batched GPU Inference**
The new approach sends all slices to GPU **in a single batch**:

# New method
batch_tensor = torch.cat([transform(s) for s in slices])
batch_results = model(batch_tensor)  # Single GPU call

Pull Request:
I've opened a PR to contribute this solution to the SAHI community:
**[feat: Add batched GPU inference for 5x performance improvement #1222] (#1222)

Implementation Details:

• Uses sahi.slicing.slice_image for image slicing (fast)
• Manual batched inference with PyTorch
• Coordinate transformation back to original image space
• Fully backward compatible implementation
  Community Impact:
• Real-time SAHI applications now feasible
• Much better GPU hardware utilization
• Foundation for future performance optimizations

The issue was that SAHI was underutilizing GPU resources by:

1. Individual tensor transfers per slice (high overhead)
2. Sequential processing instead of parallel batching
3. GPU idle time between slice inferences

The solution batches all slices into a single tensor, performs one GPU inference call, then maps results back to original coordinates.

Performance Metrics:

Metric	Before	After	Improvement
FPS	3	14+	5x
GPU Util	15%	80%+	4x
Processing Time	0.33s	0.045s	87% faster

For Others Facing This Issue:
If you're experiencing similar low GPU utilization with SAHI:

1. Check the PR above for the batched inference implementation
2. Consider using sahi.slicing.slice_image + manual batching
3. Monitor GPU utilization during inference

Hope this helps anyone else facing similar performance bottlenecks with SAHI! 🙌