Difficulty applying StyleShot to medical images (poor results & edge detection)

[wwwmj23]

Dear authors,

First of all, thank you very much for your excellent work and for making the source code publicly available. We truly appreciate your contribution.

We have been experimenting with StyleShot for medical image applications. Specifically, we tested the generation performance using styleshot_image_driven_demo.py and styleshot_controlnet_demo.py. However, the results were not as satisfactory as expected.

style:

content:

generate:

We also visualized the intermediate results (e.g., contour and lineart) and found that the edge extraction performance was not ideal either.

Lineart:

Contour:

Could you please share some suggestions on how to improve the performance in such scenarios?
Do you think retraining StyleShot on our medical dataset might help? Our dataset, however, does not contain any textual annotations — would it still be possible to train StyleShot under this condition?

Thank you again for your outstanding work and for taking the time to provide any insights!

[Jeoyal]

Hi @wwwmj23 , thank you for your interest in our work. Your style image is merely a binary grayscale image, which makes it difficult to identify the style. I recommend re-rendering the provided content image using a segmentation-based approach.

[wwwmj23]

Dear authors,

If we retrain the StyleShot model specifically on a medical image dataset—where the content is represented by extracted lineart features—do you think this approach would be more likely to achieve better performance in medical image style transfer tasks?

Thank you once again for your valuable time and expertise!

[Jeoyal]

I recommend training a simple LoRA model to check whether your dataset is suitable for the medical image style transfer task.

[wwwmj23]

Thank you very much for the suggestion! I have previously used a StyleID on my dataset, which achieved quite promising results. Given that, I was wondering whether fine-tuning your model on this dataset might also be suitable or potentially lead to good performance.

[Jeoyal]

StyleID is a training-free method, which maintains most content information during stylization. Training a simple LoRA to verify if the training-based methods work well on your dataset.