Implement capture sharpening inside demosaic module

[jenshannoschwalm]

This has been discussed many times here and on pixls.us so here we go. As we have D&S giving good results i was somewhat reluctant to work on this and do a pr but as CS works so good on RT and after a lot of testing (and using this on my personal images for a while with 3 auto applied presets for my fixed-lens camera - one for high-iso with CS turned off and two presets for wide open and "normal") i am pretty confident it's worth it and would be a significant improvement.
(It took quite a while to spot a very bad bug in GPU code leading to system instability and crashes so unfortunately not for 5.2)

Please test in depth and let's decide if this is for master. It's a lot of processing - so not a superfast module.

---

Capture sharpening has been implemented to work inside the demosaic module so it's raw only.
Credits to: Ingo Weyrich ([email protected]), he implemented the original algorithm for rawtherapee, this implementation is based on his work, especially the convolution kernels.
CPU and OpenCL code paths are both available.
Demosaic module gets more parameters so there is a version bump.

The "mini manual" (regularly updated to latest commit)

Capture sharpening (CS) tries to recover details lost due to in-camera blurring, which can be caused by diffraction, the anti-aliasing filter or other sources of gaussian-type blur. Prerequisites are

• good white balance parameters (same requirement as for highlights reconstruction or demosaic)
• no chromatic aberration, you might want to add the "raw chromatic aberration" module
• low noise as noise will be amplified by CS

Controls

1. capture sharpen
   switches CS on if above zero and defines the strength of overall effect.
   CS works in an iterative process, this defines the number of iterations, mostly a setting of 10 will be enough.
2. radius
   defines the basic convolution gaussian sigma.
   This should not be set by "creative means" but to the blurring radius of the optical system and sensor, too large values will lead to artifacts like halos.
   Calculating a correct radius is provided internally. This will be done either if you
   a) click on the button besides the slider
   b) activate capture sharpen the first time after resetting to demosaic defaults or developing old edits.
3. contrast threshold
   As sensor noise will be amplified by CS we take some care about this by a per pixel variance analysis and use a logistic function with this threshold to avoid CS in noisy areas. The default is good for low iso images.
   Note the visualizing button at the right.
4. corner boost
   Increase the convolution radius in image corners outside the sharp centre.
5. sharp centre
   Define the "sharp" centre part of the image
   Note the visualizing button at the right.

[HansBull]

A lovely enhancement, and reasonably fast even on my aging i7-2670QM!

[da-phil]

Nice effort, I like it 👍
However for some images I noticed a color shift the more iterations I choose for CS. Can you also reproduce it? Try very colorful images and maybe bump up the saturation a little bit in order to make the difference more visible.
In my case I also overlayed the non-CS with the CS image and used the layer mode "difference" to also see the color cast quantitatively.

[jenshannoschwalm]

However for some images I noticed a color shift the more iterations I choose for CS. Can you also reproduce it?

I couldn't yet, could you somehow share such raw files? (And possibly the xmp too? It might be related to white balance coeffs. A pre-requisite would be a correct WB)

As CS is an iterative process any error would be amplified by more iterations.

In my case I also overlayed ...

I don't understand your "reproducer procedure" here.

[TurboGit]

Just did a quick test, for a consistent UI I would propose to make the sliders non sensitive when sharpen is 0 but not to hide them.

[jenshannoschwalm]

I can do so of course. EDIT latest commit changes this but for my taste it's looking worse. We also have the other optinoal sliders in this module only visible if "relevant"

[da-phil]

In my case I also overlayed ...

I don't understand your "reproducer procedure" here.

What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".
For some images I also noticed that the image with CS turned on appear slightly brighter.

I'll try to find images which I can share for you to reproduce.

[jenshannoschwalm]

What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".

You are aware of the --dump-pipe module option?

[jenshannoschwalm]

Squashed and force-pushed an updated version

1. convolution kernels support sigma-steps of 0.01 instead of 0.02
2. slightly improved CPU convolution performance
3. the per-pixel variance anylysis is done in a circle with a diameter of 5 instead of 3 for improved stability in noisy regions and at sharp luminance transitions.

[da-phil]

What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".

You are aware of the --dump-pipe module option?

Nope, never used it, what is it good for?

[jenshannoschwalm]

What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".

You are aware of the --dump-pipe module option?

Nope, never used it, what is it good for?

It writes pfm files to tmp directory while processing the specified module in the pipe, it's meant for debugging exactly such issues :-)

darktable --dump-pipe demosaic would be your friend (use HQ processing mode)

[jenshannoschwalm]

Latest version includes a variance lut for performance plus some readability maintenance. (Some NaN issues spotted while testing this are in a bugfix PR for 5.2)

[jenshannoschwalm]

Latest version got some subtle perf gain, more important is improved stability for high-noise underexposed images. Right now i think it's good as is and i would appreciate testing. Please bear in mind, i have squashed and force-pushed the update to keep number of commits low.

I am still not convinced about the UI.

1. Is it really better to make the unused sliders insensitive instead of just hiding?
2. Any other idea?

[wpferguson]

@jenshannoschwalm do you have suggested settings or at least a good starting point?

[jenshannoschwalm]

Normally with a strength of 10 would be fine. Evaluate at 100% zoom.

[jenshannoschwalm]

Latest version - as usual load the whole pr for testing -

1. Does automatic calculation of radius if capture sharpen has been enabled
2. Some improved tooltips
3. The maximum radius has been increased to 2.0 as this is still acceptable with the convolution kernels.

[jenshannoschwalm]

@TurboGit @da-phil @weltyj friends i think i have sorted it out in an acceptable (for us all) way.

1. All capture related ui elements have been moved into a collapsible section as used all over the code for such circumstances. Internals parameters are all gtk_widget_set_sensitive() if applicable so no fiddling around to-be-visible-or-not.
2. Slightly increase resolution of iteration settings - for those of us who think that maths go beyond capabilities of our retina :-)
3. Some improvements in the 'mini manual' and tooltips

So ready for reviewing pascal.

[weltyj]

LOVE the UI now!!

Iterations maxing out at 25, vs the previous (20**1.3) ~= 50, does make a visibile difference. I just fooled around and this

iterations = d->cs_iter + (int)(0.000065f*powf((float)d_cs->iter, 4.0f))

will give good precision for low iterations (<= 10), and then start ramping up to get iterations of 50 if d->cs_iter goes to 25

;

[jenshannoschwalm]

will give good precision for low iterations

perfect and done

[weltyj]

@jenshannoschwalm I'm not sure what happened but capture.c looks like it didn't change at all after your last update to your github branch...

[jenshannoschwalm]

I'm not sure what happened

You are looking at the wrong code, it's in demosaic commit_params() including a comment :-)

[jenshannoschwalm]

Just a fix for sharpening at image borders

[da-phil]

@TurboGit @da-phil @weltyj friends i think i have sorted it out in an acceptable (for us all) way.

1. **All** capture related ui elements have been moved into a collapsible section as used all over the code for such circumstances. Internals parameters are all `gtk_widget_set_sensitive()` if applicable so no fiddling around to-be-visible-or-not.

2. Slightly increase resolution of iteration settings - for those of us who think that maths go beyond capabilities of our retina  :-)

3. Some improvements in the 'mini manual' and tooltips

So ready for reviewing pascal.

Thank you so much, this looks perfect now 🙏

[TurboGit]

This UI is far better to me and integrate well with current dt UI. Well done, no more comment on my side, works great. Thanks @jenshannoschwalm for this great piece of work.

[weltyj]

Indeed @jenshannoschwalm , thanks for all the hard work and seeing it through. Time to pull the cork on the champagne!

[jenshannoschwalm]

Thanks, there is soon coming more work for you :-)

[weltyj]

Thanks, there is soon coming more work for you :-)

Sounds like fun :-)

[mikae1]

Happy to see a long awaited feature in the works @jenshannoschwalm!

Calculating a correct radius is provided internally. This will be done either if you
a) click on the button besides the slider
b) activate capture sharpen the first time after resetting to demosaic defaults or developing old edits.

I interpret this as settings will scale internally with the resolution of the image? As an example, will the radius be “internally” higher for a 61MP image than a 24MP image? If not, users will have to calculate to dial in and calculate to get the same amount of capture sharpening on images from different MP sensors.

[jenshannoschwalm]

It's in absolute pixels.

[jenshannoschwalm]

Yes, different sensors, different lens, different settings :-)

[europlatus]

In my testing of this, I have been missing an option to turn it off and on easily. Obviously the Demosaic module can't be turned off, so you just have to slide the Sharpen iterations to zero and then back up again. It's a bit cumbersome. I imagine this was done to reduce clutter, but would it make sense to have an "eye" icon like we have for masks so we can quickly turn the effect off and on again?

[jenshannoschwalm]

but would it make sense to have an "eye" icon like we have for masks so we can quickly turn the effect off and on again?

Feel free to implement this :-)

[weltyj]

but would it make sense to have an "eye" icon like we have for masks so we can quickly turn the effect off and on again

@europlatus A checkbox "[ ] enable capture sharpening" would be more appropriate than an eye icon.

[jenshannoschwalm]

Yes, a checkbox might be used/added but i'd prefer to keep the UI as small as possible. If someone wants to propose an idea, feel free to open a PR so we can test/discuss.

[europlatus]

@europlatus A checkbox "[ ] enable capture sharpening" would be more appropriate than an eye icon.

I thought of that too, but the sharpening slider currently functions as an On/Off button as well, so a checkbox potentially adds unnecessary clicks (one click for the dropdown, one for the checkbox, and one for the slider). Another idea would be for the dropdown arrow to auto enable/disable capture sharpening, so you could click that to turn the effect on and off. But this would mean having the module always expanded if you want capture sharpening on.
I thought the eye icon was the most consistent with what we use for parametric/drawn masks.

I'll open a feature request to continue this discussion...