[OTHER - Clarification] Order of operations in camera pipeline

[julianstirling]

Just a clarification.

"Contrast" is the gamma correction in this image:

This means that the CCM is applied to the non-linear data once the gamma correction is complete?

Why I am asking

In case it helps I'll let you know why I am asking

We are trying to do spacially variant colour unmixing to remove colour desaturation that comes from the very strong lens shading. To do this we basically create a tensor which is a CCM for each pixel.

I am trying to linearise the calibration images and the images to be corrected. It is my understanding that I need to invert the CCM before I degamma? Is CCM coming after gamma hard coded into the pipeline or is there a way to adjust it?

[davidplowman]

Hi, good question. Those blocks actually represent the control algorithms running in software, not the order in which things happen in hardware, which I appreciate could be a bit confusing. The CCM algorithm (the algorithm that chooses the CCM) uses the results provided by the AWB and Lux algorithms, and outputs a matrix.

The diagram you want is on page 67 of this document. The CCM is the first thing after Demosaic, followed by Sharpen and Gamma before the output pipes (cropping, Colour Space Conversion, resizing and output).

I don't know if it helps, but you could easily edit the camera tuning file to not apply a CCM, or not apply any gamma. On a Pi 5 you can also ask for 16-bit output samples (i.e. 48 bits per pixel). Would that make things any easier?

(It might help to say what kind of Pi and what sensor/camera you're using.)

[julianstirling]

Thank you! This is very helpful.

For some clarification, this is for the OpenFlexure Microscope. We are using a Pi 4B, with a PiCamera v2. We would like to move to the Pi5 eventually but we have a fairly complex stack of dependencies so this upgrade isn't likely to happen in the near future. We are looking at moving to the HQ camera, which will make this unmixing issue largely go away as we don't have the lenslet array.

In the microscope we the light approaching the sensor, meaning we need a strong adjustment of the lens shading table, but we still get desaturation so we need spatially varying CCMs to fully unmix colour.

I have thought about not applying CCM or gamma; or about getting the Raw data however we are trying to balance 3 things when we set up the microscope to automatically scan a sample and stitch the images:

1. Getting a live preview that looks pretty good to the microscope user (CCM, Degamma, and lens shading)
2. Capturing 100s to thousands of images as quickly and reliably as possible during sample scanning
3. Having everything fully corrected for the final stitched image

I think we need to have gamma and CCM on for the live preview to look good unless we added custom WebGL shaders to do it in GPU which seems like a lot of work.

We have also found that using the PIL image capture is the most reliably way to capture lots of images; so some very quick analysis for sharpness; and save them to disk as quickly as possible, ready to move on.

We then run a separate background process which does image correlation and stitching; this is where we want to integrate the colour correction.

Examples

Individual images on the microscope end up looking like this:

We can apply a spatial colour correction to get it to look much better (but not perfect):

Once we start stitching images together we get grids visible from the desaturation:

The grid can be minimised using the above colour correction:

However some desaturation remains. I did attempt more aggressive corrections but it turned white images green which I think relates to some non-linearities. But also I think relates to the fact that our custom (fixed) whitebalance is a bit wonky so the images are already a bit green; we just exaggerate it when we try to colour correct.

[davidplowman]

Fabulous pictures! Unfortunately it's difficult to think of anything that might help you.

Do you have a separate preview phase, and do a mode switch for capture? Or are you capturing continuously whilst showing the preview at the same time? In the former case one could think about doing the preview with CCM/gamma, and the capture without - at least you wouldn't have to undo them before applying your own processing.

A really crazy idea might be to turn off CCM and gamma, and use the CSC block on just one output pipe to apply the CCM. Then you'd get one image without CCM/gamma (for your own extra processing), and another that only requires gamma to "look nice". Applying gamma is just a look-up table, so even the boring old CPU should do that very quickly. But this would require a certain amount of custom C++ hackery to get a CCM into the CSC block, so it's not straightforward.

What kind of image format are you saving before you stitch them all together? Just curious, because the fastest way to do this these days is, I believe, to request YUV420 output and so save those directly as JEPGs (capture_file or whatever). If you can keep them in memory rather than writing them to disk, that's normally a big win too (though I don't know if any of this applies in your case).

It's also a bit unlucky that when it comes to colour shading, the imx219 is the worst of all our sensors by a country mile. Literally anything else would be better!!

[julianstirling]

The customs CSC block is an interesting idea. Something to mull over for the long term perhaps.

We don't do a mode switch as it is too slow. So we have the lores streaming a preview to the user while we capture images in 8MP for the scan. We use RGB rather YUV420 as we want rhe full depth of colour data as it is important to have as much colour info as we can.

We actually box downsample the images to 2MP before saving. We do this because the optical resolution of our system is ~2MP, but capturing in 2MP each BAYER channel is 0.5MP, there is a significant quality difference between 8MP capture downsampled to 2MP compared to 2MP capture, and not that much advantage to keeping all 8MP.

We do save them to disk before doing the stitching process, as we need to take about 8 to 10 images at each site (normally only saving 1) and some scans are over 1000 images. Using some fun tricks caching the cross correlations used for stitching, and using PyVIPS to stream images into a file we can actually create stitched images on the Pi that would be larger than the RAM if they were opened directly. We then use DZI for local viewing.

If we stay away from Red samples we have very few colour artifacts. For example this deep zoom image of a cervical smear has almost no artifacts and you can zoom right into the cellular detail. I am just trying to get the same level (or similar) continuity with H&E stained samples.

It is perhaps worth mentioning as a total aside to this converation that OpenFlexure Microscope development is now supported by a UK charity and we have both healthcare and education pilots happening in Africa, Latin America, and Ukraine all using the PiCamera driven microscope; So if the Raspberry Pi Foundation wanted to get involved or discuss the projects that would be really cool.

[julianstirling]

Unfortunately it's difficult to think of anything that might help you.

Also I feel it is worth mentioning that our progress to date has been hugely helped by things you have told us over multiple issue threads; including this one! Thank you!

[davidplowman]

It is perhaps worth mentioning as a total aside to this converation that OpenFlexure Microscope development is now supported by a UK charity and we have both healthcare and education pilots happening in Africa, Latin America, and Ukraine all using the PiCamera driven microscope; So if the Raspberry Pi Foundation wanted to get involved or discuss the projects that would be really cool.

Of course, we're not the Foundation, but if you were interested in doing something like a guest blog post that might help to publicise what you do and generally raise the profile. I think the comms/website folks are often on the lookout for interesting articles!

Also I feel it is worth mentioning that our progress to date has been hugely helped by things you have told us over multiple issue threads; including this one! Thank you!

Keep asking away, then! 😄

[mmoollllee]

Unfortunately it's difficult to think of anything that might help you.

Also I feel it is worth mentioning that our progress to date has been hugely helped by things you have told us over multiple issue threads; including this one! Thank you!

totally agree! 🥰