Add support for DICOM dermoscopy objects

[fedorov]

This follows up on the discussion with @Punzo on slack.

The samples can be accessed in https://challenge2020.isic-archive.com/.

If it turns out the effort required is significant, we will need to reconsider.

FYI: @dclunie

[Punzo]

@fedorov it turns out that OHIFv2 can load dermoscopy data without any issue.

I loaded locally the DICOM corrected Test Data (6,7 GB) from https://challenge2020.isic-archive.com/ .

There is an issue: in the metadata the PhotometricInterpretation is set to YBR_FULL_422, however when downloading the image the array length is = rows * columns * 3 (instead of 2, i.e., each pixel should have 2 values in YBR_FULL_422, which can be converted to RGB/RGBA values, see link).

if I force the PhotometricInterpretation to RGB, the image is loaded properly:

Therefore, I think that the images are actually encoded in RGB. I could put a check on the downloaded image array and set a special case (i.e., if the data array length is = rows * columns * 3 and PhotometricInterpretation = YBR_FULL_422, ignore PhotometricInterpretation and convert anyway as RGB images) in https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/convertColorSpace.js, but I would prefer to avoid putting hacks in place for defective datasets.

3DSlicer loads the images without problems, but I think it ignores the PhotometricInterpretation, and it just uses the SamplePerPixel attribute (which for YBR_FULL_422, SamplePerPixel is 3 by the standard, even if before the conversion to RGB/RGBA the array values for each pixel are 2).

Anyway, for datasets with non-defective metadata, DICOM dermoscopy objecjs will be loaded with no problems with OHIFv2 with latests cornerstoneWADOImageLoader (see PR: #2944)

[fedorov]

I would prefer to avoid putting hacks in place for defective datasets

I agree.

I believe @dclunie is quite familiar with this dataset - would be good to have his guidance here.

[dclunie]

@Punzo, AFAIK, all these images are JPEG baseline compressed (Transfer Syntax 1.2.840.10008.1.2.4.50).

The DICOM Photometric Interpretation describes what is encoded in the compressed JPEG bitstream, not the RGB that might (or might not) come out of a JPEG decoder that converts the YCbCr chrominance downsampled compressed data into uncompressed RGB. See:

https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_8.2.html#para_4bcb841e-c6bf-4e26-82a5-3fad3c942da0

and the notes after the table.

So, the logic in OHIF (or the image loader it is using) needs to account for whatever the JPEG decoder it is using actually does, and not be literally applied to the decompressed pixel data (i.e., it should know there are 3 pixels per sample in the decompressed data not 2 if chrominance downsampling has already been reversed), and that the color space has already been transformed by the decoder from YCbCr to RGB (both are usually the case for baseline JPEG decoders such as those that are libpeg-based).

Or to put it another way, each DECOMPRESSED pixel should have 3 values not 2 as you are assuming, regardless of the PhotometricInterpretation of the COMPRESSED pixel data being YBR_FULL_422. The only time that the PixelData length should be considered in this respect is when the encoding is "Native" (uncompressed), not "Encapsulated" (compressed), i.e., Note 2:

https://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.6.3.html#para_442b66ec-6d91-42ba-bfeb-738c94d2d6d3

does not apply to these JPEG compressed images.

Your decoder should not be guessing that Photometric Interpretation might be "wrong", rather it should know exactly what the compressed pixel data decoder has done, and behave accordingly.

Be aware that not all JPEG compressed pixel data is YCbCr - sometimes it really was compressed as RGB components (e.g., Aperio WSI).

Also, for JPEG compressed images, though YBR_FULL_422 is always used when a YCbCr color transformation has been applied before compression, the actual extent of chrominance downsampling varies in practice, and is not always 2x2:

https://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.6.3.html#para_054ed567-c63e-4450-a128-0c8af7adc764

Further, some of the YBR family of Photometric Interpretations are used for non-JPEG compressed pixel data (e.g., some RLE or uncompressed ultrasound images, in which case a color space transformation +/- a chrominance channel upsampling may be required). That should not apply to these ISIC dermatology challenge images, though changes to your logic might affect handling of other images, which is why I mention it.

@chafey might want to comment on what Cornerstone does/should do in this respect for compressed images.

[Punzo]

Found the issue. The wado parser in cornerstoneWADOImageLoader sends a XMLHttpRequest as type=application/octet-stream; transfer-syntax=*
Then it uses the mime information from the server response to guess the transfer syntax of the donwloaded frame.

https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/wadors/loadImage.js#L79
https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/wadors/loadImage.js#L10

However in the case of requests as type=application/octet-stream; transfer-syntax=* the right mime will not be known
(we had similar issues with SLIM: https://github.com/herrmannlab/dicom-microscopy-viewer/blob/master/src/pyramid.js#L426-L433). In fact in the case of this dermoscopy datasets, we would not get an a "image/jpeg" mime response (i.e. the frame is decompressed by the server and returned already as uncompressed color converted rgb image).

Therefore the cornerstoneWADOImageLoader will load internally the frame as if the Transfer Syntax is 1.2.840.10008.1.2 instead of 1.2.840.10008.1.2.4.50. This finallly generates issues, because color transformations are applied client side since the guessed transfer syntax is wrong (not all decoders apply automatically the color transformations, so for a set of transfer syntaxes corresponding to such decoders, the color treansformations are appllied as in https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/createImage.js#L238).

We could improve the cornerstoneWADOImageLoader either:

1. using the dicom-webclient instead to use a simple XMLHttpRequest (e.g., in Slim we use it to pass an array of transfer syntaxes https://github.com/herrmannlab/dicom-microscopy-viewer/blob/83739f1ba11840df7f4487cdf8015b7450ee419d/src/pyramid.js#L436-L491 and the first compatible one will used). In this way (having the "image/jpeg" mime of higher priority than application/octet-stream), we can get the original compressed jpeg file for the frame and we will decompress/apply color transformations client side.

2. After fetching the arrayBuffer, we could guess the right frame mime from the array buffer rather then from the mime information from the server response. E.g. similar to the approach that we use in SLIM: https://github.com/herrmannlab/dicom-microscopy-viewer/blob/83739f1ba11840df7f4487cdf8015b7450ee419d/src/webWorker/decodeAndTransformTask.js#L99-L150

@sedghi @swederik, it would be nice to have your feedback as well. Thanks!

[Punzo]

@fedorov moving this back in Progress