Large deskewed image throws error on napari-lattice (GPU memory issue)

[emmageddon-wehi]

Hi Team,

I tried working with a 1 GB dataset and encountered some memory(?) issues.

Case 1: Doesn't Process

• I'm running Napari-lattice through BASH with version 2.3.0.
• The environment is the vwimaris-03 VM, with a GTX 1080 GPU and 464 GB of RAM.
• The file was opened using Napari’s built-in functions.
• File size: 910 MB (Slices: 401, Width: 512, Height: 2304)
• dx: 0.101, dy: 0.101, dz: 1.2
• Angle: 35, Device: NVIDIA RTX A4000
• Skew direction: X
• Initialization was successful, but no deconvolution.
  Here's the output from BASH from conda activation through to crashing: BASH_napari-lattice.txt

Case 2: Does Process

• When I reduced the file size by cropping the pixel height, the lattice plugin worked perfectly!
• Same setup: Napari-lattice 2.3.0 through BASH, vwimaris-03 VM, GTX 1080, and 464 GB RAM.
• File size: 470 MB (Slices: 401, Width: 512, Height: 1152)
• dx: 0.101, dy: 0.101, dz: 1.2
• Angle: 35, Device: NVIDIA RTX A4000
• Skew direction: X
• Initialization was successful, and no deconvolution.
• Deskewing was also successful.

Let me know if you need anything else (I'm more than happy to share the data) or if there's any troubleshooting we can try!

[multimeric]

Thanks for the error report. Can you please attach the results of pip freeze?

[emmageddon-wehi]

requirements.txt

[multimeric]

So this is using napari-lattice==0.2.7 which is a previous version. Can I ask which instructions you followed to install this?

[emmageddon-wehi]

Hi Mike,

This was an older version which I installed following the previous installation instructions on the Wiki in October 2024. With help from Pradeep, I used the newer lattice plugin on Jupiter 2, and found the same issues. Please find attached screenshots of case 1, case 2, and the traceback for case 2.

traceback.txt

[multimeric]

This seems to be a different error? I can't see OUT_OF_RESOURCES in the original log.

In this case I wonder if the image is simply too large for the GPU. I guess pyopencl is failing at the point where it tries to push the data to the GPU. I'm not sure if we can solve this other than by adding a more informative error message. When this happens I guess your options are to either reduce the image size as you have done above, or to switch to CPU processing (the Graphics Device menu item).

[pr4deepr]

It does look like a GPU error.
Although, the machine @emmageddon-wehi is using has a 24GB GPU, so I'm surprised it throws this error..
Will investigate this.

[pr4deepr]

So, I get the same error when running the deskew operation. But the error happens after I try to pull the deskewed image from the GPU.

import numpy as np
import pyclesperanto_prototype as cle

np.random.seed(0)
image = np.random.randint(0, 255, (401, 2304, 512), dtype=np.uint16)

angle_in_degrees=35
voxel_size_x=0.101
voxel_size_y=0.101
voxel_size_z=1.2


deskewed = cle.deskew_x(image,angle_in_degrees=35,
                        voxel_size_x=voxel_size_x,
                        voxel_size_y=voxel_size_y,
                        voxel_size_z=voxel_size_z,)
deskewed.shape

(294, 2304, 5184)

The deskew process works.

When I pull the image from the GPU,

deskewed_np = cle.pull_zyx(deskewed)

I get this error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 1
----> 1 deskewed_np = cle.pull_zyx(deskewed)

File ~\AppData\Roaming\Python\Python39\site-packages\pyclesperanto_prototype\_tier0\_pull.py:36, in pull_zyx(gpu_array)
     31 import warnings
     32 warnings.warn(
     33     "Deprecated: `pull_zyx()` is now deprecated as it does the same as `pull()`.",
     34     DeprecationWarning
     35 )
---> 36 return pull(gpu_array)

File ~\AppData\Roaming\Python\Python39\site-packages\pyclesperanto_prototype\_tier0\_pull.py:28, in pull(gpu_array)
      4 def pull(gpu_array):
      5     """Returns an image from GPU memory as numpy compatible array
      6 
      7     .. deprecated:: 0.6.0
   (...)
     26     .. [1] https://clij.github.io/clij2-docs/reference_pull
     27     """
---> 28     return Backend.get_instance().get().asarray(gpu_array)

File ~\AppData\Roaming\Python\Python39\site-packages\pyclesperanto_prototype\_tier0\_opencl_backend.py:18, in OpenCLBackend.asarray(self, image)
     17 def asarray(self, image):
---> 18     return np.asarray(image)

File ~\AppData\Roaming\Python\Python39\site-packages\pyclesperanto_prototype\_tier0\_pycl.py:111, in OCLArray.__array__(self, dtype)
    109 def __array__(self, dtype=None):
    110     if dtype is None:
--> 111         return self.get()
    112     else:
    113         return self.get().astype(dtype)

File c:\ProgramData\Miniconda3\envs\napari_lattice_ilastik\lib\site-packages\pyopencl\array.py:921, in Array.get(self, queue, ary, async_, **kwargs)
    915 if async_:
    916     warn("calling pyopencl.Array.get with 'async_=True' is deprecated. "
    917             "Please use pyopencl.Array.get_async for asynchronous "
    918             "device-to-host transfers",
    919             DeprecationWarning, stacklevel=2)
--> 921 ary, event1 = self._get(queue=queue, ary=ary, async_=async_, **kwargs)
    923 return ary

File c:\ProgramData\Miniconda3\envs\napari_lattice_ilastik\lib\site-packages\pyopencl\array.py:880, in Array._get(self, queue, ary, async_, **kwargs)
    874     raise ValueError("Cannot copy array to host. "
    875             "Array has no queue. Use "
    876             "'new_array = array.with_queue(queue)' "
    877             "to associate one.")
    879 if self.size:
--> 880     event1 = cl.enqueue_copy(queue, ary, self.base_data,
    881             src_offset=self.offset,
    882             wait_for=self.events, is_blocking=not async_)
    884     self.add_event(event1)
    885 else:

File c:\ProgramData\Miniconda3\envs\napari_lattice_ilastik\lib\site-packages\pyopencl\__init__.py:2013, in enqueue_copy(queue, dest, src, **kwargs)
   2006             warn("The 'device_offset' argument of enqueue_copy "
   2007                     "is deprecated. Use 'src_offset' instead. "
   2008                     "'dst_offset' will stop working in 2023.x.",
   2009                     DeprecationWarning, stacklevel=2)
   2011             kwargs["src_offset"] = device_offset
-> 2013         return _cl._enqueue_read_buffer(queue, src, dest, **kwargs)
   2015 elif src.type in _IMAGE_MEM_OBJ_TYPES:
   2016     origin = kwargs.pop("origin")

RuntimeError: clEnqueueReadBuffer failed: OUT_OF_RESOURCES

Surprised we get this error, as the deskewing is done..

Will follow up with the developer of pyclesperanto.

Cheers
Pradeep

[pr4deepr]

Definitely a GPU memory issue.
Will need to catch it by checking for maximum image size supported by each GPU.

clEsperanto/pyclesperanto_prototype#344 (comment)

Will need to include tiling support.

[pr4deepr]

Just to sum up, even though the input image is 1GB, the metadata dictates the size of the final deskewed image.
If the deskewed image is really large, the GPU can throw a memory error.

---

The deskewed image size is actually ~6.5 GB.

Using pyclesperanto we can view the information for each GPU

import pyclesperanto_prototype as cle
print(cle.cl_info())

The MAX_MEM_ALLOC_SIZE field indicates maximum image size supported by the GPU. If it goes over that, we get an OUT_OF_RESOURCES error.

The GPU on our VMs are NVIDIAa Quadro RTX 6000, 24GB of RAM and have:
MAX_MEM_ALLOC_SIZE:6039732224, so ~6GB

As the deskewed image is 6.5GB, we get an error!