Skip to content

Allow partial processing of read datasets #63

@johnlees

Description

@johnlees

For large read datasets, their size (especially plus a large countmin table) may exceed device memory. However they can happily be loaded (asynchronously) in blocks, as each read is independent.

This change will require:

  1. Remove the read interleaving, and loading into shared. A sync block load can be used to load all the reads for a warp into shared directly and more simply.
  2. Pin the host memory in the DeviceReads class
  3. Add a loadBlock method to DeviceReads which loads up to a specified size using memcpy async on a non-default stream
  4. Launch the kernel on a second non-default stream.
  5. Iterate through loadBlock, waiting on kernel completion.

The block size will be (device memory - size of countmin table) / 2 - epsilon

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions