[API Proposal]: System.Numerics.Tensors serialization

[vpenades]

Background and motivation

Given that the new family of Tensor<T>, specially when using strides for sparse tensors is not trivially serializable, I would like to ask about guidelines about how to generalize a roundtrip serialization for Tensor<T> family.

API Proposal

To begin with, it's not trivial to serialize Tensor<T> because T although most of the time it will be a known, serializable type.

Then it comes the formatting, I am not aware if there's a standarization for that, so my API proposal would be:

namespace System.Numerics.Tensors

public class SerializableTensor<T>
{
   public SerializableTensor(Tensor<T> sourceTensor) { ... }

   public nint[] Lengths { get; }

   public nint[] Strides {get; }
  
   public List<T[]> InternalData { get; }  // not sure about how to deal with this one, but it needs to cover +2Gb and sparse data.

  public Tensor<T> ToTensor() { ... }

  // easter eggs
  public void WriteBinary(Stream s) { ... }
  public void ReadBinary(Stream s) { ... }
}

Or maybe, instead of an internalData, it could be possible to create some kind of "visitor" API for reading and writing chunks of the internal tensor data, which would help avoid creating a copy of the data.

API Usage

Tensor<float> someTensor = ...

var sTensor = new SerializableTensor<float>(someTensor);

using Stream s = File.Create("tensordata");

sTensor.WriteBinary(s);

Alternative Designs

Maybe putting a class such as SerializableTensor in System.Numerics.Tensors is too much, so I would be happy to have these helpers, maybe in a Microsoft.Extensions.Tensors lateral library?

But I would be glad to just have a code snippet, because right now, I can't see a way through, given some of the mechanics of the tensors are not directly accessible.

Risks

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-numerics-tensors
See info in area-owners.md if you want to be subscribed.

[teo-tsirpanis]

You can use the CopyTo methods to extract data from a tensor to an arbitrary destination. The problem with round-trippable serialization of tensors is that there are metadata about a tensor (e.g. its shape) with many possible ways of encoding them in binary, and System.Numeric.Tensors is a too low-level and unopinionated library to pick an encoding format.

You might be interested in something like Apache Arrow tensors.

[vpenades]

I suspect the CopyTo method expands the tensor in case it is a sparse tensor, thus, not suitable for serialization.

Notice that I am not asking about a direct serialization method, I'm asking about an intermediate object (SerializableTensor) that is more suited to be used for serialization.

Also I'm not asking these changes to be in the core libraries, but in extension libraries, or at least to exist as code snippets.

[tannergooding]

Tensor is an "out-of-band" (does not ship with the core libraries and requires an explicit package reference) type. It already provides all the relevant APIs for a user to successfully create their own custom converter appropriate for their serialization framework of choice if the default functionality provided by that framework (such as via the types implementing IEnumerable<T> and friends) is not sufficient. -- That is, the proposed surface area is just duplicating what Tensor<T> already supports and exposes, just in a slightly different way.

https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/converters-how-to would be a getting started guide appropriate for System.Text.Json; other frameworks provide their own getting started guides.

The core libraries don't tend to expose dedicated serialization types, rather the default handling for primitives and collections is typically sufficient for most domains and everything else is left up to the user writing a custom converter for their domain.

CC. @eiriktsarpalis, @GrabYourPitchforks

---

It's also worth noting that System.Numerics.Tensors does not currently support "sparse tensors", at least not in the traditional sense of the term as used by other tensor frameworks. It rather supports "broadcast tensors" which avoids repeating the same dense data over and over, allowing tensors of different but compatible sizes to be used together and "slices", which have consistent padding between sequences of dense data (typically between rows).

[tannergooding]

Users should also, in general, be mindful of serializing potentially "large data". This can trivially run into bandwidth, memory, or IO (disk, network or device) limitations. If done naively, it can lead to stalls, hangs, or potential DDoS concerns and can even just generally be bad for performance due to lack of appropriate hardware utilization.

---

Computers are designed and optimized around working with small sets of data at a time, where small is "CPU scale". That is, a modern CPU typically works with cache line sized units of memory (often 64 bytes) and then has several layers of cache, typically no more than around 2MB per core on the very high end (and more typically around 256KB per core) before you start hurting utilization.

The default approach of processing data tends to be:

1. load all data
2. process all data
3. store all data

And while this is incredibly simple and easy to do, it becomes a severe bottleneck when you start approaching or exceeding the limits mentioned. You instead need to start switching to a "chunking" approach where you do something more like:

1. Load some data, ideally an amount that doesn't exceed more than 25-50% of the L3 cache, this is the "current" chunk
2. Start processing the "current" chunk
3. While that is being processed, asynchronously start loading the "next" chunk of data (same size)
4. When the processing of the "current" chunk is done, start storing the data (can be synchronous or asynchronous, depending on scenario)
5. Await the loading of the "next" chunk to finish, it is now the "current" chunk, and repeat from step 2 until complete

This setup requires a few more steps and breaking it apart into a few bits, but it is much more efficient and doesn't bottleneck on having enough memory to load everything in at once, doesn't bottleneck on loading all of that data from storage at once, allows parallelization of resources and better hardware utilization, allows setting up your own caching systems if you need more random access (such as would be required for sorting large data), doesn't completely pessimize the hardware by trashing and invalidating the cache, etc.

It is the type of thing that can be a several hundred times performance improvement due to appropriately using the hardware available since working with "big data" is so fundamentally different to how hardware is tuned. And this is notably even more true on powerful machines that have lots of memory and expensive hardware with a lot of bandwidth, where the extra capabilities are there specifically because you have a high number of cores all able to work in parallel, each doing part of this work and fully saturating the multiplex capabilities available.