Error with keras models containing a Concat layer

[Theodlz]

I've been trying to convert this model to onnx using the code from examples/keras-tract-tf2 (with a few edits to match the input of my model).

While converting to onnx doesn't yield any errors, and nor does loading it with tract_model = tract.onnx().model_for_path("acai.onnx"), tract_model.into_optimized() (both in rust and python) yields the following error:

Error: Failed analyse for node #46 "model/concatenate/concat" InferenceConcat

Caused by:
    0: Infering facts
    1: Applying rule inputs[0].shape[0] == inputs[1].shape[0]
    2: Impossible to unify Sym(unk__64) with Sym(unk__65).

This model is multimodal, it has a CCN branch and a very basic MLP at the other branch. After flattening the CNN, the results from both branches of the model are concatenated and following by a couple of simple layers leading to a single neuron as the output (to get a score from 0 to 1).

I'm not entirely sure why concatenating 2 1D tensors is failing here. This model, in tensorflow, is valid and runs just fine. So either the conversion to onnx is incorrect, or tract itself is having an issue loading it and optimizing it?

[Theodlz]

I should add that:

import onnx
onnx_model = onnx.load("acai.onnx")  # Load the ONNX model
onnx.checker.check_model(onnx_model)  # Check that the model is well formed
print("ONNX model is valid.")

runs fine, indicating that onnx considers the model as valid.

[Theodlz]

Ok so looking at the onnx graph, I see that the inputs of the model are:

input {
  name: "features"
  type {
    tensor_type {
      elem_type: 1
      shape {
        dim {
          dim_param: "unk__64"
        }
        dim {
          dim_value: 25
        }
      }
    }
  }
}
input {
  name: "triplets"
  type {
    tensor_type {
      elem_type: 1
      shape {
        dim {
          dim_param: "unk__65"
        }
        dim {
          dim_value: 63
        }
        dim {
          dim_value: 63
        }
        dim {
          dim_value: 3
        }
      }
    }
  }
}

Could it be that tract doesn't deal well with multimodal inputs, and is confused as to why the 2 inputs found in onnx have different dims (not sure why they should?).

[Theodlz]

Also running it with the onnxruntime works just fine, which makes me think that this is indeed an issue with tract itself:

import numpy as np
import onnxruntime as ort
# Create an ONNX Runtime session
ort_session = ort.InferenceSession("acai.onnx")
# Prepare input data for the model
input_images = np.random.random((1, 63, 63, 3)).astype(np.float32)
input_metadata = np.random.random((1, 25)).astype(np.float32)

# run the model
ort_inputs = {
    ort_session.get_inputs()[0].name: input_metadata, # The first input is the metadata
    ort_session.get_inputs()[1].name: input_images # The second input is the images
}
ort_output = ort_session.run(
    output_names=[ort_session.get_outputs()[0].name], 
    input_feed=ort_inputs
)
print(f"ONNX Runtime Output: {ort_output}")

[kali]

Hello, can you give me access to the model somehow ?

[Theodlz]

Hey @kali! Sure thing, I linked the h5 above but let me send the onnx version. Here it is:

acai_h.onnx.txt

PS: Sorry I had to add another extension at the end for it to upload, you might have to rename it

[Theodlz]

BTW I should mention that since I opened that issue, I got it to work with pykeio's ort crate. Validating that the issue doesn't come from the model being invalid or something like that.

[kali]

Alright, so it's a matter of tract being stricter than most training frameworks. We expect to be able to make sense of the dimensions, including when they are "dynamic". tract uses "symbols" and expressions with them to describe the missing bits in the various tensor dimensions, and they must be valid. Other framework are more lenient.

Here we have a model that uses two different "dim_param" that should be equal for the model to make sense (they're some form of batch dimension). There is actually a third one in the output that will also conflict. To make tract happy with your model, you can override the input shapes, and ignore the output shape (as tract will compute the output shape) like below.

Now translating this to rust code looks like this:

     let model = tract_onnx::onnx().model_for_path("acai_h.onnx").unwrap();
    let b = model.sym("b");
    let model = model
        .with_input_fact(0, f32::fact(&[b.to_dim(), 25.to_dim()]).into())
        .unwrap()
        .with_input_fact(
            1,
            f32::fact(&[b.to_dim(), 63.to_dim(), 63.to_dim(), 3.to_dim()]).into(),
        )
        .unwrap()
        .with_output_fact(0, Default::default())
        .unwrap()
        .into_optimized()
        .unwrap();

Note that if you don't need batching, you could probably get Keras to not use dynamic dimensions at all and get a "clean" model out of the box.

[kali]

Moving this to a discussion as it is a recurring question.