Support for multiple objects in single image

[dereklukacs]

FasterRCNN supports training with multiple objects per image (according to following)

From: https://pytorch.org/docs/stable/torchvision/models.html#object-detection-instance-segmentation-and-person-keypoint-detection

During training, the model expects both the input tensors, as well as a targets (list of dictionary), containing:

• boxes (FloatTensor[N, 4]): the ground-truth boxes in [x1, y1, x2, y2] format, with values of x between 0 and W and values of y between 0 and H

• labels (Int64Tensor[N]): the class label for each ground-truth box

Is this supported by detecto? I have only been able to pass in {"boxes": Tensor(1, 4), "labels": str}

Would you be open to a pull request fixing this? It looks like the only place where there is an issue is in core.Model

class Model:
...
    # Converts all string labels in a list of target dicts to
    # their corresponding int mappings
    def _convert_to_int_labels(self, targets):
        for target in targets:
            # Convert string labels to integer mapping
            if _is_iterable(target["labels"]):
                target["labels"] = torch.tensor(self._int_mapping[label] for label in target["labels"])
            else:
                target["labels"] = torch.tensor(self._int_mapping[target["labels"]]).view(1)

This would now accept target["labels"] = "one_object" or target["labels"] = ["obj1_class", "obj2_class"] and would still return a tensor of length equal to the number of objects.

[alankbi]

If you're using Detecto's Dataset to read in your images/XML files, then having multiple labels per image should automatically be taken care of (it just splits them into separate items). Are you using some custom code somewhere in your program? If so, then that might be the cause.

However, that change seems totally reasonable so I'd be happy to accept a pull request for that!

[AndreaParmiggiani]

I run into the same issue using core.Dataset, it worked for one class but not for multiple ones, I checked the xml file and it looks fine with multiple and the transformation in a csv also looks good with each object having its own row, but for some reason when I fit the model it gives me nan loss from the first epoch.

Thank you, and beside this problem I'm really grateful for this library!

[alankbi]

Could you share the code you wrote as well as what those XML/CSV files look like?

Off the top of my head though, your issue with nan loss seems similar to #36. Could you take a look at that thread and see if that helps at all?

[AndreaParmiggiani]

`dataset_o = core.Dataset(r'/content/drive/My Drive/ReconAI/split/train/other_labels.csv',
r'/content/drive/My Drive/ReconAI/split/train/other_frames')

val_dataset_o = core.Dataset(r'/content/drive/My Drive/ReconAI/split/val/other_labels.csv',
r'/content/drive/My Drive/ReconAI/split/val/other_frames')

model_o = core.Model(['varioussignals','balise','trafficsignals'])

model_o.fit(dataset=dataset_o, val_dataset=val_dataset_o, epochs=5, learning_rate=0.0001, lr_step_size=1)`

I tried to slow the learning rate but it wasn't the source of the problem.
I also tried to convert the xml to csv before feeding it into the model and this is what it looks like in the DataFrame version:

[alankbi]

When you train the model, do you get similar results to the following?

Epoch 1 of 10
Loss: nan
Epoch 2 of 10
Loss: nan
...

Similarly, what happens if you try to predict on an image after training the model?

[alankbi]

The original issue mentioned has just been addressed in the release of v1.2.0 - special thanks to @TasinIshmam :)