Add support for Apple's Depth-Pro

[geetu040]

What does this PR do?

Fixes #34020

This PR adds Apple's Depth Pro model to Hugging Face Transformers. Depth Pro is a foundation model for zero-shot metric monocular depth estimation. It leverages a multi-scale vision transformer optimized for dense predictions. It downsamples an image at several scales. At each scale, it is split into patches, which are processed by a ViT-based (Dinov2) patch encoder, with weights shared across scales. Patches are merged into feature maps, upsampled, and fused via a DPT decoder.

Relevant Links

• Research Paper: Depth Pro: Sharp Monocular Metric Depth in Less Than a Second
• Authors: Aleksei Bochkovskii, Amaël Delaunoy, and others
• Implementation: apple/ml-depth-pro
• Models Weights: apple/DepthPro

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@amyeroberts, @qubvel

[geetu040]

I have implemented the foundational components of the model and manually loaded the weights to ensure that the architecture aligns with the original design and produces consistent output.

Below is a concise overview of the class hierarchy. I would greatly appreciate your feedback or any suggestions for improvements:

DepthProForDepthEstimation
├── depth_pro: DepthProModel
│   ├── encoder: DepthProEncoder
│   │   ├── patch_encoder: DepthProViT
│   │   │   ├── embeddings: DepthProViTEmbeddings
│   │   │   └── encoder: DepthProViTEncoder
│   │   ├── image_encoder: DepthProViT
│   │   │   ├── embeddings: DepthProViTEmbeddings
│   │   │   └── encoder: DepthProViTEncoder
│   ├── decoder: DepthProDecoder
│   └── fov_model: DepthProFOVModel
│       ├── encoder: DepthProViT
│       │   ├── embeddings: DepthProViTEmbeddings
│       │   └── encoder: DepthProViTEncoder
└── head: DepthProDepthEstimationHead

I have a couple of questions:

1. The encoder: DepthProEncoder outputs features processed at various scales, including hidden states from the intermediate layers of ViTEncoder. Currently, I use BaseModelOutput, returning all features in the last_hidden_state argument. Should I create a dedicated ModelOutput class for DepthProEncoder? If so, it should reside in the same file as the DepthPro classes since it is specific to them.

2. For handling the FOV (Field of View) output, would it be appropriate to create a class named DepthEstimatorOutputWithFOV in transformers.modeling_outputs, or should it also remain within the DepthPro context?

[Rocketknight1]

cc @pcuenca as well!

[qubvel]

Hi @geetu040! Thanks for working on this model!

Regarding model outputs they should be written if you want to add a new argument or write better docs. In case of intermediate outputs you can store them in BaseModelOutput.hidden_states, for example mllama set default output_hidden_states=True and then select required hidden states from vision transformer.

[geetu040]

@qubvel @pcuenca Thanks, I have updated the code for hidden_states.

I still need an opinion with fov (field of view)
DepthPro returns the predicted_depth as well as the fov which is a scaler value.

The existing DepthEstimatorOutput class in transformers/src/transformers/modeling_outputs.py looks like this:

class DepthEstimatorOutput(ModelOutput):
    loss: Optional[torch.FloatTensor] = None
    predicted_depth: torch.FloatTensor = None
    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
    attentions: Optional[Tuple[torch.FloatTensor, ...]] = None

Q1: Do I create a new class DepthEstimatorOutputWithFOV or update the existing class?
Q2: User should be given the option to turn the FOV on or off because calculating FOV requires extra processing. In this case should this parameter be a part of model initialization DepthProForDepthEstimation(config, return_fov=True) or should it be kept inside config.

[qubvel]

Thanks @geetu040

Q1:

class DepthProDepthEstimatorOutput(DepthEstimatorOutput):
    fov: Optional[torch.FloatTensor] = None

This output can be returned in both cases: fov=None and not None.

Q2:

Yeah, this can be a parameter of the config, but also should be an argument in forward method to override the config parameter (similar to output_hidden_states)

Please, let me know if you have more questions!

[geetu040]

Yeah, this can be a parameter of the config, but also should be an argument in forward method to override the config parameter (similar to output_hidden_states)

This needs to be done during __init__, because it requires fov_model (another vision transformer) to be initialized.

[qubvel]

OK, got it! Then it should be done with config! And anyone can just load a model as following:

model = DepthProForDepthEstimation(checkpoint, fov_model=True)
# or
model = DepthProForDepthEstimation(checkpoint, fov_model=False)

With such initialization fov_model param will be overridden in config

[geetu040]

• currently an image is down-scaled to medium resolution (high / 2) and low resolution (high / 4)
• then patches are created from high, medium and low and concatenated.

I was wondering can we also give this option to the users to decide which scales to use, for example, a user tells in config to use these custom scales image_scales=[0.6, 0.4, 0.3]

• now an image will downscale to these 3 scales
• then patches are created from high and scaled images and concatenated.

@qubvel I have looked into the code how this can be implemented, it is do-able and I can easily make this option available and I would prefer that, but I have to ask you as well, do you think this option should be given to the users?

[qubvel]

Hi @geetu040, we try to avoid overcomplicated code with lots of parameters, the general rule is to get rid of different code paths / unused params that are not different across pretrained checkpoints. For this particular case, feel free to add it, but only in case it will not introduce extra complexity to the modeling code.

[geetu040]

Hi @qubvel I have a question about the image processor.

the source code from apple/depth-pro preprocesses the image in this sequence normalize -> resize, however in conventional image processor for vit and dpt, the sequence is resize -> normalize

this causes the two outputs to be slightly different from each other.

do you suggest I stay with the convention and ignore the minor difference in output or I make the implementation exactly like the source code, I am not very sure how to do this because the original resize function gives an error if it is simply moved above normalization code and if I use torch.nn.funtional.interpolate that is also not very optimal, it requires data conversions.

Here are the outputs

Different in Outputs

there is a slight difference, this happens because of how the image is pre-processed before being given to the model

Source code results

ic| depth: tensor([[0.9604, 0.9329, 0.8837,  ..., 3.0123, 2.9720, 2.9517],
                   [0.9210, 0.8995, 0.8605,  ..., 3.0148, 3.0120, 3.0106],
                   [0.8811, 0.8655, 0.8366,  ..., 3.0245, 3.0473, 3.0592],
                   ...,
                   [1.2283, 1.2263, 1.2225,  ..., 1.2698, 1.2818, 1.2881],
                   [1.2228, 1.2241, 1.2266,  ..., 1.2679, 1.2806, 1.2872],
                   [1.2167, 1.2223, 1.2333,  ..., 1.2655, 1.2757, 1.2810]])
ic| depth.shape: torch.Size([2268, 3024])
ic| focallength_px: tensor(3362.0200)

HF code results

ic| predicted_depth: [tensor([[0.9727, 0.9443, 0.8937,  ..., 3.0023, 2.9608, 2.9399],
                             [0.9320, 0.9097, 0.8693,  ..., 3.0045, 3.0006, 2.9987],
                             [0.8899, 0.8737, 0.8439,  ..., 3.0129, 3.0352, 3.0469],
                             ...,
                             [1.2393, 1.2373, 1.2334,  ..., 1.2805, 1.2934, 1.3001],
                             [1.2344, 1.2356, 1.2379,  ..., 1.2802, 1.2935, 1.3004],
                             [1.2286, 1.2341, 1.2447,  ..., 1.2788, 1.2892, 1.2947]])]
ic| fov: [tensor(3383.9839)]

---

Difference in Output Image

visually no difference in the 2 images

Input Image

Source code results

HF code results

[geetu040]

Also how does the weight conversion work?

I have created the script for weight conversion, but when and who uploads that on huggingface? because I would need these converted weights for examples in docstring.

[pcuenca]

Checkpoint has been transferred.

[qubvel]

run-slow: depth_pro

[github-actions]

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/depth_pro']
quantizations: [] ...

[geetu040]

Checkpoint has been transferred.

@qubvel Shouldn't the checkpoint be apple/depth-pro-hf instead of apple/DepthPro-hf as you suggested? The code uses apple/depth-pro-hf everywhere.

[pcuenca]

@geetu040 I followed the same pattern for the model family. Let me check if there's anything we can do.

[geetu040]

@geetu040 I followed the same pattern for the model family. Let me check if there's anything we can do.

Otherwise I can change the checkpoint in the code, and it would also require to be changed in the model card.

[pcuenca]

https://huggingface.co/apple/depth-pro-hf now redirects to DepthPro-hf. So code should work.

[pcuenca]

But good point about the model card, I'll change so it's less confusing. Edit: updated.

[qubvel]

Thanks @pcuenca!

@geetu040 can you please update the code references? sorry for the back-and-forth changes.

Waiting for the slow CI to merge it then.

[geetu040]

@geetu040 can you please update the code references? sorry for the back-and-forth changes.

sure, updated!

[qubvel]

run-slow: depth_pro

[github-actions]

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/depth_pro']
quantizations: [] ...

[geetu040]

@qubvel, the tests did not run in the last 2 attempts.

Error Log

Run PR_MERGE_SHA=$(git log -1 --format=%H)
  PR_MERGE_SHA=$(git log -1 --format=%H)
  if [ $PR_MERGE_SHA != $VERIFIED_PR_MERGE_SHA ]; then
    echo "The merged commit SHA is not the same as the verified one! Security issue detected, abort the workflow!";
    exit -1;
  fi
  shell: sh -e {0}
  env:
    HF_HOME: /mnt/cache
    TRANSFORMERS_IS_CI: yes
    OMP_NUM_THREADS: 8
    MKL_NUM_THREADS: 8
    RUN_SLOW: yes
    HF_HUB_READ_TOKEN: ***
    SIGOPT_API_TOKEN: ***
    TF_FORCE_GPU_ALLOW_GROWTH: true
    RUN_PT_TF_CROSS_TESTS: 1
    CUDA_VISIBLE_DEVICES: 0,1
    matrix_folders: models_depth_pro
    VERIFIED_PR_MERGE_SHA: 850bdaaad25826da72b87e9455296742bb83e331
The merged commit SHA is not the same as the verified one! Security issue detected, abort the workflow!
/__w/_temp/881ad002-6e8a-4896-968a-9a2b93d19337.sh: 4: exit: Illegal number: -1
Error: Process completed with exit code 2.

Looks like something is wrong with the workflow itself.

[qubvel]

@geetu040 yes, smth wrong with CI, waiting for the team to fix it

[qubvel]

run-slow: depth_pro

[github-actions]

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/depth_pro']
quantizations: [] ...

[qubvel]

@geetu040, congratulations on getting the model merged! 🎉🚀
Fantastic work! Huge thanks for all your iterations to make the modeling code simpler, faster, and more export-friendly. It was a pleasure to support you in contributing this model, thank you!

[geetu040]

Thanks @qubvel

[Boulaouaney]

Hi!

I know this is merged already but I wanted to point out a small detail missing compared to Apple's implementation.

In the load_rgb() method in Apple's implementation, they check for the focal length in the EXIF data and extract it if it exists.

Reference: Apple's implementation load_rgb() method

Then later, although by default they predict the FOV if use_fov_head == True, they only use the predicted FOV to calculate the focal length if there was none in the EXIF data.

Reference: Apple's implementation depth_pro.py

I think this would be good to add to Hugging Face's implementation since it removes one layer of estimation, lowering the margin of error.

[qubvel]

Hi @Boulaouaney, thanks a lot for pointing this out! Would you like to propose a PR to update it?

[geetu040]

Hi @qubvel and @Boulaouaney
FOV value is used only by the DepthProImageProcessor.post_process_depth_estimation, can one not update the outputs like this?

...
model = DepthProForDepthEstimation.from_pretrained("apple/DepthPro-hf", use_fov_model=False)
outputs = model(**inputs)

# use the infered fov from given image
outputs.field_of_view = torch.tensor([0.4])

post_processed_output = image_processor.post_process_depth_estimation(
    outputs, target_sizes=[(image.height, image.width)],
)
...

[Boulaouaney]

Hi @geetu040,

Yes, you're right. One could manually adjust the image processing if they know the focal length used to take the image. However, I still believe it would be more efficient to handle this automatically within the image processor. Automating this process ensures consistency and reduces the potential for human error. Additionally, it would streamline the workflow for users who may not be familiar with the technical details of image processing.

@qubvel yes, if necessary I can open a PR when I'm free to update this

[geetu040]

@Boulaouaney I understand, so you are suggesting that we extract the FOV value as well along with the RGB values from the raw input image right?

[qubvel]

I'm not sure if we need to read it in the image processor, because I'm not sure how consistent it is among image formats. Currently, image/metadata reading is the user's responsibility. However, we should clearly describe this in the documentation and the model card. Also, it's better to avoid modifying output and to make FOV an optional argument for the postprocessing method.

[Boulaouaney]

@geetu040 yes I am suggesting to extract the focal length in 35mm from the image's metadata if it exists and convert it to focal length in pixels f_px to be used for calculating the metric depth instead of the estimated value from the fov_model.

The logic to be added would be similar to apple's implementation:

# Extract focal length at 35mm from exif data
f_35mm = img_exif.get(
    "FocalLengthIn35mmFilm",
    img_exif.get(
        "FocalLenIn35mmFilm", img_exif.get("FocalLengthIn35mmFormat", None)
    ),
)

# if extracted, convert it to pixels
if f_35mm is not None:
    f_px = f_35mm * np.sqrt(width**2.0 + height**2.0) / np.sqrt(36**2 + 24**2)

But I guess @qubvel 's solution to leave the metadata handling to the user during image loading would also be good.

[qubvel]

Ok, it would be nice to mention it in docs (depth_pro.md), feel free to open PR to clarify it, then we can update examples and model cards on our side.