Names:

• Adina Faye Bondoc
• Laura Isabella Forero Camacho
• Chidiebere Ogbuchi

3D reconstruction using Neural Radiance Fields (NeRF) is computationally demanding, making it impractical for real-time applications. The MobileNeRF and PlenOctree models offer promising solutions, enabling real-time rendering. This project seeks to replicate and evaluate these models to gauge their effectiveness in rendering NeRF models within a web browser using custom data.

In our project, we have embarked on a journey of exploration and experimentation, particularly focusing on the initial phase, known as "Step 2." This phase involves diving into the world of 3D reconstruction using Neural Radiance Fields (NeRF) and harnessing the power of the MobileNeRF PlenOctree for real-time rendering. To facilitate this exploration, we have organized our directory structure as follows:

This repository is dedicated to the exploration and enhancement of 3D reconstruction using Neural Radiance Fields (NeRF). The conventional NeRF models face computational challenges, rendering them impractical for real-time applications. In response, this project focuses on the implementation and evaluation of two promising models, MobileNeRF and PlenOctree, which offer solutions for achieving real-time rendering.

• metrics: Utility functions for evaluating model metrics.
• viewer: Components related to the web-based viewer.
  • nerf_synthetic: Synthetic data for NeRF model evaluation.
    • Specific scenes such as ficus_phone, drumsBDRP2_phone, ship_phone, etc.

• EDP_NeRF: Implementation of a Small NeRF variant.
• Tiny_NeRF: Implementation of another Small NeRF variant.

• nerfvis: PlenOctree visualization using Python. Based on the python library of the same name
• plenoctree: PlenOctree implementation: NeRF-SH training and Octree building.
  • nerf_sh: Training and evaluation of the NeRF-SH model.
  • octree: Creation of the plenoctree from a NeRF model (NeRF-SH or JaxNeRF).
• volrend: PlenOctree visualization using a C++ application (local and web available).

• datasets: Datasets for model training and evaluation.
  • nerf_synthetic: Synthetic data for NeRF model training.
    • Specific scenes such as chair, drums, etc.
• codenotebook.md: Information about running in various environments and common errors.
• results: Model training results and associated data.
  • Specific subdirectories for different scenes (drumsStage2, drums, chair, etc.).
    • Subdirectories further organized by metrics, object files, weights, and samples.

• MobileNeRF implementation optimized for M2-based MacBook Pro.

• MobileNeRF implementation optimized for TPU (Notebooks).
• datasets: Datasets specific to TPU-Colab-OPTAX setup.
  • Subdirectories for weights, chair dataset, and samples.

• Implementation of MobileNeRF optimized for GPU with the OPTAX optimizer.

This organized folder structure facilitates the replication, evaluation, and improvement of MobileNeRF and PlenOctree models, with a specific focus on rendering NeRF models within a web browser using custom data.

NeRF Original Paper

• Title: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
• Published in: ECCV 2020
• Link: Read the paper

NeRF on Device

• Authors: Jennifer(Kaiqi) Wu and Venkat S. Rao
• Published on: Stanford University CS231A (Previous Projects 2022)
• Link: Access the document

PlenOctrees for Real-time Rendering of Neural Radiance Fields

• Authors: Alex Yu, Deqing Sun, Forrester Cole, Jitendra Malik
• Published on: arXiv, 2021
• Link: Read the paper

MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures

• Authors: Zhiqin Chen, Thomas Funkhouser, Peter Hedman, Andrea Tagliasacchi
• Published on: The Conference on Computer Vision and Pattern Recognition (CVPR), 2023
• Citation: Chen, Z., Funkhouser, T., Hedman, P., & Tagliasacchi, A. (2023). MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures. In The Conference on Computer Vision and Pattern Recognition (CVPR).
• Link to the Paper: Read the paper