Towards Efficient Neural Radiance Field For Real-world Applications

Author

Aliyu Bagudu, Mohamed bin Zayed University of Artificial IntelligenceFollow

Date of Award

12-2023

Document Type

Thesis

Degree Name

Master of Science in Computer Vision

Department

Computer Vision

First Advisor

Dr. Fahad Khan

Second Advisor

Dr. Rao Anwer

Abstract

Neural Radiance Field (NeRF) is state-of-the-art computer vision technology that has tremendious applications and opportunity of improvements. NeRF is built upon funda- mentals of computer vision (pinhole camera model) and therefore provides an opportunity for a wide range of applications that can be built off of it. To clarify, computer vision is a primary branch of artificial intelligence (AI). Thus, NeRF has wide range of applications that makes research in it undoubtedly worthwhile. It applications include but not limited to generation 3D models of objects, modeling of virtual and augmented reality scenes in the metaverse, rendering scenes for 3D video games, rendering and persisting of medical imagery data, such as magnetic resonance imaging (MRI) and ultrasound scans. Speaking generally, NeRF is applicable to processing (interpolation and efficient persistence) of any type of signals(i.e., ariel satellite images, time-series-based). However, NeRF requires further research, as the technique does face limitations. Specif- ically, this technique currently needs improvements in composability, time-sensitivity, and editability. Thus, the research on optimizing NeRF is highly valuable in light of the vast, real- world applications that the optimization of NeRF stands to benefit. Effiecient NeRF solutions are proposed. This results from a novel, recursive, value based, volume rendering model proposed. This feat is accomplished through an application of a Computational Science & Engineering (CSE)centric approach to NeRF, whereby the NeRF model is viewed from a CSE lens. This provides better understanding of the problem and solution algorithms and thereby enables proposal of suitable modifications. This work reduces NeRF solution to a Reinforced Learning (RL) solution with massive implication that NeRF solutions can benefit from RL solutions and vice versa. An abstract advantage Reinforced NeRF gains from RL is that a real-time value based solution is fea- sible without caching (or baking) a pre-trained model. On the other hand, NeRF solution is able to predict and represent a complex high dimensional signal in great resolution and accuracy from a sparse lower dimensional training samples. Therefore RL solutions stand to benefit from this. JAXNeRFnoRays is proposed as by product of efforts on Reinforced NeRF. This is a single shot prediction of ray values. JAXNeRFnoRays out performs NeRF and JAXNeRF by order of hundreds magnitude in inference speed. It also has a better space performance due to the use of a single MLP model.

Comments

Thesis submitted to the Deanship of Graduate and Postdoctoral Studies

In partial fulfilment of the requirements for the M.Sc degree in Machine Learning

Advisors: Fahad Khan, Rao Anwer

Online access available for MBZUAI patrons

Recommended Citation

A. Bagudu, "Towards Efficient Neural Radiance Field For Real-world Applications,", Dec 2023.