OpenAI created the hottest transformer-based text-to-image model, DALL-E, and then last week took another step to create text-to-3D generation, Point-E. From the depths of neural network fathom, Point-E is meant to be faster than Google’s DreamFusion and other text-based 3D models. 3D reconstruction models have proven to be very useful in improving the metaverse. Since Google released NeRF (Neural Radiance Field), a lot of new ones, used to make 3D models, have been created.

Check out these text-to-3D models to try, apart from Point-E.

Read: 3D Reconstruction Models Make Metaverse More Possible

DreamFusion

Google was one of the first ones to develop a 3D mesh creation software, DreamFusion. It uses 2D Diffusion to synthesise text-to-image and then using NeRF, the 3D model is rendered that predicts 2D images from random angles, which are then combined to create a single object. The model’s researcher paper proposes Score Distillation Sampling (SDS) for generating samples from diffusion models through optimization of loss function for allowing users to optimise samples in 3D spaces. 

Click here to read the research paper.

Magic3D

In November, NVIDIA released Magic3D for synthesising text into 3D models that offers even better quality output than DreamFusion by addressing two limitations – slow optimisation of NeRF and low-resolution space supervision of NeRF. NVIDIA leveraged the stylistic capabilities of eDiffi, their text-to-image diffusion model for transferring the style of the images into the 3D models. The best feature of Magic3D is that it allows users to edit the given input image through fine-tuning DreamBooth for optimising the 3D models using prompts, resulting in maximum fidelity in the generation.

Read the research paper here.

Text2Mesh

Taking a different approach of stylysing a 3D mesh by predicting local geometric details and colours of an image, Text2Mesh, creates 3D models by conforming the details to the text prompt. Text2Mesh does not require a pre-trained generative model or a 3D mesh dataset and can also handle low-quality meshes. The researchers used CLIP’s ability to embed images and text to produce view-consistent and meaningful stylisation over the entire 3D shape. 

Click here to learn more about Text2Mesh.

CLIP-Mesh

Developed by researchers at Concordia University, CLIP-Mesh uses zero-shot generation technique for the generation of 3D models with a text prompt. As the name suggests, the generation relies on a pre-trained CLIP model used for comparing input text prompts with rendered images.

The research paper points out the problems with Google’s method of converting meshes using NeRF, as it requires an additional step of converting an image into 3D models. CLIP-Mesh can directly generate 3D models. 

Click here to learn more about CLIP-Mesh.

The post 4 Alternatives to OpenAI’s Point-E appeared first on Analytics India Magazine.