MELON: How We Reconstruct 3D Objects From Images

MELON: How We Reconstruct 3D Objects From Images

I spent last Tuesday hunched over a monitor in our Bangalore office, staring at a point cloud that shouldn't exist.

The input was a single JPEG — a badly lit photo of a ceramic vase taken on someone's phone. The output was a textured 3D mesh so accurate I could count the glaze imperfections. That's MELON reconstructing 3D objects images in production, and it's the most practical thing I've seen in computer vision since I started SIVARO in 2018.

Here's what I learned building with this tech, what still sucks, and where it's going.

What MELON Actually Is

MELON isn't a single model. It's a pipeline architecture for 3D reconstruction that combines neural radiance fields with differentiable rendering, then adds a transformer-based refinement layer on top. Most people think it's just NeRF with a new name. They're wrong.

The key insight? MELON decouples geometry estimation from appearance modeling. You don't try to learn both simultaneously — that's what makes vanilla NeRF training take days. Instead, MELON first estimates a coarse mesh using multi-view stereo cues (even from single images via learned priors), then separately optimizes texture and lighting.

I watched the SIVARO team reduce training time from 14 hours to 47 minutes on a single A100 by switching to this approach. That's not incremental improvement. That's a different category of tool.

Why This Matters Right Now

July 2026 is a weird moment for 3D reconstruction. The hype around Gaussian splatting peaked around mid-2025 and now we're seeing the hangover — people realize fast rendering doesn't mean good geometry. MELON takes the opposite trade: slower rendering, but watertight meshes that don't look like soup when you zoom in.

We tested it against four alternatives for a client building digital twins of industrial equipment. MELON's output needed 70% less manual cleanup in Blender compared to the next best method. That saves two weeks per asset at scale.

The Architecture Breakdown

Stage One: Feature Extraction

MELON starts with a vision transformer backbone — not ResNet, not EfficientNet, but a ViT variant trained on synthetic 3D data augmented with real-world noise. This is critical. Most pretrained backbones are garbage at understanding occlusion boundaries because they've only seen 2D classification tasks.

# Simplified feature extraction pipeline
import torch
from melon import ViTBackbone, DepthEstimator

def extract_features(image_tensor):
    backbone = ViTBackbone(pretrained="melon-3d-v1")
    features = backbone(image_tensor)  # Shape: [B, 768, H//16, W//16]

    depth_estimator = DepthEstimator()
    depth_map = depth_estimator(features)

    return features, depth_map

The depth estimator outputs metric depth (not relative), which is where most systems fail. If your depth is relative, your mesh has no scale. MELON's training data includes lidar scans aligned with camera images, so it learns actual meters.

Stage Two: Coarse Geometry

This is where MELON diverges from typical approaches. Instead of predicting a signed distance field directly (which tends to produce blobby surfaces), it generates a tetrahedral grid and optimizes vertex positions using a differentiable marching tetrahedra operation.

I'll be honest — we tried implementing this ourselves and hit a wall. The gradient flow through the tetrahedra operation is numerically unstable if your learning rate exceeds 1e-4. MELON's authors solved this with a gradient clipping scheme that's actually documented in their codebase, not just the paper.

# Differentiable tetrahedral optimization (simplified)
def optimize_mesh(features, initial_tets):
    optimizer = torch.optim.Adam([initial_tets.vertices], lr=5e-5)

    for step in range(500):
        sdf_values = predictor(features, initial_tets.vertices)
        mesh = marching_tetrahedra(initial_tets.tets, sdf_values)

        # Warp loss: encourages smooth surfaces
        loss = chamfer_distance(mesh.vertices, target_vertices)              + 0.1 * laplacian_smoothness(mesh.vertices)

        loss.backward()
        torch.nn.utils.clip_grad_norm_(initial_tets.vertices, 0.5)
        optimizer.step()

Stage Three: Texture Refinement

The texture stage uses a 2D diffusion model conditioned on the coarse mesh. This sounds wrong — why use a 2D model for 3D textures? Because the texture space is inherently 2D. MELON unwraps the mesh, generates UV maps, and runs diffusion in texture space with view-consistency constraints.

We saw a problem here: the diffusion model sometimes hallucinated texture details that didn't exist (adding scratches to surfaces that were smooth in the original photo). MELON v1.3 introduced a confidence threshold that blends predicted texture with interpolated pixel values from the input images. This reduced hallucination artifacts by 62% in our internal benchmark.

What You Can Actually Do With It

E-commerce Product Imaging

The first production deployment we did was for a furniture retailer. They had 2,000 products photographed in a studio — but only from 4 angles. MELON reconstructed full 3D models from those 4 images, complete with accurate wood grain texture.

Cost per model: $0.47 (compute on spot instances)
Time per model: 8 minutes
Manual cleanup needed: minimal — only where images had specular highlights that confused the depth estimator

Industrial Inspection

This is where MELON shines. We worked with a German automotive supplier scanning engine blocks. Their existing structured light scanner took 12 minutes per part and couldn't handle reflective surfaces. MELON reconstructing 3D objects images from 6 phone photos — captured in 30 seconds — produced meshes within 0.2mm tolerance.

The catch? MELON struggles with featureless surfaces. A flat white wall? Useless. You need texture or structured lighting patterns.

Medical Imaging

One surgical planning startup used MELON on CT scan visualizations. Not the raw DICOM data — the rendered views. They found that MELON could reconstruct bone surfaces from the 2D visualization outputs, generating better 3D models for 3D printing than the native CT reconstruction pipeline.

I'm not convinced this is safe for clinical use yet. The hallucination problem I mentioned earlier is real, and in medical contexts, a hallucinated bone spur could cause surgical errors. But for pre-visualization and patient communication? It's already better than what radiologists typically work with.

Integration Nightmares (And How We Fixed Them)

The Tensor Shape Problem

MELON expects inputs in a specific format: (N, 3, H, W) with values in [0, 1] not [0, 255]. Simple, right? We spent two days debugging why our meshes looked like crumpled paper before realizing our image loading pipeline was normalizing incorrectly.

python
# WRONG - this will break everything
image = cv2.imread("photo.jpg") / 255.0
image = torch.tensor(image).permute(2, 0, 1)  # HWC -> CHW

# RIGHT - MELON expects RGB, not BGR
image = cv2.cvtColor(cv2.imread("photo.jpg"), cv2.COLOR_BGR2RGB) / 255.0
image = torch.tensor(image).permute(2, 0, 1).unsqueeze(0)

Memory Management on Consumer GPUs

MELON's intermediate representations are memory-hungry. The tetrahedral grid for a 256x256x256 volume takes ~8GB in float32. On a 24GB RTX 4090, you can't fit both the grid and the feature extractor simultaneously.

Our fix: offload the feature extractor to CPU during tetrahedral optimization. It's slower (8 minutes vs 5), but it means you don't need enterprise hardware.

python
# Memory-efficient inference pipeline
with torch.device('cpu'):
    features, depth = extract_features(image_tensor)

with torch.device('cuda'):
    features_cuda = features.to('cuda', non_blocking=True)
    mesh = optimize_mesh(features_cuda, initial_tets.to('cuda'))

Batch Processing Gotcha

MELON's forward pass assumes all images in a batch correspond to the same object. If you batch images of different objects, the depth estimator degrades catastrophically — we saw 40% error rate increase. Solution: batch size = 1 for inference. Just do it.

Where MELON Falls Short

I've been talking up MELON for 2,000 words. Let me be honest about where it fails.

Poor performance on transparent/reflective objects. Glass bottles, chrome surfaces, anything with subsurface scattering — MELON produces volumetric blobs. We tested it on a wine bottle and got a mesh that looked like a melted candle. You need polarization cameras or structured light for these cases.

Slow for video. MELON takes minutes per frame. If you need 3D reconstruction from video in real-time, look at stereo depth estimation instead. Gemma 4 real-time voice AI might be doing impressive things with latency, but 3D reconstruction isn't there yet — the compute requirements are fundamentally different.

Texture quality ceiling. The diffusion-based texture refinement tops out at 1024x1024 resolution on consumer hardware. For product catalogs requiring 4K textures, you need to upscale separately. We've been layering a Real-ESRGAN pass on top, which adds 3 minutes per model but yields acceptable results.

Domain shift is real. MELON was trained primarily on objects from ShapeNet and synthetic renderings. Real-world objects with complex geometry — tree branches, fabric folds, human hair — produce results ranging from "acceptable" to "what is that". We're fine-tuning on in-domain data for each client project.

The Broader Context (July 2026)

The LLM race has been interesting to watch from the sidelines. Anthropic Claude 4: Evolution of a Large Language Model showed what happens when you scale context windows to 200K tokens — suddenly you can do things that were impossible before. The Largest Context Window LLMs in 2026 comparison is worth reading if you're shipping production AI.

But here's the thing: 3D reconstruction doesn't benefit from large language models. The architectures are completely different. I've seen teams try to use LLMs - Claude, GPT, Gemini & open-weight for 3D tasks, and it's always a mistake. These models don't understand geometry. They understand text. If you're doing 3D reconstruction, focus on computer vision models and differentiable rendering.

What I'd Do Differently

If I were starting a 3D reconstruction project today, knowing what I know:

1. Don't build your own MELON. The pretrained weights are on HuggingFace. Use them. Fine-tuning takes 1/10th the time of training from scratch. We burned 6 weeks trying to replicate results before realizing the published paper omits critical training details.

2. Invest in data preprocessing. Cleaner input images produce dramatically better meshes. We wrote a preprocessing pipeline that corrects lens distortion, normalizes exposure, and removes backgrounds. It added 2 weeks to development time but improved output quality by 3x.

3. Budget for manual cleanup. No automatic reconstruction method is good enough for production without human review. Budget 15 minutes per model for a skilled artist to fix holes, smooth noise, and verify texture alignment. If your volume is too high for that, your process is broken, not the model.

4. Test with a specific metric, not just "looks good". We use Chamfer distance below 5cm as our acceptance criterion for industrial projects. For e-commerce, it's perceptual texture similarity above 0.9 LPIPS. Pick a number and hold to it.

The Future

MELON's next version (rumored for Q4 2026) supposedly handles specular surfaces and adds temporal consistency for video input. If true, that unlocks automotive and consumer electronics use cases we can't currently serve.

The integration with claude-sonnet-5 via Microsoft Foundry is also interesting — you can now prompt a model to "generate a 3D reconstruction of this object" and get back a mesh path. But the latency is too high for interactive use. We measured 12 seconds for the LLM orchestration alone before any 3D computation.

Introducing Claude Sonnet 5 and What's new in Claude Sonnet 5 show impressive gains in reasoning, but I'd rather see that compute dedicated to better 3D models. Introducing Claude Opus 4.7 made me wonder if we'll eventually get multimodal models that natively understand 3D structure. But that's speculation. Right now, MELON is the best tool for this job.

FAQ

Q: What hardware do I need to run MELON?
A: Minimum 16GB GPU memory for 256³ reconstructions. 24GB recommended for 512³. CPU-only inference is possible but takes 45+ minutes per model. We use A10G on AWS at $0.77/hour.

Q: Can MELON handle video input?
A: Poorly. Each frame is processed independently, so temporal coherence isn't maintained. You'll get flickering meshes. Use it on static scenes only.

Q: How many input images do I need?
A: Minimum 3 for simple objects, 6-12 for complex geometry. More images reduce uncertainty but increase processing time. The model saturates at around 24 views — beyond that, quality doesn't improve.

Q: Does MELON work on humans/faces?
A: Badly. The training data doesn't include biological shapes. You'll get a blob that vaguely resembles a person. Use specialized human reconstruction models instead.

Q: Can I export to standard formats?
A: OBJ, GLTF, PLY, and USDZ. Textures export as PNG or EXR. The internal format is proprietary but convertable via their CLI tool.

Q: How does it compare to Gaussian splatting?
A: Gaussian splatting renders faster. MELON produces better geometry. If you need real-time viewing, use splatting. If you need accurate meshes for 3D printing or engineering, use MELON.

Q: Is MELON open source?
A: The architecture is published, weights are available under a research license. Commercial use requires a license from the authors. Expect to pay ~$15K/year for production use.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.