Drone-3DGS/README.md

Drone-3DGS

Two-flight DJI drone footage → 3D Gaussian Splatting pipeline for Apple Silicon Macs (M1/M2/M3).

Takes two .mp4 videos of the same scene from different angles, runs Structure-from-Motion via COLMAP, and produces a .ply Gaussian splat you can view in a browser.

---

Requirements

brew install colmap        # COLMAP 4.x (SfM)
brew install ffmpeg        # full version with all filters
python3 -m venv venv
source venv/bin/activate
pip install torch torchvision
pip install nerfstudio

Python version: 3.10 recommended (tested with 3.10.18 via pyenv).

---

Project structure

.
├── 1.mp4                  # first drone flight
├── 2.mp4                  # second drone flight
├── main.py                # Step 1 – extract frames + COLMAP feature extraction
├── match_features.py      # Step 2 – within-video SIFT matching (Python, bypasses COLMAP crash)
├── match_crossvideo.py    # Step 3 – cross-video exhaustive matching (v1×v2)
├── run.sh                 # Runs main.py (frame extraction + feature extraction)
└── train_splat.sh         # Steps 4–6: ns-process-data → splatfacto → export .ply

---

How to run

Step 1 – Extract frames and COLMAP features

source venv/bin/activate
bash run.sh

This calls main.py which:

1. Extracts frames from 1.mp4 and 2.mp4 at 2 fps into my_scene/images/ (named v1_*.jpg / v2_*.jpg)
2. Runs colmap feature_extractor — SIFT features written to my_scene/database.db
3. Runs match_features.py — sequential within-video matching (overlap=50) via OpenCV BFMatcher

Why not colmap exhaustive_matcher? COLMAP 4.x has a threading bug on Apple Silicon ARM64 causing a SIGSEGV in all matcher variants. match_features.py replaces it entirely: reads SIFT descriptors from the SQLite database, matches with OpenCV BFMatcher + Lowe ratio test + RANSAC, and writes two_view_geometries back to the DB. The mapper only needs that table.

Step 2 – Cross-video matching

python3 match_crossvideo.py

Matches every v1_* frame against every v2_* frame (14,900 pairs) so the two flights stitch into a single model. Takes ~70 min on M1 Pro CPU (~0.28 s/pair with OpenCV BFMatcher).

Step 3 – COLMAP mapper

colmap mapper \
  --database_path my_scene/database.db \
  --image_path my_scene/images \
  --output_path my_scene/sparse

Produces sparse models in my_scene/sparse/. The largest (most registered images) is the one to use. With overlapping flights you should get ~90–95% of frames in a single model.

Steps 4–6 – Convert, train, export

bash train_splat.sh

This script automatically:

1. Finds the largest COLMAP model in my_scene/sparse/
2. Converts it to Nerfstudio format with ns-process-data
3. Trains splatfacto — live viewer at http://localhost:7007 during training
4. Exports the Gaussian splat to my_scene/exports/splat.ply

---

Known issues and fixes applied

COLMAP 4.x matcher segfault (Apple Silicon)

All COLMAP matcher variants (exhaustive_matcher, sequential_matcher, vocab_tree_matcher) crash with SIGSEGV on ARM64 due to a bug in the SIFT worker thread initialization. Fix: match_features.py and match_crossvideo.py replace the COLMAP matcher entirely using OpenCV.

ffmpeg fps and split filters missing

The nerfstudio-bundled ffmpeg is compiled with a minimal filter set. Fixes:

• main.py uses -r output flag instead of -vf fps=...
• train_splat.sh prepends /opt/homebrew/opt/ffmpeg/bin to PATH so ns-process-data uses the full Homebrew ffmpeg

nerfstudio splatfacto hardcoded .cuda() calls

Two lines in the installed splatfacto.py call .cuda() unconditionally. Patched in-place:

Location	Original	Fix
populate_modules()	shs = torch.zeros(...).float().cuda()	shs = torch.zeros(...).float()
get_outputs_for_camera()	K = ....cuda()	K = ....to(self.device)

If you reinstall nerfstudio, re-apply with:

F=venv/lib/python3.10/site-packages/nerfstudio/models/splatfacto.py
sed -i '' 's/\.float()\.cuda()/\.float()/g' "$F"
sed -i '' 's/get_intrinsics_matrices()\.cuda()/get_intrinsics_matrices().to(self.device)/g' "$F"

---

3DGS on Apple Silicon — current status

splatfacto uses gsplat as its rasterizer. gsplat 1.x requires CUDA — there is no MPS or CPU fallback. On Apple Silicon the CUDA extension is None at load time and crashes at first use.

Two options for actual Gaussian Splatting:

Option A — Brush (recommended, uses Apple Metal natively)

# Install Rust (one-time)
brew install rustup && rustup-init -y && source ~/.cargo/env

# Build and run
cargo install --git https://github.com/ArthurBrussee/brush brush-cli
brush-cli --source my_scene/sparse/3

Outputs a .ply and has a built-in web viewer.

Option B — Google Colab (free GPU)

The scene is already in Nerfstudio format at my_scene/ns_data/. Zip it, upload to a Colab T4 instance:

!pip install nerfstudio
!ns-train splatfacto --data /content/ns_data --vis wandb

Download outputs/*/splatfacto/*/splat.ply when done.

---

Viewing results

What	How
During splatfacto training	http://localhost:7007 (Nerfstudio Viser viewer)
Sparse point cloud (ready now)	Drag my_scene/exports/sparse_pointcloud.ply into https://playcanvas.com/supersplat/editor
Final Gaussian splat	Drag my_scene/exports/splat.ply into https://playcanvas.com/supersplat/editor

PlayCanvas SuperSplat runs 100% in-browser — the file never leaves your machine.

---

Re-running from scratch

rm -rf my_scene outputs
bash run.sh                          # frames + features (~10 min)
python3 match_crossvideo.py          # cross-video matching (~70 min)
colmap mapper \
  --database_path my_scene/database.db \
  --image_path my_scene/images \
  --output_path my_scene/sparse      # mapping (~10 min)
bash train_splat.sh                  # convert + train + export