Ryukijano/agentic-sfm-qwen35-2b

Agentic-SfM: Qwen3.5-2B LoRA adapters

Research adapters for an RL-trained multimodal agent that orchestrates geometric-vision tools for hard structure-from-motion: cropping, matcher choice, doppelganger filtering, image retrieval, COLMAP execution, and inspect-or-retry decisions. Rewards come from geometric ground truth (inlier counts, pose error, registration, point counts), not human labels.

Status: negative so far. The pair-level agent does not beat plain LoFTR matching, and the scene-level evaluation is not a valid benchmark. Details below.

Provenance: the research idea is by Gabriele Berton (public research post). This repository implements and extends it. Code: github.com/Ryukijano/agentic-sfm.

Contents

Path	What it is	Stage
pair_sft/epoch_5/	Supervised warmup on oracle pair-matching trajectories	Phase 1 warmup
phase1_grpo/latest/	Pair-level GRPO checkpoint (LoRA). Training stopped at about step 200 when its 2-day time limit ran out.	Phase 1 RL
scene_sft/epoch_5/	Scene-level SFT on oracle retrieve, match, doppelganger_check, sfm_run, inspect, done trajectories	Phase 2 warmup
figures/	Match overlays, training-episode images, Phase 0 comparison, ScanNet ground-truth cloud	Results

The Phase 2 scene-level GRPO checkpoint is not in this repository. It is kept on the training cluster.

Adapter details

• Base: Qwen/Qwen3.5-2B. There is no separate -Instruct checkpoint for this size.
• LoRA: rank 32, alpha 64, on attention, MLP, and vision projections.
• Built for vLLM dynamic LoRA reload (VLLM_ALLOW_RUNTIME_LORA_UPDATING=1).

from peft import PeftModel
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-2B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(model, "Ryukijano/agentic-sfm-qwen35-2b", subfolder="phase1_grpo/latest")

Results

Pair level (hard validation set, n = 50)

Method	Success	Geometric reward
Trained GRPO agent	62%	0.595
Plain LoFTR	88%	0.965
Oracle crop baseline	90%	1.111

The trained agent is worse than LoFTR. Source: outputs/eval_pair/phase1_hard_best_8019429.json in the repository. The reward used in training includes a +0.5 first-call validity bonus, so training reward is not comparable with LoFTR's geometric score.

Scene level

The 10-scene evaluation (30% success) is not a valid benchmark. Two of its scenes were training scenes, the other eight were used to make design decisions, and the baselines were not run on the same budget or reconstruction path. Do not cite it as a result.

Pose priors (ScanNet, 30 frozen scenes, 1,170 pairs per method)

Pose source	AUC@5	AUC@10	AUC@20	Pairs with a pose
LoFTR, RANSAC essential matrix	0.143	0.260	0.353	1,170
VGGT correspondences, RANSAC	0.059	0.099	0.123	256
VGGT own extrinsics head	0.372	0.631	0.772	1,170

VGGT's extrinsics head is the strongest pose source measured here. Its correspondence path returned no matches on 862 of 1,170 pairs, which points to the confidence threshold. VGGT's paper lists ScanNet among its training datasets and says it excluded the corresponding ScanNet-1500 scenes from its own matching comparison. The paper does not list those scene IDs, so overlap with these evaluation scenes is unlikely but unconfirmed. Treat the prior numbers as provisional. The MASt3R row was not measured in this run because MASt3R was not importable in that job; it has since been installed and is queued for re-benchmarking. Source: outputs/bench_pose_priors/run_8213282/ in the repository.

Training evidence

Episode visualisations from GRPO on hard MegaDepth pairs. Green marks a verified inlier and red an outlier, from the geometric verifier. Titles give that episode's inlier count and reward.

Step 10 (Brandenburg Gate)	Step 20 (Brandenburg Gate)	Step 30 (St. Peter's)

Episode images show single episodes. They are not a measure of policy quality.

Data

• Pair-level RL uses 1,535 hard pairs. All of them come from two MegaDepth scenes, 0015 (402 pairs) and 0022 (1,133). Earlier descriptions of "102 scenes" for this data are wrong.
• Scene-level work uses 100 ScanNet indoor RGB-D scenes (RGB, depth, intrinsics, and per-frame poses).

Dense ground-truth ScanNet cloud (about 200k points, from depth and pose). It is the evaluation target, not an agent output.

Caveats

• Match figures show the top 100 correspondences. Titles give the full inlier count.
• The ScanNet point cloud is ground truth from depth and pose. It is never an agent output.
• These are research snapshots. Phase 1 stopped before convergence, and the held-out evaluation has not been run.