Tracking shiny objects without a CAD model: the light field as the object model
This post opens our series Light-field research, read for practice: every Tuesday until early January we pick one recent paper on light fields, explain what it does in plain terms and say what we think it means for industrial machine vision. The figures are our own illustrations based on the papers.
Anyone who has set up bin picking for polished or chrome-plated parts knows the problem: the object you want to locate looks different from every angle. Highlights wander across the surface as the part moves, and the reflections of the surroundings change with them. Classic pose estimation either needs a CAD model, or a set of reference images recorded beforehand – and both struggle as soon as the appearance is dominated by reflections.
Nikolai Goncharov, James L. Gray and Donald G. Dansereau from the Australian Centre for Robotics (University of Sydney) suggest a different route. Their argument is simple and, from a light-field perspective, elegant: a light-field image already is an object model. It records the scene from many viewpoints at once, so it contains both the geometry and the view-dependent appearance that a tracker would otherwise have to learn from reference views.
How it works
The method builds a new object model from every light-field frame and registers consecutive models against each other:
- Segmentation. The object is named by a text prompt (“jug”). Grounding DINO finds it in one sub-view, SAM 2 segments it and propagates the mask through all sub-views of the light field.
- Depth. A sparse but reliable disparity map comes from the classic structure tensor on epipolar-plane images. Depth Anything provides a dense but scale-free map; a linear fit to the sparse values gives it the right scale. From this, the object becomes a 3D point cloud.
- Gaussian splats. Every 3D point collects its colour from all sub-views in which it is visible. That is exactly where reflections are handled: the colour is stored as a view-dependent function (spherical harmonics), not as a single value. A short optimisation over all sub-views refines the splats.
- Registration. The model from frame t+1 is aligned to the one from frame t – first coarsely with feature matching and coloured ICP, then by rendering both splat sets and minimising the photometric error across all sub-views. The result is the relative 6DoF motion of the object.
What the results show
The authors recorded their own light-field dataset with 9 × 9 sub-views and objects of varying reflectivity, with ground-truth poses from motorised rotation and translation stages. As a baseline they use FoundationPose in its model-free mode, which needs reference images of the object – the light-field method needs none.
The picture is honest rather than triumphant. On a tilted, highly reflective jug the light-field tracker clearly wins: ADD 0.772 against 0.513, and a mean absolute rotation error of 8.3° against 53.8°. On the same jug moving straight, FoundationPose is better (ADD 0.909 against 0.805), and a sequence with motion along the viewing axis is a clear weak spot of the new method. Averaged over the recorded sequences, both approaches end up roughly on par – with the difference that one of them needed no prior information about the object at all.
The authors name the limits themselves: fast, abrupt motion is a problem, the per-frame processing of the light field is computationally expensive (no runtime is reported), and there is no global trajectory optimisation yet.
Our take
Three things stand out for industrial practice.
- Reflections become information. Most pipelines treat highlights as noise that has to be suppressed. Storing appearance per viewing direction turns them into a cue – the same idea that makes light-field cameras robust on glossy parts in the first place.
- No CAD, no teach-in. In a production cell a CAD model is usually available, so the real value lies elsewhere: mixed or changing parts, logistics, returns, or objects whose CAD model does not match the real part (coatings, deformation, tolerances).
- Not real-time yet. Without runtime figures and with foundation models in the loop, this is a research prototype. For cycle-time-critical cells we would expect the ideas – view-dependent appearance, online models from one light-field shot – to arrive first in components of a pipeline, not as a complete tracker.
Next Tuesday we stay with depth: how a focused plenoptic camera delivers metric depth from a single shot, with a little help from a foundation model.
The paper
Nikolai Goncharov, James L. Gray, Donald G. Dansereau: Light Field Based 6DoF Tracking of Previously Unobserved Objects. arXiv:2512.13007, December 2025. arxiv.org/abs/2512.13007
More on the basics: Plenoptic cameras in industry and how light-field technology works.