Reference-Prompted Instance Segmentation for Growing Retail Catalogs

Abstract: Retail and warehouse perception systems work against a catalog that never stops changing: a segmenter deployed today will eventually be asked to find products that did not exist when it was trained. A closed-set segmenter can only grow its vocabulary by widening its classification head and fine-tuning it, which risks the classes it already handled, or by retraining on the whole accumulated catalog, which costs more with every product added. This thesis studies reference-prompted segmentation, which specifies the target product with a canonical image rather than a class label or a text description, so that a product's identity enters the model as an input rather than as a slot in its output. We fine-tune a reference-conditioned segmenter one new product at a time -- twenty-seven sequential steps growing a three-product catalog to thirty -- using Learning without Forgetting, and compare it against a closed-set segmenter given the identical recipe, budget, and data. Our method retains its earlier products substantially better than the closed-set baseline. Every dataset in this thesis comes from isaac_datagen, a standalone system built on Isaac Sim as a synthetic-data engine for retail scenes. Under this protocol and at this scale, reference prompting lets a perception system keep pace with a growing catalog without ever retraining on all of it. Committee: Jeff Ichnowski (advisor) Shubham Tulsiani Bardienus Duisterhof