Robotic Localization and Mapping of Disease in Apple Orchards

Abstract: United States apple growers lose more than $100 million a year to fire blight, a bacterial disease caused by Erwinia amylovora that is detrimental to pome fruit trees such as apple and pear.This bacterium infects blossoms, shoots, and branches during the bloom season causing tissue to die. Therefore, the detection and removal of infected tissues over the dormant season is critical to prevent an outbreak in the following spring. However, the symptoms of fire blight are subtle and difficult to detect, and finding them still depends almost entirely on manual scouting, a process that does not scale to large orchards and is prohibitively expensive and time-consuming for growers to perform at the scale required for effective disease management. With the decreasing availability of labor in the agricultural sector, there is a pressing need for automated solutions that can perform this inspection at scale, and with high accuracy. This thesis aims to develop a robotic system capable of performing that inspection autonomously. Since the visual cues that distinguish infected tissue are subtle, easily occluded, vary with natural lighting, and are reliably resolved only at close range, the system relies on active perception: a manipulator positions a camera to acquire discriminative, task-relevant views of the canopy. We have collected a multi-modal dataset of dormant apple trees, the first to pair dense near-infrared imagery with flash-illuminated stereo RGB for this task, and trained detectors to recognize disease symptoms across both modalities. We then introduce a confidence-aware semantic mapping method that fuses these per-view detections into a persistent 3D representation of disease confidence across the canopy, and a next-best-view planner that actively selects viewpoints to refine the map's least confident, most contested predictions. Finally, we integrated the full pipeline onto Erwin, a mobile active perception robot built using an Amiga base and an xArm6 arm for manipulation of the camera rig. We successfully validated the system both in simulation and in the field on 12 trees at the Penn State Fruit Research and Extension Center under a genuine train-test domain gap, the semantic planner more than doubles the detection accuracy of a complete planar scan by the halfway point of the inspection, concentrating its views on the map's most contested disease evidence. This result demonstrates the central promise of active perception for orchard disease mapping: by deciding where to look next a robot can build maps of orchard disease that are accurate enough to support autonomous disease management, while minimizing the time and energy spent on inspection. Thesis Committee: Prof. Abhisesh Silwal (chair) Prof. Oliver Kroemer Itamar Mishani