ASABE Technical Library - Abstract

Click on “Download PDF” for the PDF version or on the title for the HTML version.
If you are not an ASABE member or if your employer has not arranged for access to the full-text, view access and subscription options

A Vision-Guided Digital Twin for Robotic Harvesting of Greenhouse Lettuce Using SAM3D and Isaac Lab

Published by the American Society of Agricultural and Biological Engineers, St. Joseph, Michigan www.asabe.org

Citation:  2026 ASABE Annual International Meeting  2600626.(doi:10.13031/aim.202600626)
Authors:   Al Bashir, Partho Ghose, Yibin Wang, Prem Raj, Azlan Zahid
Keywords:   3D reconstruction, Real-to-Sim, Robotic harvesting, Simulation, Lettuce

Abstract. Greenhouse lettuce is a high-value leafy crop, yet harvesting remains one of the most labor-intensive operations and often depends on skilled workers. Robotic automation is therefore crucial, but achieving reliable autonomy in greenhouses is challenging due to tight spacing, structural complexity, high occlusions, and variable plant geometry. This study introduces a vision-guided virtual environment—a digital twin of greenhouse-grown lettuce—designed to accelerate the development and training of autonomous harvesting behaviors. RGB-D data were captured using an Intel RealSense D435i camera and annotated to segment key scene elements, including lettuce and surrounding structures. Annotated data were utilized with SAM3D to generate consistent 3D scene and object-level segmentations, enabling the creation of a physics-aware virtual model. The reconstructed environment was imported into NVIDIA Isaac Lab, where we trained a UR10e manipulator equipped with a custom lettuce-harvesting end-effector to perform harvesting trajectories under realistic greenhouse conditions. System performance was quantified using standard trajectory and task-level metrics. Results demonstrate that the proposed pipeline produces a reusable, high-fidelity training environment and a learned control policy capable of autonomous harvesting motions. The presented framework provides a scalable baseline for rapid iteration and benchmarking in agricultural robotics and supports future extensions toward sim-to-real deployment in operational greenhouses.

(Download PDF)    (Export to EndNotes)