ASABE Technical Library - Abstract

Click on “Download PDF” for the PDF version or on the title for the HTML version.
If you are not an ASABE member or if your employer has not arranged for access to the full-text, view access and subscription options

A VISUAL-PROPRIOCEPTIVE CONTROL FRAMEWORK FOR SOFT ROBOTIC MANIPULATORS IN ORCHARD OPERATIONS

Published by the American Society of Agricultural and Biological Engineers, St. Joseph, Michigan www.asabe.org

Citation:  2026 ASABE Annual International Meeting  2600921.(doi:10.13031/aim.202600921)
Authors:   Syed Usama Bin Sabir, Divyanth Loganathan Girija, Ryan Dorosh, Manoj Karkee, Ming Luo
Keywords:   Agricultural robotics, automation in specialty crops, multi-sensor fusion, soft robotic manipulator, visual servoing, soft robotics.

Abstract. Labor-intensive orchard operations including fruit harvesting, pruning, and blossom thinning face growing challenges from seasonal labor shortages, rising labor costs, and the demand for greater precision. Rigid robotic manipulators, though extensively studied, have limited compliance that restricts safe operation in cluttered orchard canopies. Soft robotic manipulators offer inherent compliance and safety advantages, but their limited proprioceptive sensing makes accurate end-effector (EE) state estimation and closed-loop control challenging. This study presents a multi-sensor visual–proprioceptive control framework for precise EE positioning of a soft robotic manipulator in orchard environments, using apple picking as a representative use case. The system integrates a global RGB-D camera for target detection and coarse localization, a local end-effector-mounted depth camera for fine alignment, and an Inertial Measurement Unit (IMU) for orientation estimation. Apple targets are detected using a custom You Only Look Once (YOLO)v11 model with 3D positions estimated from registered depth map. An Unscented Kalman Filter (UKF) fuses AprilTag-based visual measurements with IMU data to produce a continuous 6 Degree of Freedom (DOF) EE pose estimate, driving a two-phase control strategy: global position-based visual servoing for coarse approach and local image-based visual servoing (IBVS) for precise final alignment. The framework was evaluated across 46 laboratory trials and 12 orchard trials, achieving a mean target localization error of 2.83 mm and a mean end-effector positioning error of 8.99 mm, with all errors below 20 mm, demonstrating sub-centimeter target localization and reliable closed-loop manipulation despite soft manipulator compliance and unstructured orchard variability.

(Download PDF)    (Export to EndNotes)