Click on “Download PDF” for the PDF version or on the title for the HTML version.

If you are not an ASABE member or if your employer has not arranged for access to the full-text, Click here for options.

Remote Surveillance Video Activity Recognition Using Spatiotemporal Convolutional Neural Networks for Greenhouse Workload Analysis

Published by the American Society of Agricultural and Biological Engineers, St. Joseph, Michigan

Citation:  2021 ASABE Annual International Virtual Meeting  2100608.(doi:10.13031/aim.202100608)
Authors:   Liang-Chien Liu, Dan Jeric Arcega Rustia, Ta-Te Lin
Keywords:   activity recognition, spatiotemporal convolutional neural network, computer vision, deep learning, greenhouse management

Abstract. With the advances of computer vision over the recent years, its applications have extended from handling static images to tackling the challenging domain of video analysis, with human activity recognition being among the core research topics. However, recognizing and recording the activity of agricultural workers through surveillance videos is still understudied. By workload efficiency analysis, the quality of agricultural produce can be determined and improved. In this work, an IoT system that utilizes MQTT messaging protocol to stream visual data collected from wireless monitoring nodes was developed. The video streams were displayed in a mobile APP for farm owners to remotely monitor their workplace and crop situation. A spatiotemporal convolutional network was trained to recognize the actions of the workers from the video streams, while the timestamp and duration were recorded in a remote server. Through vision-based analysis, workload data can be seamlessly extracted from the streaming footage, thereby eliminating the need for additional hardware setup, such as wearable sensors. The proposed algorithm was capable of achieving an average F1 score of 96.9%. The workload data analysis results could be beneficial for farm owners in reevaluating and improving their management strategy. Accurate detection and recording of farm activities will allow them to see if a particular activity had been performed appropriately and at the proper time. Furthermore, the usage of resources, including pesticide and water, can be tracked based on the frequency and duration of the activities. The proposed techniques and methods can be employed not only for agricultural applications, but also for other workplace monitoring applications.

(Download PDF)    (Export to EndNotes)