An Open-Source Infrastructure for Real-Time Automatic Agricultural Machine Data Processing

Abstract. Real-time big data technologies have seen an unprecedented growth due to their crucial impacts on different scenarios, such as error detection, live monitoring and metrics analytics. Recently, the agricultural industry has begun leveraging some of those technologies for advancements in digital agriculture; however, it largely ignores one of the richest data sources – the machine data generated from modern agricultural machinery. This machine data set contains critical as-harvested and as-applied data that lead to powerful insights and predictions of a farm‘s operational plan, current state and productivity. Furthermore, farm operators will be able to make adaptive logistics decisions based on predictive analytics. Recent attempts to automate the machine data collection process have been made. The challenge is to construct a scalable ecosystem that ingests and processes the collected machine data efficiently and quickly.

This paper presents an open-source infrastructure for collecting and processing agricultural machine data to produce useful data analytics and visualizations in real-time. The paper consists of three main parts. The first part of the paper discusses the making of ISOBlue2, a hardware device that serves as the starting point of the infrastructure for automatic agricultural machine data collection. The second part of the paper gives a detailed explanation of the Cloud components within the infrastructure. Specifically, this part analyzes how Apache Kafka and Apache Storm can work concurrently to enable a real-time, robust and scalable processing platform for agricultural machine data. The next part of the paper provides evaluations of a test infrastructure. Finally, the last part of the paper demonstrates a couple of scenarios that would benefit from applying this infrastructure.