|
Testing the Agreement Between a Traditional and UAV-Based Method for Quantifying Skips in Suboptimal Cotton Stands
Enrique E. Peña Martinez1,*, Jason K. Ward1, Guy Collins2, Natalie Nelson1
Published in Journal of the ASABE 66(1): 149-153 (doi: 10.13031/ja.14760). Copyright 2023 American Society of Agricultural and Biological Engineers.
1Biological and Agricultural Engineering, North Carolina State University, Raleigh, North Carolina, USA.
2Crop and Soil Sciences, North Carolina State University, Raleigh, North Carolina, USA.
*Correspondence: eepena@ncsu.edu
The authors have paid for open access for this article. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License https://creative commons.org/licenses/by-nc-nd/4.0/
Submitted for review on 16 July 2021 as manuscript number ITSC 14760; approved for publication as a Research Article by Associate Editor Dr. Joe Dvorak and Community Editor Dr. Naiqian Zhang of the Information Technology, Sensors, & Control Systems Community of ASABE on 28 November 2022.
Highlights
Abstract. When suboptimal cotton stands occur, growers face the decision to accept or reject the stand. The replanting decision is difficult because the tradeoffs associated with replanting expenditures and reduced yields are difficult to objectively assess. Traditional methods like visual assessments and manual counts of cotton stands are commonly used to support a replanting decision. Typically, manual counts of skip size and frequency will provide more accurate assessments of the stand than visual assessments, but they are cumbersome to conduct and may not provide clear evidence that a replant is needed. Still, manual counts are popular among cotton farmers and the scientific community. Skip counts generated with the help of unmanned aerial vehicles (UAV) are less popular among cotton growers but provide more coverage and a larger sampling size across a given field. Therefore, UAVs have the potential to overcome the limitations associated with traditional methods. The motivation behind this study is to inform readers if manual methods can still be used for accurate decision-making regarding the replanting decision. More specifically, we study the interchangeability, or agreement, between a manual and a UAV-based method using Bland-Altman plots. Each method quantified skips greater than or equal to 0.91 m at different sampling sizes. Treatment plots varied in their stand counts, skip size, and skip frequency. Agreement between both methods was only found in the lowest stand treatment, where skips of large sizes were predominant. Conversely, methods disagreed in the higher stand where skips greater than or equal to 0.91 m were scarce.
Visual suboptimal plant stands are a common occurrence in cotton farming (fig. 1). Inconsistent cotton emergence has been associated with planting time, seed vigor or size, seed quality (warm or cold germination), and poor environmental conditions (Collins and Edmisten, 2016; Pettigrew and Johnson, 2005; Pettigrew and Meredith, 2009). In the presence of suboptimal stands, growers need to decide if replanting and the associated costs and risks are justified. The decision to accept or reject a stand has consistently been one of the most challenging tasks for cotton growers in previous years (Butler et al., 2019; Craig, 2010; Jost et al., 2006). Previous research has suggested that replanting is justified when a field contains skips greater than or equal to 0.91 m (3 ft) in 50% or more of the planted area (Jost et al., 2006). This recommendation has been adopted in a few cotton-producing states, such as North Carolina and Tennessee (Collins and Edmisten, 2018; Craig, 2010).
Manual assessments usually consist of strategically sampling portions of the field and extrapolating the results to the entire field. Manually counting cotton skips provides more accurate information about the stand’s health than visual assessments. However, advancements in remote sensing technologies like UAVs are now being used for plant scouting in entire fields. Recent studies have proposed using UAVs for plant counting, which can be useful for cotton stand assessments. These methods incorporate image processing techniques that can potentially replace or enhance data obtained manually to support a replant decision. Plant detection accuracy using UAVs has been demonstrated to be reliable. Chen et al. (2018) compared UAV detections to manual counts, and the study resulted in an 88.6% identification accuracy. Feng et al. (2019) successfully detected 98% of the plant populations using a hyperspectral sensor mounted on a UAV. These plant counts can then be used to measure skip counts of 0.91 m or greater, which are a better indicator of yield than plant population (Jost et al., 2006; Butler et al., 2019), and inform the growers if a replant is needed. Although UAVs are a very powerful tool, they have a low adoption level among cotton farmers. So, the question now becomes, can cotton farmers use a manual sampling method with a smaller sampling potential than a UAV-based method to accurately inform a replanting decision?
 |
| Figure 1. (a) Visually optimal cotton stand and (b) Visually suboptimal cotton stand. Peanut Belt Research Station. Lewiston, NC. 2019. |
Butler et al. (2019) came closest to overcoming barriers to the use of UAVs for informing the replant decision. The study used a UAV equipped with an optical multispectral sensor to measure plant uniformity and gaps, or skips, between plants. They argued that the identification and quantification of skips is the most important component in deciding whether to replant or not. Their methods consisted of combining a series of ArcMap (v10.8.1) tools to quantify the uniformity of emerged cotton plants based upon in-row plant distances, then determining when yield reductions would most likely occur based on uniform skip spacing. There is an opportunity to build on this study by conducting a statistical comparison between a UAV-based method and a traditional manual method.
It was our interest to explore if a traditional manual method, which has a smaller sampling size, would still inform the same replant decision as a UAV-based method. Therefore, our goal was to test the agreement of a UAV-based method and a manual method in quantifying skips that were greater than or equal to 0.91 m in length. Our objectives were (1) to quantify the skipped area percentages using both methods and (2) to conduct a statistical comparison of both methods’ agreement in their estimates of skipped area.
Materials and Methods
Experimental Design
Experiments were conducted at three sites in North Carolina during 2019: the Upper Coastal Plains Research Station (35.89°, -77.68°), the Peanut Belt Research Station (36.13°, -77.17°), and the Tidewater Research Station (35.85°, -76.65°), located near Rocky Mount, Lewiston, and Plymouth, NC, respectively. At each location, both early and late planted trials were conducted. Each planting trial indicated a different time of planting. The early planting trial indicated a typical planting window, and the late planting trial indicated late planting windows. Within each trial, four treatments were arranged in a randomized complete block and were replicated four times. Each plot contained four rows that were planted at a rate of 107,637 seeds ha-1 (43,560 seeds acre-1). Treatments consisted of varying ratios of transgenic (DP1646 B2XF) and non-transgenic (DP493) cotton seed: 100%, 75%, 50%, and 25%. The expected seed spacing for a 100% treatment plot was 0.1 m (4 in). The length of all trials varied between 90 m (295 ft) and 171 m (561 ft). Early planting for the early trials across all locations occurred between 29 April 2019 and 7 May 2019. Late planting for the late trials occurred between 23 May 2019 and 28 May 2019. Following emergence, all treatments except the replant treatment received three weekly sequential applications of glyphosate (32 oz/A) and gluphosinate (42 oz/A) to eliminate the non-transgenic seedlings and create natural, random, non-systematic skips that varied in size and frequency among adjacent plants in each row.
Manual and UAV-based Methods
Two methods were used for measuring skips in the field: a manual method described by Jost et al. (2006) and a UAV-based method. Manual measurements consisted of randomly subsampling skips in 61 m (200 ft) sections within the two center rows in each plot (30.5 m for each row). The subsamples constituted between 9% and 17% of the plot length, so they were representative of the entire plot and were deemed acceptable to conduct a statistical comparison between both methods.
The UAV-based method collected images over the entire plot area using a DJI Matrice 600 Pro (DJI, Shenzen, China) UAV equipped with a DJI Zenmuse X5 (DJI, Shenzen, China) RGB camera. The camera featured 16 megapixels of resolution and a focal length of 15 mm. Autonomous flights were generated using Precision Flight software (PrecisionFlight, Raleigh, NC) installed on a handheld tablet. Imagery was collected using a setting of 80% front and side overlap. The study areas were flown after all herbicide applications (three applications: one application per week) were completed three to four weeks after planting and after non-transgenic seedlings were terminated. At this point, most plants had between two and four true leaves. The skipped areas were derived using equation 1:
(1)
where
S = skips greater than or equal to 0.91 m
L = length of the plot (L = 30.5 m for manual method)
i = index counter for skips greater than or equal to 0.91 m
n = total number of skips in the plot.
The UAV flew over the early and late trials at an altitude of 53 m above ground level and generated imagery with a ground sampling distance resolution of 2.3 cm px-1. Imagery was then stitched and evaluated using PrecisionHawk Ag Analytics (PrecisionHawk, Raleigh, NC). The PrecisionHawk proprietary plant counting algorithm was applied to the stitched imagery, which returned an orthomosaic of the field and a plant count file with georeferenced point data of the plants. A plant detection accuracy of over 90% was expected based on company reports from their website articles (PrecisionHawk, 2019). A portion of both the plant count file and the orthomosaic map is shown in figure 2.
Data Processing for the UAV-based Method
Using QGIS (v3.8.0) and its Distance Matrix function, the linear distances between a reference plant (the first plant observed in the center rows of each plot) and the remaining plants in each row were calculated. Similar to the manual method, only the inner two rows for each plot were subjected to this analysis (fig. 3). This time, entire rows were sampled, constituting 50% of the entire plot. Within each row, two selections were made: a reference and a target selection. The reference plant is the first plant in the row, and it was selected and saved as a new layer. Similarly, the remaining plants in the same row (the target plants) were selected and saved as another layer. For each row, the QGIS Distance Matrix tool was used to generate a temporary layer that contained the distances between the reference and each of the target plants in the row. Each layer was then saved in CSV format and stored in a known folder. An algorithm in R (v.4.0.2) was written to transform the linear distances between the reference and target plants into distances between adjacent plants in each row. Similar to the manual method, equation 1 was used to derive the skipped area percentages for the UAV-based method.
 |
| Figure 2. Plant count layer over orthomosaic map in Lewiston, NC. Seedlings were detected with UAV-based method. Treatments produced wide spectrum of skip sizes and frequencies. |
 |
| Figure 3. Selection of reference (black) and target (blue) plants in rows of interest in QGIS 3.8.0. |
Although both methods differed in the percentage of the sampled area for each plot, the percentages for each sample were representative of the plot. More importantly, the total number of samples (one sample per plot) was the same for each method, thus making a comparison between methods possible.
The Bland-Altman Analysis
The Bland-Altman analysis (Bland and Altman, 1999) was conducted to evaluate the agreement between the manual method and the UAV-based method. The analysis generated Bland-Altman plots to visualize and compare the ability of both methods to quantify skipped area percentages. The analysis identified systematic differences between both methods. This simple method evaluated the bias between the mean differences from the two methods and helped establish limits of agreement at a 95% confidence interval (CI) (Giavarina, 2015). More specifically, the plots compared the agreement between the manual and UAV-based methods in measuring the skipped area percentage. Differences between methods were calculated using equation 2, and means were calculated using equation 3. We identified the treatments that captured a difference of zero (or “line of equality” as described by Giavarina [2015]) in the 95% CI of the mean difference. We also analyzed several treatment combinations to further explore the dynamics across treatments. Our combinations were: 100% + 75% + 50% + 25%, 100% + 75% + 50%, and 75% + 50%. An interval that captured the line of equality would suggest that both methods statistically agreed in the mean of their differences.
(2)
where
??????????????,?? = the percent of planted area occupied by skips>0.91 m detected by the manual method in plot i
????????,?? = the percent of planted area occupied by skips>0.91 m detected by the UAV-based method in plot i.
(3)
The blandr.method.comparison and blandr.output.report functions from the “blandr” package (Datta and Love, 2018) were used in R to evaluate and visualize differences between methods. Paired t-test p-values were obtained to inform whether both methods agreed in their measurements or not. A p-value greater than 0.05 would indicate a lack of evidence against the null hypothesis (eq. 4) and suggest that the mean difference between both methods is not significant.
(4)
Results
As expected, the number of skip detections varied across treatments and methods. The manual method detected few skips at the higher plant stands; almost no skips were detected at the 100% and 75% stand treatments in all trials and sites (tables 1, 2, and 3). In contrast, the UAV-based method detected a larger number of skips in both treatments across all trials and sites.
A statistical summary of p-values obtained from a t-test is shown in table 4. The paired t-test across all treatments combined found the mean differences of both methods to be significant. This suggests that both methods differ in their measurements of skipped area. T-test results also indicated that methods disagreed in each of the mean differences across the 100%, 75%, and 50% treatments. Therefore, results suggest that post-emergence assessments that use the UAV-based method will most likely disagree in the mean of the skipped area with those obtained via the manual method. In contrast, no evidence was found against the null hypothesis in the 25% treatment, suggesting that methods could have been used interchangeably to evaluate extremely critical stands.
| Table 1. Mean skipped area percentage in Lewiston, NC during 2019. | ||||||
| Planting Trial | Treatment | Method | Mean Difference (Manual – UAV) | |||
| Manual (SD) | UAV (SD) | |||||
| Early | 100 | 0.4 (0.4) | 1.9 (0.6) | - 1.5 | ||
| 75 | 1.0 (1.0) | 3.2 (3.3) | -2.2 | |||
| 50 | 1.7 (1.7) | 4.6 (3.0) | -2.9 | |||
| 25 | 33.9 (9.5) | 18.9 (10.4) | 15.0 | |||
| Late | 100 | 0.0 (0.0) | 0.1 (0.2) | -0.1 | ||
| 75 | 0.0 (0.0) | 2.0 (3.4) | -2.0 | |||
| 50 | 3.4 (0.9) | 0.1 (0.2) | 3.3 | |||
| 25 | 35.2 (6.8) | 6.5 (1.5) | 28.7 | |||
Table 2. Mean skipped area percentage in Rocky Mount, NC during 2019. | ||||||
| Planting Trial | Treatment | Method | Mean Difference (Manual – UAV) | |||
| Manual (SD) | UAV (SD) | |||||
| Early | 100 | 0.0 (0.0) | 3.7 (3.0) | -3.7 | ||
| 75 | 1.4 (1.5) | 7.9 (4.0) | -6.5 | |||
| 50 | 0.8 (1.3) | 19.4 (2.3) | -18.6 | |||
| 25 | 34.0 (7.5) | 59.4 (6.5) | -25.4 | |||
| Late | 100 | 0.0 (0.0) | 0.5 (0.6) | -0.5 | ||
| 75 | 0.0 (0.0) | 2.9 (1.9) | -2.9 | |||
| 50 | 2.2 (1.4) | 10.5 (5.3) | -8.3 | |||
| 25 | 35.4 (2.2) | 44.9 (5.7) | -9.5 | |||
Table 3. Mean skipped area percentage in Plymouth NC, during 2019. | ||||||
| Planting Trial | Treatment | Method | Mean Difference (Manual – UAV) | |||
| Manual (SD) | UAV (SD) | |||||
| Early | 100 | 0.8 (1.4) | 1.4 (1.6) | -0.6 | ||
| 75 | 4.1 (3.4) | 6.2 (2.6) | -2.1 | |||
| 50 | 9.5 (7.7) | 17.4 (5.4) | -7.9 | |||
| 25 | 66.3 (6.9) | 55.7 (6.3) | 10.6 | |||
| Late | 100 | 0.0 (0.0) | 4.0 (2.2) | -4.0 | ||
| 75 | 1.0 (1.0) | 15.9 (8.2) | -14.9 | |||
| 50 | 5.4 (4.7) | 41.7 (6.4) | -36.3 | |||
| 25 | 36.5 (9.2) | 77.7 (4.1) | -41.2 | |||
Table 4. T-test p-values between the UAV-based method and manual method across individual treatments and combined treatments. | ||||||
| Treatment | P-value | |||||
| Combined treatments | >0.001[a] | |||||
| 100% | 0.002[a] | |||||
| 75% | >0.001[a] | |||||
| 50% | >0.001[a] | |||||
| 25% | 0.101 | |||||
[a]Statistically significant at an alpha level 0f 0.05 | ||||||
The Bland-Altman plot comparing methods across all treatments combined is shown in figure 4. The plot revealed a mean difference of -7.65 and a range of agreement between -40.94 and 25.62. However, the 95% CI of the mean difference did not capture the line of equality shown in green (fig. 4). This means that the methods were significantly different in the mean of their differences and mirrors the results obtained in table 5. The V-shaped scatter plot indicated agreement in the random relative error between methods and
suggested that larger skip measurements will contain larger measurement errors.
 |
A cluster of points belonging to the 100%, 75%, and 50% was distinguished in the left-most portion of figure 4. Figure 5 shows the Bland-Altman test for the combined 100%, 75%, and 50% treatments, excluding the 25% treatment. Like figure 4, the 95% CI of the mean difference (-6.81) did not capture the line of equality. A negative trend was observed as the mean increased in magnitude, which suggests a proportional constant error between both methods and clear statistical evidence of a disagreement. This was expected, considering that the UAV-based method detected a larger number of skips than the manual method.
A Bland-Altman plot was generated for each individual treatment, but only the plot for the 25% treatment was included in this study. A mean difference of -10.10 was observed when comparing both methods’ performance in this treatment (fig. 6). This time, the 95% CI (-20.24, 0.03) of the mean difference captured the line of equality and provided evidence of agreement between methods. No outliers were observed. All other plots with individual treatments did not capture the line of equality and evidenced disagreement between the methods.
Table 5 summarizes the Bland-Altman bias estimates and the 95% CI of the mean differences for each of our treatments and treatment combinations. Only at the 25% treatment did the 95% CI capture the zero-mean bias. In addition, no other treatment or combination of treatments captured the line of equality. This indicated that the data from each of these tests were not consistent with equal population means between the UAV-based and manual methods.
Discussion
The main takeaway from this study is that in the presence of extremely poor stands (25% stand), both the UAV-based and manual methods agreed in their estimates of the skipped area percentage. The Bland-Altman method allowed for a comparison between methods that have different sampling sizes and clarified where results from each method could be used interchangeably. Results from this study also suggested that mediocre-to-good stands would generate poor agreement between both methods. This raises a question: which method is most appropriate for quantifying skips in fields where stand losses are not clearly visible? It can be argued that if a stand is not visibly suboptimal, then there is no need to perform manual or remote sensing assessments, and consequently no need to replant. But if for any reason there is a need to evaluate a visually mediocre cotton stand, then a UAV-based method would provide more coverage and hence detect more skips in the assessment. Growers must also understand that there are limitations to UAV-based assessments, like the timing of detection after emergence, detection accuracy based on flight altitude, and unfavorable weather and lighting conditions. In any case, farmers have been manually counting skips for decades to assess the severity of a suboptimal stand, and manual methods like those proposed by Jost et al. (2006) are still accepted in industry and academia.
 |
| Figure 5. Bland-Altman plot comparison of UAV-based and manual methods at 100%, 75%, and 50% treatment levels. |
 |
| Figure 6. Bland-Altman plot comparison of UAV-based and manual methods at 25% treatment levels. |
Acknowledgments
Thanks to Dr. Ed Barnes and Cotton Inc. for funding this project. Thanks to the North Carolina Cotton Producers Association, NC State Extension, Dr. Clyde Bogle, Churchill Hodges, and Tommy Corbet for providing and maintaining a space to conduct this research. Finally, thanks to each Advanced Ag Lab member from Weaver labs at NC State who helped during lab work and on-site work. This work is supported by the USDA National Institute of Food and Agriculture, Hatch projects 1017485 and 1016068.
References
Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Stat. Methods Med. Res., 8(2), 135-160. https://doi.org/10.1177/096228029900800204
Butler, S. (2019). Making the replant decision: Utilization of an aerial platform to guide replant decisions in Tennessee cotton. PhD diss. University of Tennessee. https://trace.tennessee.edu/utk_graddiss/5959
Chen, R., Chu, T., Landivar, J. A., Yang, C., & Maeda, M. M. (2018). Monitoring cotton (Gossypium hirsutum L.) germination using ultrahigh-resolution UAS images. Precis. Agric., 19(1), 161-177. https://doi.org/10.1007/s11119-017-9508-7
Collins, G., & Edmisten, K. (2016). Seedling vigor: Why is it important? NC State Extension. Retrieved from https://go.ncsu.edu/readext?404840
Collins, G., & Edmisten, K. (2018). Re-planting decisions and managing late planted cotton. NC State Extension. Retrieved from https://go.ncsu.edu/readext?529285
Craig, C. (2010). W073 - Making cotton replant decisions. University of Tennessee Agricultural Extension Service. Retrieved from http://trace.tennessee.edu/utk_agexcrop/86/
Datta, D., & Love, J. (2018). Deepankardatta/blandr: A Bland-Altman method comparison package for R (v.0.5.1). Zenodo. https://doi.org/10.5281/zenodo.1244740
Feng, A., Sudduth, K. A., Vories, E. D., & Zhou, J. (2019). Evaluation of cotton stand count using UAV-based hyperspectral imagery. Paper No. 1900807. Proc. 2019 ASABE Annual International Meeting. ASABE. https://doi.org/10.13031/aim.201900807
Giavarina, D. (2015). Understanding Bland Altman analysis. Biochemia Medica, 25(2), 141-151. https://doi.org/10.11613/BM.2015.015
Jost, P. H., Stewart, A. M., & Siebert, J. (2006). Skippy stands and replanting: Development of a decision-aid. Proc. 2006 Beltwide Cotton Conf., (pp. 1856-1858). Retrieved from https://www.cotton.org/beltwide/proceedings/2005-2022/data/conferences/2006/pdfs/1856-1858.pdf
Pettigrew, W. T., & Johnson, J. T. (2005). Effects of different seeding rates and plant growth regulators on early-planted cotton. J. Cotton Sci., 9(4), 9805-9809. Retrieved from https://pubag.nal.usda.gov/catalog/1799
Pettigrew, W. T., & Meredith, W. R. Jr. (2009). Seed quality and planting date effects on cotton lint yield, yield components, and fiber quality. J. Cotton Sci., 13(2), 37-47. Retrieved from https://pubag.nal.usda.gov/catalog/32317
PrecisionHawk. (2019). Automated plant counts with drones for precision agriculture. PrecisionHawk. Retrieved from https://www.precisionhawk.com/blog/media/topic/understanding-your-aerial-data-plant-counting