ASABE Logo

Article Request Page ASABE Journal Article

Deploying Machine Learning Methods to Predict Global Trade Patterns: The Case of Beef

Sei Jeong1, Munisamy Gopinath1,*, Ajay Kulkarni2, Feras Batarseh2,3

Published in Journal of the ASABE 67(1): 219-232 (doi: 10.13031/ja.15619). Copyright 2024 American Society of Agricultural and Biological Engineers.

1Agricultural and Applied Economics, University of Georgia, Athens, Georgia, USA.

2Commonwealth Cyber Initiative, Virginia Tech, Arlington, Virginia, USA.

3Biological Systems Engineering, Virginia Tech, Blacksburg, Virginia, USA.

*Correspondence: m.gopinath@uga.edu

Submitted for review on 3 April 2023 as manuscript number ITSC 15619; approved for publication as a Research Article and as part of the Cyberbiosecurity: Securing Water and Agricultural Systems Collection by Community Editor Dr. Seung-Chul Yoon of the Information Technology, Sensors, & Control Systems Community of ASABE on 20 October 2023.

Highlights

Abstract. In international economics, there has been a steady stream of innovations to explain patterns of trade between and among countries with emerging techniques. The most recent - Poisson Pseudo Maximum Likelihood (PPML) estimator – corrects for a potential bias caused by the large proportion of zero observations in bilateral trade data. Alternatively, this study offers Machine Learning (ML) as an option, especially in the presence of finer data on bilateral trade patterns. Using monthly and HS-6-digit (product) level data, the study finds that the main advantage of PPML is its accuracy of forecasts in-sample, but feature selection is somewhat rigid due to the inclusion of a large number of pair-wise fixed effects. ML models have the advantage in selecting features when a long list of explanatory variables is to be considered. Model validation statistics such as MAE and RMSE favor ML methods, but PPML tends to yield higher goodness of fit. In the out-of-sample context, ML has better accuracy than PPML, and a one-step walk-forward ML approach further improves the accuracy of ML forecasts. While PPML has a rich research and application history, emerging ML techniques have sufficient room for improvement in their adaptation to economic analysis.

Keywords. Beef, Bilateral Trade, Boosting, Machine learning, ML for Policy

Cross-border trade and its dramatic growth in the past two decades have been critical to economic well-being in developed and developing countries (Arkolakis et al., 2012; Bekkers et al., 2023). In particular, expanding international trade in agricultural commodities, especially meat products such as beef, pork, and chicken, has played a major role in serving the food and nutritional (protein) needs of the global populace (Beckman et al., 2017). Given the increasingly globalized agricultural markets, changes in the production or consumption of agricultural commodities in one place can now drastically affect other economies, even if they are not directly linked to trade (Schierhorn et al., 2016). Therefore, the relationship between trade flows and economic factors has become a key research priority for economists in recent years.

Tinbergen (1962) advanced one of the first empirical frameworks to relate bilateral trade to economic factors, i.e., the gravity model, where the magnitude of trade flows between any two countries is directly (inversely) proportional to their combined economic scale (distance between them). Anderson (1979) and Bergstrand (1985) provide theoretical foundations for the gravity model, augmenting the negative distance effect to include other trade costs such as tariffs and logistics. Eaton and Kortum (2002) further advanced the theory behind gravity models by using a Ricardian model of heterogeneous firms in each country, while Anderson and van Wincoop (2003) developed an endowment-economy model along the lines of Anderson (1979). Most recently, Allen et al. (2020) provided a universal framework for understanding the general equilibrium forces in gravity trade models.

Parallel to the advances in the theory of gravity models of bilateral trade is innovative empirical approaches to quantify the underlying economic relationship. A key challenge in empirics has been the fact that many countries do not trade with many others, i.e., a large number of zeros in observed data. Silva and Tenreyro (2006) proposed the Poisson Pseudo Maximum Likelihood (PPML) to deal with a potential bias arising from zero observations in bilateral trade flows data, the left-hand-side of the gravity equation. The PPML model is robust to different patterns of random error structure and provides a natural way to address sample selection issues arising from the omission of zero trade flows (Silva and Tenreyro, 2006; Yotov et al., 2016; Grant et al., 2018). Following that econometric advance, gravity models have sharply focused on the effects of distance (Melitz, 2007; Anderson, 2014), migration (Ghatak et al., 2009; Figueiredo et al., 2020), currency unions (De Sousa, 2012; Glick and Rose, 2016; Larch et al., 2019), trade facilitation, and related policies (Kepaptsoglou et al., 2010; Felipe and Kumar, 2012; Kuik et al., 2019; Lopez, 2020).

The PPML-based gravity model estimation remains popular, but has recently faced some challenges (Varian, 2014). For instance, Figueiredo et al. (2020) indicated that with product level data (Harmonized System, HS, 6-digits), conventional techniques, including the PPML estimator, face convergence issues due to the large number of estimated coefficients. Martin (2020), using Monte Carlo simulations, demonstrated the biases inherent in PPML and related techniques. Most applications of PPML are limited to three or four-digit HS codes, partly due to the fact that disaggregated data, i.e., more zero observations, create convergence and collinearity challenges. In addition, imposing pair-wise fixed effects, often in three dimensions (exporter, importer, and time), likely exacerbates the collinearity problem in PPML estimation.

With advances in data availability, e.g., monthly trade data has become available more recently, there is an opportunity now for recent computational advances, such as Machine Learning (ML) together with big data technologies and high-performance computing, to unravel, quantify, and understand economic relationships (Mullainathan and Spiess, 2017; Liakos et al., 2018; Gopinath et al., 2021). One of the main advantages of the ML techniques, often categorized into three main groups: supervised learning, unsupervised learning, and reinforcement learning, is that they are capable of autonomously solving large non-linear problems using large datasets from multiple sources (James et al., 2013; Chlingaryan et al., 2018; Liakos et al., 2018; Sharma et al., 2020).

Meat products account for about 17 percent of global food trade, estimated at over $1.6 trillion in 2020, with beef as the primary product in that category (Beckman et al., 2017; Food and Agriculture Organization, 2020). The meat trade, especially beef, has aided diet pattern changes and increased the nutritional content of diets in many developing countries (FAO, 2023). The US Department of Agriculture (USDA) routinely publishes trade forecasts every quarter to keep agribusiness companies informed of potential export or sourcing opportunities. For example, if the USDA saw some factors affecting Brazil’s soybean output (e.g., transport or policy issues), they would lower the projection of Brazil’s exports, providing information to global consumers of potential scarcity. The WTO also makes trade forecasts to help countries make sound trade policy decisions. Lower GDP forecasts for a region likely lower trade with that region, alerting potential buyers and sellers.Given the recent and substantial growth in meat trade, this study aims to compare ML methods to conventional techniques (PPML method) in terms of their predictive ability in alternative policy scenarios at the product (Harmonized System, HS-6 digit) level. Specifically, the PPML estimation with high-dimensional fixed effect (HDFE) is applied to allow for year and country-type (exporter or importer) effects and their interactions. In the case of ML, supervised learning, as in Gopinath et al. (2021) and its recent extensions, with classification and regression, is selected to compare to ML in the investigation of the relationship between trade flows and economic factors.

Conventional Gravity Model and Data

Poisson Pseudo Maximum Likelihood (PPML)

The basic gravity model is given by:

(1)

where yijt refers to the value of imports in US dollars of country i from j at time t and Xijt is a set of features varying by i, j and t dimensions. The dit, djt, and dij are referred to as pair-wise fixed effects, and in the case of equation 1 refer to importer-year, exporter-year, and importer-exporter fixed effects, respectively. In the gravity literature, the use of time-varying exporter and importer fixed effects account for multilateral resistance - the other barriers to trade between a pair of countries – implied by the theory (Anderson and van Wincoop, 2003; Feenstra, 2004; Baldwin and Taglioni, 2007). For example, while the U.S. and China may be engaged in a trade war by raising tariffs on each other, individually they may simultaneously lower barriers towards others, e.g., Vietnam or India. Traditional estimation techniques have shown that the inclusion of time-invariant pair-wise fixed effects (exporter-importer) not only accounts for unobservable frictions to trade but also addresses the endogeneity (simultaneity) of trade policy, as in the example above (Baier and Bergstrand, 2007). For the purposes of this study, PPML-HDFE is applied to allow for all three types of pair-wise fixed effects and interactions, and the standard errors are clustered on the pair of exporter and importer countries.

Data Governance and Features

The dependent variable, monthly bilateral import data of beef disaggregated at the HS 6-digit level from January 1990 to December 2020, was obtained from the Global Trade Atlas (GTA). International bilateral trade flow data were accessed with a subscription to GTA: https://www.gtis.com/gta/. There are six beef commodities at the HS-6 level, including fresh or chilled bovine meat – 020110 (Bovine carcasses and half carcasses), 020120 (Bovine cuts bone in), 020130 (Bovine cuts boneless)– and frozen bovine meat – 020210 (Bovine carcasses and half carcasses), 020220 (Bovine cuts bone in), 020230 (Bovine cuts boneless). The dynamic gravity data collected by Gurevich and Herman (2018) is the main source for the gravity variables. The data is available from 1990 to 2019, but current gross domestic product (GDP) obtained from the World Development Indicators replaced the GDP data from the source due to its absence after 2015 (https://databank.worldbank.org/source/world-development-indicators). The GDP is in nominal rather than real terms to proxy economic size (Shepherd, 2013; Gurevich and Herman, 2018; Kuik et al., 2019). Most of the gravity data are time invariant variables, and only annual GDP is available for many countries, and so the monthly gravity data were derived using the annual data repeatedly. Overall, monthly data from January 1990 to December 2019 is the time dimension of the data used for the analysis.

The tariff rate was the first feature considered, given the recent trade wars with reciprocal tariffs. The tariff data was obtained from a United Nations database, TRAINS, for the time period 1989-2014, and recent tariff data after 2014 was obtained from the Market Access Map (MAM) (https://trainsonline.unctad.org/home and https://www.macmap.org/). Missing data of the tariff rate was replaced by using several methods, including using the data from the observable preceding period and the importer countries’ average tariff rate toward the rest of the world. For the data of each EU country, each country’s EU entry year was considered to solve the missing data issue.

With the help of previous studies, other features or explanatory variables for the international trade flow of beef commodities are considered. For instance, distance between capital cities, common language, and contiguity, e.g., sharing a border, are introduced in many gravity studies (Anderson, 2014; Kuik et al., 2019; Conconi et al., 2020; Figueriredo et al., 2020; Lopez, 2020). Furthermore, trade policies such as customs unions (CU) and free trade agreements (FTA) are also considered since they help trade facilitation.

As noted earlier, the PPML HDFE model using disaggregated data encountered multicollinearity problems requiring some transformations of the features. For example, the summation of the GDP of the two trading partners had to be used to replace the GDP of each exporter and importer countries in the final estimation. The north-south distance between trading partners, defined as NSij = abs(Latitudei – Latitudej) in Anderson’s (2014) study, replaced the latitude of each country, again due to collinearity issues. The north-south distance potentially considers different life cycles of beef production between the Southern and Northern Hemispheres. In addition, inclusion of the difference in longitude between importer and exporter countries, population, and importer-exporter fixed effects produced results suggesting severe multicollinearity (lack of significance due to high variance inflation factors).

Machine Learning Techniques for the Gravity Model

ML methods involve a learning process with the objective of using experience to perform a task. For example, upon training based on past experience, a model can be used to classify, predict, or cluster new examples. An advantage of ML methods for identifying patterns and relationships is that their performance on a new task or forecast can be progressively improved (James et al., 2013; Liakos et al., 2018; Mohri et al., 2018).

Usually, ML methods are categorized into three main types: supervised learning, unsupervised learning, and reinforcement learning (James et al., 2013; Sharma et al., 2020). In supervised learning, data are presented with example inputs and corresponding outputs with the objective of constructing a relationship between inputs and outputs. A predictive model is developed using the example data with prior knowledge of the input and the corresponding output variables during training. Then, future values are predicted by applying the same knowledge to new input data. In other words, the relationship between trade and features or explanatory variables is derived from a “training” data sample and then applied to a “test” data sample to compute predictions through the supervised (trained) ML models. In the context of this study, bilateral trade data and features for a part of the sample (1990-2010) can be used to train a model, to which features during 2011-19 can be fed. Then, actual bilateral trade data can be compared with the predictions from the supervised ML model. 

Classification and regression are the main two subcategories of the supervised learning. Unlike unsupervised learning, which does not have pre-defined response variables with unlabeled datasets, in supervised learning, the dataset is labeled (Jordan and Mitchell, 2015; Traore et al., 2017; Liakos et al., 2018; Gopinath et al., 2021). Decision trees are classification or regression models formulated in a tree-like architecture. They are segmented into branches (splits) and leaves (nodes). Each leaf node contains a simple regression model and represents the final decision or prediction taken after following the path from root to leaf (Belson, 1959; Rodriguez-Galiano et al., 2015; Ahmad et al., 2018). In this procedure, each training sample of the decision tree is dependent on previous trees, and they are chosen sequentially.

Boosting algorithms further help in refining the supervised learning process and progressively improving performance on new tasks or forecasts. When boosting techniques are applied, they collect several weak prediction models in order to further improve the classification. Following Bruce et al. (2020), a brief overview of the boosting algorithm is provided below:

  1. Initialize the maximum number of models to fit (M), iteration counter (m = 1), and observation weights: wi = 1 / N for i = 1,2,…,N. Also, initialize the ensemble model F^0 = 0.
  2. Train a model (f^m) using observation weights w1,…,wn to minimize the error (em).
  3. Add the model to the ensemble: F^m = F^m–1 + ??mf^m where ??m = log(1 – em) / em.
  4. Update the weights for the misclassified observations and increment the model counter by 1, i.e., m = m + 1.
  5. Go to step 2 and repeat these steps until m = M.

Additionally, boosting trees are trained using data and labels, which helps the models capture patterns or relationships in the data more effectively than conventional models (Gopinath et al., 2020). This gives AI approaches an advantage, especially when providing anomaly detections or during outlier events from the data (Patcha and Park, 2007). Another advantage of boosted trees is that the data points producing high errors in the predictions get assigned higher weights in the next iteration of model development. Thus, these weights can point towards the records with higher error contributions in the prediction; providing information on the model’s decision-making reasoning, i.e., furthering explainability – an important pillar of assuring AI systems (Batarseh et al., 2021). LightGBM and XGBoost are commonplace boosting algorithms for their efficiency and ability to improve overall performance. The key difference between LightGBM and XGBoost is that XGBoost splits the tree nodes one level at a time (level-wise), while LightGBM splits leaf-wise. The leaf-wise algorithm can reduce information loss relative to the level-wise algorithm and hence can yield better accuracy (Khandelwal, 2017; Ke et al., 2017; Gopinath et al., 2021). Extremely randomized trees (extra trees) algorithm is a type of ensemble learning technique, developed as an extension of random forest algorithm (Geurts et al., 2006). It is similar with random forest in that it uses a random subset of features to train each base estimator. The difference between extra trees and random forest is that the best feature along with the corresponding value for splitting the node is randomly selected in extra tree, and the extra trees converges faster than random forests. The entire training dataset is used to train each regression tree in extra trees, while the random forest uses a bootstrap replica to train the model (John et al., 2015; Ahmad et al., 2018).

Table 1. PPML estimation of the gravity model, providing estimates of parameters from the most common approach to modeling bilateral trade flows (PPML applied to the gravity model), including model validation metrics.[a],[b],[c]
020110020120020130020210020220020230
ln(GDP)1.05+0.54+0.76**1.30***0.80**0.44
(0.60)(0.30)(0.25)(0.34)(0.30)(0.34)
ln(Distance)-0.30-1.47***-1.67***-1.58***-1.45***-0.83**
(0.33)(0.20)(0.17)(0.28)(0.30)(0.32)
ln(1 + (Tariffij / 100))-1.99*0.44-2.10**-1.84+-1.42+-1.91***
(0.83)(1.27)(0.77)(1.03)(0.86)(0.40)
Contiguity0.98**0.150.26-0.310.51+-0.15
(0.31)(0.21)(0.33)(0.31)(0.30)(0.35)
NS-0.05-0.020.02**0.03+-0.010.01
(0.04)(0.02)(0.01)(0.02)(0.01)(0.01)
Common language0.86*0.73**0.462.55***1.78***0.08
(0.39)(0.23)(0.37)(0.28)(0.30)(0.29)
CU7.91***3.87***1.78**7.48***4.35***2.01***
(1.22)(0.33)(0.60)(0.68)(0.59)(0.56)
FTA5.12***1.94***1.08***0.550.82*0.86*
(0.76)(0.33)(0.26)(0.49)(0.36)(0.42)
Constant-24.864.8450.18***-13.172.409.66
(19.20)(8.83)(7.27)(8.87)(9.12)(10.91)
N406,728781,8961,130,701330,540957,9481,563,096
Pseudo R20.900.930.890.910.900.82
MAE48,26539,802113,8424,85113,113147,196
RMSE379,549340,724943,11857,801186,7751,666,063
RMAE635061506599
RRMSE497.88424.25504.73594.46919.941,123.04

    [a]Standard errors in parentheses.

    [b]"+ p<0.10 * p<0.05 ** p<0.01 *** p<0.001".

    [c]The lower the scores of MAE, RMSE, RMAE, and RRMSE, the better the performance.

PPML and ML Results

PPML Estimation and Results

The empirical PPML estimation model, after addressing many multicollinearity issues noted earlier, contained eight features as follows:

(2)

Three fixed effects such as importer-year (dit), exporter-year (djt), and year-month (dtt) are included. Even though the PPML is advantageous in naturally accounting for zero trade pairs in the data, adding 1 to the trade values minimizes the missing observations, then logarithms of both sides of the equation were taken (Figueriredo et al., 2020).

The PPML with HDFE estimation results of the gravity model are shown in table 1. The results presented here as well as in the next sub-section and the following discussions are based on monthly data and HS-6-digit products. Overall, the PPML estimation of beef trade models show that R2 – a measure of the goodness of fit of a model, i.e., how well the regression line approximates the actual data – scores are higher than 0.8. Results in table 1 are comparable, both in significance and signs, to previous literature in most cases (Yotov et al., 2016). GDP, common language, CU, and FTA have significant and positive coefficients for most of the six beef commodities. The positive effects of GDP have already been foreseen in the fact that the meat market has grown significantly in many countries with high economic growth, as GDP provides economic snapshots of the country. The positive coefficients of FTA and CU agreements suggest the important role of trade facilitation in the case of the beef market (Moďsé and Sorescu, 2013). As Melitz (2007) and Anderson (2014) find, there is a possibility that the north-south distance can have a significant and positive impact on trade flows. For example, the coefficients of the north-south distance (NS) for commodities 020130 (Fresh or chilled bovine cuts boneless) and 020210 (Frozen carcasses and half-carcasses) are positive and significant. The positive coefficients suggest that the difference in life cycle between the northern and southern hemispheres due to the opposition of seasons significantly encourages international trade.

Distance and tariff rate have a significant and negative relationship with trade, as expected. The importer i’s tariff rate applied on imports from j has negative effects consistent with the previous literature (Fontagné et al., 2018; Figueiredo et al., 2020). Also, the impact of contiguity, i.e., sharing a common land border, on trade is positive and significant at 10% for some commodities – 020110 (Fresh or chilled bovine carcasses and half carcasses) and 020220 (Frozen bovine cuts bone in) (Gurevich and Herman, 2018; Figueiredo et al., 2020; Lopez, 2020). The sparse evidence for contiguity across the six HS codes can be attributed to observed data showing that most trading partners with the highest cumulative trade values (over the sample period) do not share a border.

Supervised ML Results

Table 2 presents the performances of the three supervised ML models measured by R2, Mean Absolute Error (MAE), and Root Mean Square Error (RMSE), and the sizes of training and test data sample for each of the six commodities. ML models with the highest R2 also often have lower MAE or RMAE scores. Based on the R2 scores, LightGBM has the best performance for one commodity (020130), while XGBoost has the best performance for two commodities (020220 and 020230), and extra tree regression fits best for the remaining three commodities (020110, 020120, and 020210).

Generally, the R2 is expected to be in the 0 to 1 range, with higher values indicating a better fit. However, from table 2, all ML methods for 020210 yield a negative R2. These validation statistics are to be viewed in the training versus test data contexts. While using training data, as shown in Appendix table A2, ML performance metrics including R2 are relatively high, but that may not translate into similar metrics when applying to test data (Tan, 2022).

Table 2. Performance scores of the supervised ML analysis, presenting model validation metrics from the application of supervised ML techniques to the gravity model.[a]
020110020120020130020210020220020230
LightGBMR20.140.400.56-2.040.100.31
MAE4,6367,40536,3361,0774,81853,091
RMSE124,492208,799633,19644,840216,1481,077,100
RMAE165120138420152151
RRMSE4,4333,3842,40217,5016,8123,065
XGBoostR20.230.370.06-1.850.490.31
MAE4,5948,69643,8729523,93950,814
RMSE118,091213,516928,53943,366162,3691,072,920
RMAE164141166372124145
RRMSE4,2053,4603,52216,9255,1173,053
Extra trees
Regression		R20.530.480.48-1.430.170.28
MAE2,6825,63830,1385653,87644,584
RMSE92,305192,787691,23740,082208,4141,098,985
RMAE9591114220122127
RRMSE3,2873,1242,62215,6446,5683,127
ObservationsTrain6,316,7886,316,7886,316,7886,316,7886,316,7886,316,788
Test3,833,6283,833,6283,833,5693,833,5563,833,5083,833,508

    [a]The lower the scores of MAE, RMSE, RMAE, and RRMSE, the better the performance.

Table 3 presents feature importance scores from the explanatory variables in the best fitted models. The values of information gain were normalized based on the highest score values, meaning that 100 indicates the variable with the highest feature importance score, and other values are relative information gain values compared to the highest value. The ranking of each variable is in parentheses. GDP and Tariff rate are the top information provider for two and three commodities, respectively. The relative importance shows that distance related explanatory – Distance and North-South Distance – also have a great influence on the learning of trade flows. On the other hand, trade policy facilitation such as customs union and FTA provide relatively lower feature importance scores.

In both PPML and ML analyses, GDP and tariffs significantly affect cross-border beef trade flows. The main differences between the PPML estimation and the supervised ML models are that the estimated coefficients of FTA and CU in the former were significant for most of the commodities and were much higher than that of north-south distance, while ML models suggest that north-south distance is relatively more important in learning the trade flows of beef commodities.

Comparison: PPML vs. ML

In this section, several criteria are introduced to evaluate the traditional PPML and new ML techniques to identify the pros and cons of the two approaches. Easy-to-see first is statistical criteria or performance scores. As seen in tables 1 and 3, four criteria, including R2, MAE, RMSE, RMAE, and RRMSE, were calculated. It is seen that PPML has higher R2 scores for all six commodities when using HS6 level data. However, when comparing other performance scores, such as MAE and RMSE, ML analysis shows better performance scores than PPML analysis. In addition, it is observed that the estimation results of the PPML have a lot of missing observations. For example, for 020110, the ML analysis uses 3,833,628 test samples, while the observation number of the PPML analysis is only 406,728. Many observations are dropped because they belong to fixed effect groups for which the dependent variable (value of imports) is always zero. For example, as explained above, importer-exporter fixed effects were added to the estimation. If only zero values are observed for the entire sample for a particular trade pair, then it is not possible to include these observations as the fixed effects that correspond to them are not defined (singletons).

Table 3. Relative importance of supervised ML features and ranking, providing the ranking of features based on information gains in the application of supervised ML techniques to the gravity model.[a],[b],[c]
020110020120020130020210020220020230
GDP100 (1)100 (1)30.35 (4)87.87 (2)28.33 (4)13.93 (5)
Distance98.75 (2)54.79 (2)100(1)71.59 (3)54.01 (2)79.30 (2)
Tariff80.02 (4)51.83 (4)71.63 (2)100 (1)100 (1)100 (1)
Contiguity7.20 (7)10.39 (6)4.91 (7)0.89 (8)1.91 (6)3.06 (6)
Common language9.80 (5)19.30 (5)8.24 (5)2.34 (6)7.31 (5)23.6 (4)
CU8.23 (6)9.11 (7)1.00 (8)1.06 (7)1.33 (7)2.93 (7)
FTA6.11 (8)7.09 (8)4.95 (6)5.31 (5)0.48 (8)2.26 (8)
NS91.12 (3)53.16 (3)32.01 (3)64.27 (4)48.84 (3)55.85 (3)

    [a]100 refers to a feature with the highest importance score.

    [b]The values are recalculated based on the highest value.

    [c]The ranking of each variable is in parentheses.

Table 4. Estimation results and performance scores of the PPML estimation using 1990-2010 data. The results are referred to as PPML2 for comparison purposes.[a],[b],[c]
020110020120020130020210020220020230
ln(GDP)1.21+0.65+0.88***0.96**0.41+-0.03
(0.68)(0.34)(0.21)(0.37)(0.23)(0.32)
ln(Distance)-0.26-1.26***-1.88***-1.31***-1.39***-0.96**
(0.34)(0.23)(0.21)(0.31)(0.27)(0.30)
ln(1 + (Tariffij / 100))-1.83*0.03-1.71*-2.160.63-1.57***
(0.81)(1.22)(0.85)(1.44)(0.66)(0.30)
Contiguity0.99**0.170.130.150.52+0.08
(0.31)(0.22)(0.35)(0.36)(0.28)(0.25)
NS-0.05-0.040.03**-0.04*-0.02*0.01
(0.04)(0.03)(0.01)(0.02)(0.01)(0.01)
Common language0.81+0.72*0.431.96***1.91***0.46
(0.42)(0.24)(0.38)(0.32)(0.36)(0.30)
CU7.63***3.40***1.33+6.86***5.00***1.82**
(0.90)(0.44)(0.79)(0.86)(0.72)(0.57)
FTA4.38***2.78***1.46***0.900.000.48
(0.78)(0.42)(0.37)(0.84)(0.39)(0.55)
Constant-28.50-0.2854.85***-4.8735.82***24.26*
(21.03)(9.97)(6.67)(9.45)(6.98)(9.94)
R20.040.430.270.020.480.26
MAE722,26329,80597,64918,95714,499102,754
RMSE45,874,356485,4512,512,565627,326469,4231,983,442
RMAE3,62386938959197
RRMSE230,1311,4002,40329,6162,9611,867

    [a]Standard errors in parentheses.

    [b]"+ p<0.10 * p<0.05 ** p<0.01 *** p<0.001".

    [c]The lower the scores of MAE, RMSE, RMAE, and RRMSE, the better the performance.

Since PPML and ML techniques employ markedly distinct approaches and employ different samples, the following models and their outcomes are compared:

Table 5. Performance scores of the supervised ML analysis one-step walk-forward validation, presenting model validation metrics from the application of one-step walk-forward ML techniques to the gravity model.[a]
020110020120020130020210020220020230
R20.570.530.74-1.100.740.55
MAE2,3765,29727,4025173,36344,602
RMSE88,525183,971485,75337,231117,599867,564
RMAE8586104202106127
RRMSE3,1522,9811,84314,5313,7062,469

    [a]The lower the scores of MAE, RMSE, RMAE, and RRMSE, the better the performance.

Observing predictions is the next step in evaluating the two methods for analyzing international beef trade flows. The primary interest is in comparing PPML2 with ML1 and ML2, but figure 1 (2) shows the aggregated trade values of each commodity from all countries observed as well as estimated by the two PPML (ML) models. Predicted import values of the PPML estimation (PPML1) track the actual import values very closely (fig. 1), while there are bigger differences between the predictions of the supervised ML and the actual values (fig. 2). The gravity equation has been found to have very high predictive power (Yotov et al., 2016). It delivers a very strong fit and plausible estimates on the standard set of gravity variables (Martin, 2020). Nevertheless, assuming that the out-of-sample PPML estimation (PPML2) follows a comparable approach to the ML1 analysis, it appears that the ML predictions better track actual import values (fig. 2). Furthermore, the machine learning predictions with walk-forward validation (ML2) slightly outperform those without validation (ML1) in tracking actual imports. These visual representations are affirmed by forecast errors for all four models plotted in figure 3. As can be inferred from figure 1, the predictions of PPML utilizing all the time data exhibit forecast errors that are close to zero. ML1 and ML2 have forecast errors closer to PPML1, but PPML2 has the largest forecast error in all six commodities.

Figure 4 depicts forecasts for the leading trade pairs utilizing the four methods mentioned earlier. The leading trade partners for each beef commodity are as follows:

  1. 020110 (fresh or chilled bovine carcasses and half carcasses): Netherlands – Italy.
  2. 020120 (fresh or chilled bovine cuts, bone in): France – Italy.
  3. 020130 (fresh or chilled bovine cuts, boneless): Australia – Japan.
  4. 020210 (frozen bovine carcasses and half carcasses): Ukraine – Russia.
  5. 020220 (frozen bovine cuts, bone in): United States – South Korea.
  6. 020230 (frozen bovine cuts, boneless): Australia – United States.

It is also seen that the predictions from the PPML estimation (PPML1) are closer to the actual values of imports overall. For four out of the six commodities modeled here, ML1 and ML2 provide fits better than PPML 2. Surprisingly, for the commodity 020110, the accuracy of prediction from the out-of-sample PPML estimation (PPML2) seems to be better than those of PPML1, ML1, and ML2. This contrasts with the outcomes observed for the entire commodity 020110, but this sub-sector has one of the lowest number of observations among the six commodities modeled here.

An added advantage to employing ML methods is the ability to conduct country level analysis by combining data on a chosen country. Since gravity models require a bilateral specification, understanding or predicting trade on a unilateral basis is tedious since the structure significantly changes by summation on one dimension. To demonstrate the flexibility of ML approaches, import data were aggregated across all six beef commodities for each country. Subsequently, the top 5 beef-exporting countries were selected for this analysis. Australia ranked first in terms of beef exports by country, followed by the United States, Brazil, the Netherlands, and New Zealand. The same explanatory variables as in table 3, but aggregated ones were incorporated into the model, and the cut-off year for the training data remained consistent, i.e., 2010, as before. The ML predictions plotted in figure 5 show a reasonably good fit of country-specific trade flows.

Conclusion

Agricultural trade encourages investment and promotes economic growth. For instance, agricultural exports generate 7,500 full-time, civilian jobs for every $1 billion in farm exports in the United States (USDA-ERS, 2018). Predicting trade flows in agricultural markets is critical in the context of economic growth in both developed and developing economies, e.g., decision makers received more accurate information for sourcing or selling. This study estimates bilateral trade patterns of beef using traditional methods and machine learning analysis for a comparison of their and prediction quality. The traditional PPML methods have been widely used to estimate trade flows. Machine Learning (ML), especially, three supervised ML models –LightGBM, XGBoost, and Extra tree regression– were implemented to compensate for the limitations of the PPML methods, such as feature selection and the multicollinearity problem, especially when using disaggregated trade data. It is clearly seen that the PPML estimation has a higher R2 score than the ML models, which implies that the PPML estimation fits better than the estimations of other ML models. As a result of the accurate modeling, its in-sample prediction closely mirrors the actual imports. The estimation results are also consistent with other previous literature results and theories. For example, GDP and trade facilitation such as CU and FTA have positive and significant effects on trade flows, while distance and tariff rates have negative effects on it. Elasticities and predictions can be easily calculated from the estimated values.

Figure 1. PPML estimation predictions from tables 1 and 4 for 2011-2019, along with actual imports during that period.

Meanwhile, several Machine Learning (ML) methods were employed with an eye towards out-of-sample predictions and the flexibility they offer in feature selection. One of the main advantages of the ML methods is that it can review a large volume of data and identify specific trends and patterns easily. As explained in appendix, 32 gravity variables were selected to analyze the beef trade flows in this study. Given lots of explanatory variables, it is difficult to select which variables can be considered as important among them. It is very important to choose meaningful and appropriate variables to establish an equation. As mentioned earlier, the PPML method has experienced severe multicollinearity issue when presented with 32 features. In addition, the estimation results vary greatly depending on which variables are included into the equation. The ML analysis, especially comparing the feature importance of the variables, not only greatly helped select the explanatory variables when estimating using conventional econometrics, but also allowed to examine the effects of each of the variables without collinearity issues. In addition, ML models have better performance scores when comparing MAE, RMSE, RMAE, and RRMSE. There is also a difference in the number of observations used by PPML and ML, with the former employing a significantly small share of available data due to the singleton issue.

Figure 2. Supervised ML prediction from the best ML methods for 2011-2019, along with actual imports during that period.

For more distinct comparisons between the PPML and ML, two more models are estimated: PPML with limited data (1990-201) for out-of-sample (2011-2020) predictions, referred to as PPML2, and ML with a one-step walk-forward approach (ML2). It is observed that ML2 has better predictive power in out-of-sample forecasting relative to PPML2. Furthermore, forecasts from the one-step walk-forward approach (ML2) were closer to the actual imports compared to those from the ML without the approach.

The gravity model using PPML methods has been used in many studies for a long time, and not surprisingly, estimation results show very high accuracy in predicting bilateral trade flows. ML techniques have recently come into the spotlight with their capability of autonomously solving large non-linear problems using datasets from multiple sources. The ML models are robust to alternate specifications, i.e., help select features of importance, employ all data, and offer routes to progressively uncover complex economic relationships. While PPML provides relatively accurate in-sample forecasts, this study offers the potential of ML providing better out-of-sample predictions, especially when the global economy has faced continual challenges, e.g., trade and military conflicts and pandemics. Since the gravity model was first introduced by Tinbergen (1962), it has gone through a lot of development and been used to reliably explain trade flows. Considering that the utilization of ML techniques in

Figure 3. Forecast errors from PPML1 and PPML2 for a comparison to those from ML1 and ML2 during 2011-2019.

trade analysis is relatively recent, there is ample potential for enhancing their applicability to economic contexts. The recent deglobalizing moves, e.g., the U.S.-China trade war, Brexit, and the COVID-19 pandemic, along with military conflicts (Russia-Ukraine), have created substantial uncertainties in global supply chains and trading patterns. Unlike the traditional approaches, the ML toolkit offers innovative approaches to studying uncertain trade patterns using outlier detection and classification, network analysis (graph neural and others), and causal inference (reinforcement learning) methods. For instance, causal learning, a growing form of unsupervised learning, can point to unconventional causal relations in big data sets and time series that span over different phases of trade patterns. New causal learning libraries, such as EconML and DoWhy, are especially tailored to trade contexts. Additionally, many forms of deep learning, such as recurrent and graph neural networks, can provide insights into economic features critical to a desired prediction and, most importantly, present explanations (i.e., XAI) to policy makers via methods such as Attention. ML and its growing library of methods are expected to be leveraged further in the coming years by decision-makers in the public and private domains.

Acknowledgment

Financial support from the Foreign Agricultural Service of the U.S. Department of Agriculture is gratefully acknowledged.

Figure 4. Predictions of top trade pair during 2011-19 when considering major partners only.
Figure 5. Supervised ML prediction of top five exporters during 2011-2019 along with actual imports when considering unilateral trade flows.

Appendix

Gurevich and Herman (2018) provided 70 gravity variables, but not all the variables covered the time period until 2019. In addition, including all the variables into the analysis model might cause several problems, including multicollinearity. Gopinath et al. (2021) add 35 predictors in the ML models and provide the feature importance of the variables. Based on the paper, 32 gravity variables were first selected to define the PPML estimation model after excluding some variables which had missing values after 2015. Appendix table A1 presents the top 10 information provider among the 32 features.

The following shows the list of the 32 variables: contiguity, distance, tariff, common language, CU, FTA, GDP of exporter, GDP of importer, population of exporter, population of importer, latitude of exporter, latitude of importer, longitude of exporter, longitude of importer, stock of exporter, stock of importer, landlocked dummy of exporter, landlocked dummy of exporter, island dummy of exporter, island dummy of importer, PTA_goods agreement, PTA_services agreement, EIA, PSA, EU dummy of exporter, EU dummy of importer, WTO dummy of exporter, WTO dummy of importer, GATT dummy of exporter, GATT dummy of importer, colonial dummy of exporter, colonial dummy of importer.

Table A1. Ranking of feature importance scores.
Rank020110020120020130020210020220020230
1GDP_DestinationPopulation_OriginLongitude_DestinationGDP_OriginStock_DestinationLatitude_Origin
2Population_OriginGDP_OriginDistanceDistanceLongitud_DestinationGDP_Origin
3GDP_OriginPopulation_DestinationLatitude_OriginLatitude_DestinationGDP_DestinationGDP_Destination
4Population_DestinationGDP_DestinationGDP_OriginCommon LanguageDistancePopulation_Origin
5Longitude_DestinationCommon LanguageLongitude_OriginGDP_DestinationPopulation_DestinationStock_Origin
6DistanceLatitude_DestinationPopulation_OriginPopulation_DestinationLatitude_DestinationPopulation_Destination
7Longitude_OriginDistanceStock_OriginIsland_OriginGDP_OriginDistance
8Latitude_OriginLatitude_OriginTariff rateTariff rateTariff rateLongitude_Origin
9Latitude_DestinationLongitude_DestinationPopulation_DestinationLandlocked_OriginPopulation_OriginStock_Destination
10Common LanguageLongitude_OriginGDP_DestinationLongitude_OriginStock_OriginIsland_Origin

Table A2. Performance scores of the supervised ML analysis (train data).[a]
020110020120020130020210020220020230
LightGBMR2 0.64 0.60 0.79 0.86 0.84 0.84
MAE 2,870 5,459 14,261 347 1,286 15,059
RMSE 99,190 155,514 351,191 19,322 40,009 274,160
RMAE 90 88 82 98 112 98
RRMSE 3,094 2,510 2,013 5,471 3,469 1,782
XGBoostR2 0.62 0.59 0.78 0.86 0.83 0.80
MAE 3,530 6,459 16,253 378 1,402 16,628
RMSE 101,320 157,904 363,828 19,636 41,047 302,167
RMAE 110 104 93 107 122 108
RRMSE 3,160 2,549 2,085 5,560 3,559 1,964
Extra trees
Regression		R2 0.64 0.61 0.826308192 0.83 0.86 0.86
MAE 2,486 4,815 10,112 240 838 9,347
RMSE 99,289 154,116 321,698 21,391 37,265 253,605
RMAE 78 78 58 68 73 61
RRMSE 3,097 2,487 1,844 6,057 3,231 1,649
ObservationsTrain 0.64 0.60 0.79 0.86 0.84 0.84
Test 2,870 5,459 14,261 347 1,286 15,059
[a]The lower the scores of MAE, RMSE, RMAE, and RRMSE, the better the performance.

References

Ahmad, M. W., Reynolds, J., & Rezgui, Y. (2018). Predictive modelling for solar thermal energy systems: A comparison of support vector regression, random forest, extra trees and regression trees. J. Cleaner Prod., 203, 810-821. doi:https://doi.org/10.1016/j.jclepro.2018.08.207

Allen, T., Arkolakis, C., & Takahashi, Y. (2020). Universal Gravity. J. Polit. Econ., 128(2), 393-433. doi:https://doi.org/10.1086/704385

Anderson, E. (2014). Time differences, communication and trade: Longitude matters II. Rev. World Econ., 150(2), 337-369. doi:https://doi.org/10.1007/s10290-013-0179-9

Anderson, J. E. (1979). A theoretical foundation for the gravity equation. Am. Econ. Rev., 69(1), 106-116. Retrieved from http://www.jstor.org/stable/1802501

Anderson, J. E., & van Wincoop, E. (2003). Gravity with gravitas: A solution to the border puzzle. Am. Econ. Rev., 93(1), 170-192. doi:https://doi.org/10.1257/000282803321455214

Arkolakis, C., Costinot, A., & Rodríguez-Clare, A. (2012). New trade models, same old gains? Am. Econ. Rev., 102(1), 94-130. doi:https://doi.org/10.1257/aer.102.1.94

Baier, S. L., & Bergstrand, J. H. (2007). Do free trade agreements actually increase members' international trade? J. Int. Econ., 71(1), 72-95. doi:https://doi.org/10.1016/j.jinteco.2006.02.005

Baldwin, R., & Taglioni, D. (2007). Trade effects of the Euro: A comparison of estimators. J. Econ. Integr., 22(4), 780-818. Retrieved from http://www.jstor.org/stable/23000919

Batarseh, F. A., Freeman, L., & Huang, C.-H. (2021). A survey on artificial intelligence assurance. J. Big Data, 8(1), 60. doi:https://doi.org/10.1186/s40537-021-00445-7

Beckman, J., Dyck, J., & Heerman, K. E. (2017). The global landscape of agricultural trade 1995-2014, EIB-181. USDA Economic Research Service.

Bekkers, E., Corong, E., Métivier, J., & Orlov, D. (2023). How will global trade patterns evolve in the long run? WTO Staff Working Paper No. ERSD-2023-03. Geneva, Switzerland: World Trade Organization.

Belson, W. A. (1959). Matching and prediction on the principle of biological classification. J. R. Stat. Soc. C Appl. Stat., 8(2), 65-75. doi:https://doi.org/10.2307/2985543

Bergstrand, J. H. (1985). The gravity equation in international trade: Some microeconomic foundations and empirical evidence. Rev. Econ. Stat., 67(3), 474-481. doi:https://doi.org/10.2307/1925976

Bruce, P., Bruce, A., & Gedeck, P. (2020). Practical statistics for data scientists: 50+ essential concepts using R and Python. O'Reilly Media.

Chlingaryan, A., Sukkarieh, S., & Whelan, B. (2018). Machine learning approaches for crop yield prediction and nitrogen status estimation in precision agriculture: A review. Comput. Electron. Agric., 151, 61-69. doi:https://doi.org/10.1016/j.compag.2018.05.012

Conconi, P., Magerman, G., & Plaku, A. (2020). The gravity of intermediate goods. Rev. Ind. Organ., 57(2), 223-243. doi:https://doi.org/10.1007/s11151-020-09762-2

de Sousa, J. (2012). The currency union effect on trade is decreasing over time. Econ. Lett., 117(3), 917-920. doi:https://doi.org/10.1016/j.econlet.2012.07.009

Eaton, J., & Kortum, S. (2002). Technology, geography, and trade. Econometrica, 70(5), 1741-1779. doi:https://doi.org/10.1111/1468-0262.00352

Fan, J., & Yao, Q. (2008). Nonlinear time series: Nonparametric and parametric methods. Berlin, Heidelberg, Germany: Springer Science & Business Media.

Feenstra, R. (2004). Advanced international trade: Theory and evidence. Princeton, NJ: Princeton University Press.

Felipe, J., & Kumar, U. (2012). The role of trade facilitation in central Asia: A gravity model. East. Eur. Econ., 50(4), 5-20. doi:https://doi.org/10.2753/EEE0012-8775500401

Figueiredo, E., Lima, L. R., & Orefice, G. (2020). Migration, trade and spillover effects. J. Comp. Econ., 48(2), 405-421. doi:https://doi.org/10.1016/j.jce.2019.10.005

Fontagné, L., Martin, P., & Orefice, G. (2018). The international elasticity puzzle is worse than you think. J. Int. Econ., 115, 115-129. doi:https://doi.org/10.1016/j.jinteco.2018.08.011

Food and Agriculture Organization. (2023). The state of food security and nutrition in the world. United Nations FAO. doi:https://doi.org/10.4060/cc3017en

Geurts, P., Ernst, D., & Wehenkel, L. (2006). Extremely randomized trees. Mach. Learn., 63(1), 3-42. doi:https://doi.org/10.1007/s10994-006-6226-1

Ghatak, S., Silaghi, M. I., & Daly, V. (2009). Trade and migration flows between some CEE countries and the UK. J. Int. Trade Econ. Dev., 18(1), 61-78. doi:https://doi.org/10.1080/09638190902757426

Glick, R., & Rose, A. K. (2016). Currency unions and trade: A post-EMU reassessment. Eur. Econ. Rev., 87, 78-91. doi:https://doi.org/10.1016/j.euroecorev.2016.03.010

Gopinath, M., Batarseh, F. A., & Beckman, J. (2020). Machine learning in gravity models: An application to agricultural trade. NBER Working Paper 27157. Cambridge, MA: National Bureau of Economic Research.

Gopinath, M., Batarseh, F. A., Beckman, J., Kulkarni, A., & Jeong, S. (2021). International agricultural trade forecasting using machine learning. Data Policy, 3, e1. doi:https://doi.org/10.1017/dap.2020.22

Grant, J. H., Ning, X., & Peterson, E. B. (2018). Trade elasticities and trade disputes: New evidence from tariffs and relative preference margins. Policy Report CAT-2018-07. Center for Agricultural Trade, College of Agriculture and Life Sciences, Virginia Tech.

Gurevich, T., & Herman, P. (2018). The dynamic gravity dataset: 1948-2016. USITC Office of Economics Working Paper 2018-02-A. U.S. International Trade Commission.

John, V., Liu, Z., Guo, C., Mita, S., & Kidono, K. (2016). Real-time lane estimation using deep features and extra trees regression. In T. Bräunl, B. McCane, M. Rivera, & X. Yu (Eds.), Image and video technology. PSIVT 2015. Lecture notes in computer science (pp. 721-733). Cham: Springer.

Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255-260. doi:https://doi.org/10.1126/science.aaa8415

Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., . . . Liu, T.-Y. (2017). Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems 30 (NIPS 2017) (pp. 3146-3154).

Kepaptsoglou, K., Karlaftis, M. G., & Tsamboulas, D. (2010). The gravity model specification for modeling international trade flows and free trade agreement effects: A 10-year review of empirical studies. Open Econ. J., 3(1), 1113. doi:https://doi.org/10.2174/1874919401003010001

Khandelwal, P. (2017). Which algorithm takes the crown: Light GBM vs XGBOOST? Analytics Vidhya. Retrieved from https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/

Kuik, O., Branger, F., & Quirion, P. (2019). Competitive advantage in the renewable energy industry: Evidence from a gravity model. Renew. Energy, 131, 472-481. doi:https://doi.org/10.1016/j.renene.2018.07.046

Larch, M., Wanner, J., Yotov, Y. V., & Zylkin, T. (2019). Currency unions and trade: A PPML re-assessment with high-dimensional fixed effects. Oxf. Bull. Econ. Stat., 81(3), 487-510. doi:https://doi.org/10.1111/obes.12283

Liakos, K. G., Busato, P., Moshou, D., Pearson, S., & Bochtis, D. (2018). Machine learning in agriculture: A review. Sensors, 18(8), 2674. doi:https://doi.org/10.3390/s18082674

Lopez, A. O. (2020). Protectionism and trade in renewable energy infrastructure. AIIB Working Paper. AIIB.

Martin, W. (2020). Making gravity great again. Policy Research Working Paper 9391. Washington, DC: World Bank. doi:https://doi.org/10.1596/1813-9450-9391

Melitz, J. (2007). North, South and distance in the gravity model. Eur. Econ. Rev., 51(4), 971-991. doi:https://doi.org/10.1016/j.euroecorev.2006.07.001

Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018). Foundations of machine learning. Cambridge, MA: MIT Press.

Moďsé, E., & Sorescu, S. (2013). Trade facilitation indicators: The potential impact of trade facilitation on developing countries' trade. OECD Trade Policy Papers. No. 144. OECD Publ. doi:https://doi.org/10.1787/5k4bw6kg6ws2-en

Patcha, A., & Park, J.-M. (2007). An overview of anomaly detection techniques: Existing solutions and latest technological trends. Comput. Networks, 51(12), 3448-3470. doi:https://doi.org/10.1016/j.comnet.2007.02.001

Rodriguez-Galiano, V., Sanchez-Castillo, M., Chica-Olmo, M., & Chica-Rivas, M. (2015). Machine learning predictive models for mineral prospectivity: An evaluation of neural networks, random forest, regression trees and support vector machines. Ore Geol. Rev., 71, 804-818. doi:https://doi.org/10.1016/j.oregeorev.2015.01.001

Schierhorn, F., Meyfroidt, P., Kastner, T., Kuemmerle, T., Prishchepov, A. V., & Müller, D. (2016). The dynamics of beef trade between Brazil and Russia and their environmental implications. Glob. Food Secur., 11, 84-92. doi:https://doi.org/10.1016/j.gfs.2016.08.001

Sharma, R., Kamble, S. S., Gunasekaran, A., Kumar, V., & Kumar, A. (2020). A systematic literature review on machine learning applications for sustainable agriculture supply chain performance. Comput. Oper. Res., 119, 104926. doi:https://doi.org/10.1016/j.cor.2020.104926

Shepherd, B. (2013). Gravity model of international trade: A user guide. Herndon: United Nations Economic and Social Commission for Asia and the Pacific (ESCAP), Asia-Pacific Research and Training Network on Trade (ARTNeT).

Silva, J. M., & Tenreyro, S. (2006). The log of gravity. Rev. Econ. Stat., 88(4), 641-658. doi:https://doi.org/10.1162/rest.88.4.641

Tan, N. W. (2022). Explaining negative R-squared. Towards Data Science. Retrieved from https://medium.com/towards-data-science/explaining-negative-r-squared-17894ca26321

Tinbergen, J. (1962). Shaping the world economy: Suggestions for an international economic policy. New York: Twentieth Century Fund.

Traore, B. B., Kamsu-Foguem, B., & Tangara, F. (2017). Data mining techniques on satellite images for discovery of risk areas. Expert Syst. Appl., 72, 443-456. doi:https://doi.org/10.1016/j.eswa.2016.10.010

USDA Economic Research Service. (2020). Retrieved from https://www.ers.usda.gov/data-products/agricultural-trade-multipliers/2018-data-overview/

Varian, H. R. (2014). Big data: New tricks for econometrics. J. Econ. Perspect., 28(2), 3-28. doi:https://doi.org/10.1257/jep.28.2.3

Yotov, Y. V., Piermartini, R., Monteiro, J.-A., & Larch, M. (2016). An advanced guide to trade policy analysis?: The structural gravity model. Geneva: World Trade Organization. doi:https://doi.org/10.30875/abc0167e-en