ASAE Conference Proceeding
This is not a peer-reviewed article.
Using Computer Models in Court: Challenges for Expert Witnesses.
R. B. Blackstock. J.D.
Pp. 273-280 in Total Maximum Daily Load (TMDL) Environmental Regulations II, Conference Proceedings, 8-12 November 2003 (Albuquerque, New Mexico, USA), ed. Ali Saleh. ,8 November 2003 . ASAE Pub #701P1503
Lawsuits often follow from disagreements over the allocation of responsibility and liability for degradations in water quality caused by point and nonpoint source pollution. The recent case filed by the City of Tulsa about water quality issues in the Eucha/Spavinaw Basin located in northeast Oklahoma and northwest Arkansas is an example. A central issue in many of these lawsuits is the reliability and admissibility of expert witness testimony based upon the results of mathematical models. Challenges to model results are typically based on objections to data quality, collection, and management and the development, application, performance and validation of the models themselves. Researchers who provide expert testimony in court based on the forensic use of models face unique challenges that differ from the ordinary peer review process with which they have become familiar in the course of their academic work. Nevertheless, they may enhance the ability of their results to survive these challenges by extending the data collection and quality control procedures they already employ and by fully documenting the model validation and calibration procedures that were used to establish the reliability of the results that form the basis for expert testimony.KEYWORDS. Water quality, point and non-point source pollution, development, application, performance and validation of models, data quality, collection and management, forensic use of computer models, Daubert challenges.
In the early part of the twentieth century, the demand for petroleum increased dramatically partly as the result of increasing use of automobiles and other devices powered by internal combustion. The city of Tulsa, Oklahoma, found itself in the middle of an economic boom due to the nearby discovery of what were the most prolific oil fields known at the time. In 1922, to meet its rapidly expanding demands for municipal water supplies, the City proposed a public waterworks project. The project would impound Spavinaw Creek, which drains a 107,484 hectare watershed basin (the “Watershed”) sixty miles east of Tulsa, creating Spavinaw Lake, a reservoir with a 32,562,903 cubic meter storage capacity, initially capable of supplying more than 100,000,000 liters of raw water per day through an enclosed aqueduct to a municipal water treatment plant. Two years later, the City completed the project. In 1954, to extend the capability of the system, the City constructed another reservoir, Lake Eucha, 6.4 kilometers upstream from Spavinaw Lake, thereby providing additional deliverability.
Using estimated background loading in the absence of human impact, the Oklahoma Water Resources Board (OWRB) estimated Spavinaw Lake would have had a total phosphorus concentration of 0.0071 mg/L in 1998 and 0.0073 mg/L in 1999 based on observed lake flow data for those years (OWRB 2002). Lake Eucha would have had estimated concentrations of 0.0022 mg/L in 1998 and 0.0023 mg/L in 1999 (OWRB 2002). The corresponding Carlson’s Trophic State Index (TSI) for the estimated pre-human impact values are 32 for Spavinaw Lake and 16 for Lake Eucha, and would have placed both Lakes within the oligotrophic category.
Actual measured phosphorus concentrations and TSI values for those years are much higher. For the period from April 16, 1998, to March 7, 2000, the median phosphorus concentration for Spavinaw Lake was 0.020 mg/L, almost three times higher than the predicted value in the absence of human contact. Several miles upstream, Lake Eucha’s actual value was 0.030 mg/L, more than ten times the predicted value in the absence of human contact (OWRB 2002). Both lakes are now classified as eutrophic. Lake Eucha and Spavinaw Creek are on Oklahoma’s 303(d) list of waters that have threatened or impaired beneficial uses. In 1997, the City of Tulsa initiated a water quality evaluation of the Eucha/Spavinaw Lake system. The study concluded that both Lake Eucha and Spavinaw Lake are nutrient-enriched and display high or excessive levels of algal production. During the study, the City experienced significant taste and odor events attributable to the algal production. At one point the concentration of geosmin, a significant chemical causative agent of taste and odor problems, exceeded 2,000 ppt as measured at Spavinaw Dam, well above the human detection threshold for geosmin at 4 ppt. Geosmin concentrations at the level measured in Spavinaw Lake were not treatable by the technology then in place at Tulsa’s water treatment plant. Consequently, Spavinaw Lake was temporarily abandoned as a raw water source for the duration of the taste and odor event. When water from Spavinaw Lake was usable, it became increasingly expensive to treat. Annual water treatment chemical costs incurred by the City of Tulsa for taste and odor control at the water plant treating water from Spavinaw Lake increased by a factor of 30 from $25,600 in 1996 to $778,500 in 1999(1). Chemical treatment costs subsequently declined due to the installation of granular activated carbon filter media at the City’s water treatment plant at a cost of $761,000 beginning in 1999.
The OWRB Study concluded the algal production was caused by excessive phosphorus loading to the Lakes (OWRB 2002). The average annual total phosphorus load to the lakes for the period from January 1998 to December 2001 was estimated at 38,432 kg.(2) Background loading was estimated to be 5,714 kg, with the remaining load attributed to agricultural activities in the Watershed and a point source at the outfall of the wastewater treatment plant operated by the city of Decatur, Arkansas.
Actual phosphorus loading to the Lakes cannot be directly measured due to the large number of agricultural nonpoint sources of phosphorus in the Watershed, which is contained primarily in Benton County, Arkansas, and Delaware County, Oklahoma. These two counties have become prolific producers of poultry products since the construction of the Lakes. The USDA Animal Census shows that the number of chickens and turkeys in Benton County increased from 4.9 million in 1945 to 136.3 million in 1997. In Delaware County during the same time period, the chicken and turkey population increased from 0.3 million to 32.8 million. There are an estimated 85 million birds rotated annually through over a thousand poultry houses located within the Watershed.(3)
Phosphorus is imported as feed for the poultry populations in the Watershed. Approximately 2.5 million pounds of phosphorus are added to the Watershed in poultry litter which is land applied for fertilizer. Some of the imported phosphorus is exported in litter from the Watershed, amounting to approximately 600,000 pounds per year since 1998. Other relatively small amounts leave in the meat that is also exported. The rest of the phosphorus causes builds up in the soil, streambeds and lake sediments in the Watershed. Prior to litter application, Mehlich III soil test phosphorus (STP) values probably ranged from 20 to 30 pounds per acre.(4) Average STP concentrations for the Arkansas portion of the Watershed are now 334 pounds per acre, and 150 pounds per acre for the Oklahoma portion of the Watershed (Storm et al., 2001).
Mass balance models were developed to simulate the water and phosphorus balances of the Lakes(5) using the phosphorus loadings developed by Dr. Dan Storm using the SWAT Model.(6) The models were calibrated to the data collected from the Lakes over the 1998-2001 period, and served to quantify the magnitudes and relative importance of both external sources and internal recycling of phosphorus as factors driving algal productivity in the reservoirs.
The model results predicted a 50% reduction in external phosphorus loading should reduce phosphorus concentrations from 24 ppb to 14 ppb after four years. Actual reductions would be subject to variations in precipitation and storm events that would occur in the Watershed. This reduction in phosphorus concentration would in turn reduce the algal bloom frequency from 41% to 15%.(7)
The SWAT Model also was used to evaluate different methods of achieving a 50% phosphorus load reduction. Measurements of phosphorus concentrations in the effluent of the wastewater treatment plant of the City of Decatur showed that the effluent from the plant contributed 23 % of the annual phosphorus load. Even the complete elimination of the point source as a phosphorus load to the Lakes would leave an additional reductions to be achieved through other means. The results of the SWAT Model indicated that litter application caused 33% of the phosphorus load. Elevated soil phosphorus levels caused an additional 16%. Land cover changes (6%), cattle (10%) and background phosphorus (11%) caused the rest. The SWAT Model results showed that to achieve a sufficient reduction in phosphorus loading to the lakes, most of the releases of phosphorus from land application of litter and Decatur’s wastewater treatment plant would have to end.
In December 2001 the City filed suit in Federal Court in Tulsa for damages and injunctive relief. The lawsuit involved the presentation of scientific evidence in the form of expert opinions on the consequences of the release of phosphorus in the Watershed. The lawsuit also provided the forum for the first recorded forensic use of results of the SWAT Model in Federal Court.
Legal Challenges to Expert Opinions
Both plaintiffs and defendants in the litigation hired expert witnesses with extensive education, training and experience who submitted their opinions to the court on the available scientific evidence. The City of Tulsa and the Tulsa Metropolitan Utility Authority as Plaintiffs submitted the expert opinion of Dr. Dan Storm and the opinions of Dr. Dennis Cooke,(8). Dr. William Walker, and Dr. Wes Jarrell, and other experts, which relied in part, either directly or indirectly on Dr. Storm’s opinion.
Defendants filed a legal challenge to Dr. Storm’s expert opinion based in part on a legal ruling in a case handed down in 1993 by the United States Supreme Court in the case of William Daubert v. Merrill Dow Pharmaceuticals, Inc.(9) In Daubert the Supreme Court established new guidelines and requirements for the forensic use of expert witness testimony. The Daubert Case involved birth defects allegedly sustained as a result of ingestion of Bendectin by pregnant women. Before the ruling in Daubert , the standard for admitting expert scientific testimony was based on an opinion not by the Supreme Court, but by a lower Court of Appeals, in Frye v. United States,(10) which was a case involving “decision concerning the admissibility of evidence derived from a systolic blood pressure deception test, a crude precursor to the polygraph machine.”(11) In Frye the Court of Appeals stated a rule that had been relied upon by Federal Courts (and most state courts) for 70 years. The rule was that a scientific principle must have gained “general acceptance” in the scientific community before opinions that rely on it are admissible as evidence, thereby creating a problem because general acceptance usually comes gradually to a scientific principle:
"Just when a scientific principle or discovery crosses the line between the experimental and demonstrable stages is difficult to define. Somewhere in this twilight zone the evidential force of the principle must be recognized, and while courts will go a long way in admitting expert testimony deduced from a well-recognized scientific principle or discovery, the thing from which the deduction is made must be sufficiently established to have gained general acceptance in the particular field in which it belongs. "(12)
The question before the Supreme Court in Daubert was whether to change the old rule which required general acceptance of a scientific principle. The Plaintiffs in Daubert contended the old rule had been superseded and replaced by new rules, specifically Rules 401 and 702, which they contended would make admissible any scientific evidence which has "any tendency to make the existence of any fact that is of consequence to the determination of the action more probable or less probable than it would be without the evidence."(13) Not all expert testimony that has “any tendency” is admissible, however, because the evidence must also satisfy Rule 702 which now provides:
If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise if (1) the testimony is based upon sufficient facts or data, (2) the testimony is the product of reliable principles and methods, and (3) the witness has applied the principles and methods reliably to the facts of the case.
The Supreme Court observed that neither Rule 401 nor Rule 702 state any requirement that “general acceptance” is a prerequisite for admissibility. The Supreme Court stated:
Frye made "general acceptance" the exclusive test for admitting expert scientific testimony. That austere standard, absent from, and incompatible with, the Federal Rules of Evidence, should not be applied in federal trials.(14)
This statement by the Supreme Court did not require that the courthouse doors were to be opened to any form of scientific expert testimony. The Supreme Court held that trial judges still must screen expert witness testimony to ensure that evidence admitted is “not only relevant but reliable.” This obligation presented a new and unique challenge to federal judges whose careers have not typically been in scientific research, but rather in law and legal research. Nevertheless, they are required under the Rules to serve in a gatekeeping role:
This entails a preliminary assessment of whether the reasoning or methodology underlying the testimony is scientifically valid and of whether that reasoning or methodology properly can be applied to the facts in issue. We are confident that federal judges possess the capacity to undertake this review.(15)
In fulfilling that role, some valid evidence might be excluded because it has not yet received sufficient analysis. The petitioners in Daubert , arguing for the admission of their evidence contended that the “recognition of a screening role for the judge that allows for the exclusion of "invalid" evidence will sanction a stifling and repressive scientific orthodoxy and will be inimical to the search for truth.”(16) The Supreme Court responded by explaining the difference between science and litigation:
It is true that open debate is an essential part of both legal and scientific analyses. Yet there are important differences between the quest for truth in the courtroom and the quest for truth in the laboratory. Scientific conclusions are subject to perpetual revision. Law, on the other hand, must resolve disputes finally and quickly. The scientific project is advanced by broad and wide-ranging consideration of a multitude of hypotheses, for those that are incorrect will eventually be shown to be so, and that in itself is an advance. Conjectures that are probably wrong are of little use, however, in the project of reaching a quick, final, and binding legal judgment--often of great consequence--about a particular set of events in the past. We recognize that, in practice, a gatekeeping role for the judge, no matter how flexible, inevitably on occasion will prevent the jury from learning of authentic insights and innovations. That, nevertheless, is the balance that is struck by Rules of Evidence designed not for the exhaustive search for cosmic understanding but for the particularized resolution of legal disputes.(17)
Rule 401 is potentially a more liberal than the Frye rule in that it may allow the admission of scientific evidence even if scholarly debate continues concerning the scientific principles upon which the evidence is based. The Frye rule made scientific evidence inadmissible unless the scientific principles upon which the evidence is based are generally accepted. Under the rule in Frye , such evidence is inadmissible during the period of controversy. Since scientific knowledge evolves, principles that are now generally accepted were once controversial. The Supreme Court quoted from a brief filed by the American Association for the Advancement of Science which observed that “[s]cience is not an encyclopedic body of knowledge about the universe. Instead, it represents a process for proposing and refining theoretical explanations about the world that are subject to further testing and refinement.”(18).
Under Daubert , federal district courts are instructed to decide the admissibility of scientific evidence not simply on the general acceptance of its underlying scientific principles, but instead by using a more flexible methodology which may include general acceptance as a component. Consequently, much more scientific evidence has been excluded after Daubert , than was excluded before it. The Daubert test involves the use of a non-exclusive list of four factors:
1) Testing: The statements constituting a scientific explanation must be capable of empirical test.(19)
2) Publication: The fact of publication in a peer-reviewed journal is a relevant but not dispositive consideration in determining the reliability of the scientific validity of a methodology upon which an opinion is premised.
3) Error rate: The court should consider the known or potential error rate of the proposed methodology.
4) Acceptance: A known technique which has been unable to garner widespread acceptance should be viewed with skepticism.
Relying on Daubert , and other cases that followed it, defendants in the City of Tulsa Case asserted Dr. Storm’s expert witness opinion was inadmissible. Defendants challenged the input data for the SWAT Model, and the use of the SWAT Model itself. Since there was no other case of record in which the results of the SWAT Model were offered as evidence, the Court’s decision on the usefulness of the SWAT Model would be significant to other Courts that might be asked to rule on the admissibility of the model results. At the hearing on the admissibility of expert testimony about the results of the model, Plaintiffs offered the testimony of Dr. Jeff Arnold and Dr. Raghavan Srinivasan, the primary developers of the model, and the testimony of Dr. David Gade, an expert with extensive experience with the model. Defendants countered with testimony of witnesses that the model and its results should not be admitted.
The Court ruled on Defendants’ motion by allowing admission of most but not all of the evidence relating to the SWAT Model results. First the Court noted:
With regard to the Soil and Water Assessment Tool (SWAT) Model utilized by Dr. Storm, the literature generally indicates that the SWAT Model is a reasonable model to use to gain an understanding of the behavior of a Watershed and to assess the likely sources of nutrient loading, including phosphorus, and/or pollutants within a Watershed.(20)
Next, the Court ruled in favor of the admissibility of the testimony based total phosphorus loading to Lake Eucha, but declined to admit evidence as to the percentage allocations of phosphorus loading attributable to particular land uses as derived from the SWAT Model. The Court stated:
The evidence clearly shows that Dr. Storm’s SWAT Model output is scientifically valid and reasonably reliable for the purposes of his opinions regarding total phosphorus loadings to Lake Eucha per year for the period for which he had observed data, and for his opinions regarding relative ranking of sources of phosphorus in the Watershed. In the Court’s opinion, his SWAT Model Output is not sufficiently reliable for attribution of specific percentages of phosphorus per year by land use to be admissible at trial.(20)
The trial court based its decision against the admission of the part of the SWAT Model Output that attributed specific percentages of phosphorus by land use on the lack of sufficient data for validation of the model results with regard to percentage allocations. Additional data obtained through further research potentially would make validation of the specific percentages possible. The Court did conclude, however, that Dr. Storm would be permitted to testify as to the relative ranking of sources of phosphorus contribution, as well as his personal experiences and observations of phosphorus sources in the Watershed. Only the mathematical output of the model that attempted to assign percentages of phosphorus contribution was deemed inadmissible for want of sufficient validating data.
The parties subsequently settled the litigation on terms that have resulted in new rules governing the management of litter application in the Watershed. The settlement agreement provides for the complete cessation of land application of litter in the Watershed until new nutrient management plans are completed by a Watershed Management Team to be supervised by a court appointed Special Master. The nutrient management plans are to be based on a phosphorus risk-based index to be developed by researchers from Oklahoma State University and the University of Arkansas. Compliance with nutrient management plans will be monitored by the Watershed Management Team. The proof of effectiveness of the process will be the reduction of phosphorus loading resulting from agricultural practices in the Watershed.
Expert witness testimony is generally admissible as evidence in Federal Court if it is reliable and relevant to the issues in the case, but not all expert testimony will be admitted. Scientific research is advanced by “broad and wide-ranging consideration of a multitude of hypotheses.”(21) The objective of law, however, is to reach a “quick, final, and binding legal judgment.”(22) This will inevitably on occasion prevent the admission of “authentic insights and innovations,” because the process is “designed not for the exhaustive search for cosmic understanding but for the particularized resolution of legal disputes.”(23) Although one result of the SWAT Model was excluded, the majority of the results was deemed admissible, and the use of the model itself as a reliable scientific tool was upheld.
Brent Blackstock is Of Counsel with the law firm of McKinney & Stringer, P.C., based in Oklahoma City and Tulsa, Oklahoma. Mr. Blackstock’s practice includes substantial trial experience in the fields of energy law, environmental litigation, business litigation and commercial transactions. He is a member of the firm’s trial team representing the City of Tulsa in an action against several poultry processing companies and the City of Decatur, Arkansas for alleged non-point source pollution of Tulsa’s public water supply caused by poultry farming and waste water treatment plant operations in the Lake Spavinaw and Lake Eucha watershed in Northeast Oklahoma and Northwest Arkansas. McKinney & Stringer is a full service lawfirm with a diverse practice and clientele, including a significant practice in environmental law. For further information see www.McKinneyStringer.com or send an email to BlackstockB@MckinneyStringer.com.
1.The analysis of water treatment costs and economic damages in the City of Tulsa case was prepared by Murry L. Fleming, P.E., and summarized in an expert witness report submitted on behalf of the City of Tulsa and Tulsa Metropolitan Utility Authority.
2.The modeling and analysis of phosphorus loading in the Spavinaw/Eucha Watershed was prepared by Dr. Daniel E. Storm, and summarized in an expert witness report submitted on behalf of the City of Tulsa and Tulsa Metropolitan Utility Authority.
3.The historical analysis of agricultural and poultry operations in the Watershed, and their impact on soils and surface water runoff was prepared by Dr. Wesley M. Jarrell, and summarized in an expert witness report submitted on behalf of the City of Tulsa and Tulsa Metropolitan Utility Authority.
5.An updated model of the phosphorus balance in Spavinaw and Eucha Reservoirs, using the most recent loading data derived from Dr. Storm’s work, was prepared by Dr. William W. Walker and summarized in an expert witness report submitted on behalf of the City of Tulsa and Tulsa Metropolitan Utility Authority.
6.The SWAT Model is a conceptual hydrologic model with spatially explicit parameterization (Arnold 2000).
8. Dr. G. Dennis Cooke, a limnologist, analyzed the historical and expected future impact of the poultry operations and Decatur plant on the Spavinaw and Eucha Reservoirs, and summarized his findings and proposed corrective actions in an expert witness report submitted on behalf of the City of Tulsa and Tulsa Metropolitan Utility Authority.
9.509 U.S. 579, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993).
10.54 App.D.C. 46, 293 F. 1013 (1923).
11. Daubert, p. 585.
12. Id. , pps. 585-6. Italics in original text.
14. Daubert , p. 589.
15. Id. , pps. 592-3.
16. Id. , p. 596.
17. Id. , pps. 596-7.
18. Id. , p. 590.
19. Id. , p. 593
20.District Court Order of March 10, 2003, in City of Tulsa et al. v. Tyson Foods, et al., Case No. 01 CV 0900EA(C), p. 3
21.District Court Order, pps. 6-7.
22. Daubert , p. 597.
Arnold, J.G., R.S. Muttiah, R. Srinivasan, P.M. Allen, Regional Estimation of Base Flow and Groundwater Recharge in the Upper Mississippi River Basin. Journal of Hydrology 227 (2000).
Oklahoma Water Resources Board. 2002. Water Quality Evaluation of the Eucha/Spavinaw Lake System.
Storm, D.E., M. White, M.D. Smolen and H. Zhang. 2001. Modeling Phosphorus Loading for the Lake Eucha Basin. Final Report. Oklahoma State University.