Congress relies on sound and accurate data to develop legislation. This is such an important element of the legislative process that Congress enacted H.R.4174, the Foundations for Evidence-Based Policymaking Act of 2018. This legislation requires federal agencies to annually submit to Congress a systematic plan that includes methods and analytical approaches that may be used to develop evidence to support policymaking. This is why it is troubling that some members of Congress are considering introducing legislation based on claims made in the Wall Street Journal’s (WSJ) series “America is Wrapped in Miles of Toxic Lead Cable.” In a forthcoming report, I have conducted a systematic, critical assessment of the scientists’ study and investigation behind this WSJ series. The study was riddled with research biases, missing key analytical data and results, and evaluated to be of very low quality.
The claims made by the WSJ series are based on research conducted by the WSJ and a team of investigators from Marine Taxonomic Services (MTS), an environmental consulting group based in San Marcos, California. According to the WSJ, investigators from MTS discovered lead cables in Lake Tahoe 10 years ago and have advocated for their removal ever since. While not disclosed initially, the WSJ notes that the study was partly funded by the Environmental Defense Fund. The MTS paper outlines the methods and findings of the research study behind the WSJ series. Using this MTS paper, I assessed the WSJ study using the Appraisal Tool for Cross-Sectional Studies (AXIS), the Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I), and the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) Approach. These are well-established scientific tools and methods used to assess research bias and the quality of evidence.
Research bias occurs when systematic errors are introduced into research, adversely affecting its quality. These errors can skew research outcomes. The assessment concluded that the study behind the WSJ series was riddled with biases and errors unacceptable at any level of research. These include sample bias, confirmation bias, confounding bias, and missing analytical data and results.
Sample bias occurs when a research sample of a population is collected in a manner that results in the collected samples not accurately representing the population being studied. Sample bias is usually mitigated by employing probability sampling. The WSJ series, however, failed to use this method. According to the MTS paper, the samples analyzed for the study were, in part, chosen for their likelihood of showing high lead levels. This can be equated to researching the eating habits of Americans and then only interviewing Americans found at a handful of vegan restaurants.
Confirmation bias occurs when a researcher selectively collects or analyzes data to suit their existing beliefs. This bias is usually mitigated by blinding, which is the process of concealing parts of a research process from the research participants and personnel. The WSJ hiring MTS to conduct a critical part of the study is like asking the wolf to guard the hen house. Confounding bias occurs when an outcome is attributed to an exposure when the observed outcome is, in fact, the result of an unmeasured or unknown variable. For example, some of the water samples used in the WSJ series were collected from waterbodies known to be highly polluted, such as the Passaic River. According to the EPA, the lower Passaic is heavily polluted “from a century of industrialization in the Passaic River Watershed.”
High-quality research provides evidence that is robust, ethical, and stands up to scrutiny. The study that led to the WSJ series was plagued with systematic problems – meaning its findings cannot be deemed reliable. However, some members of Congress continue to embrace the WSJ series’ claims. It is advisable for Congress not to rush into enacting legislation that may cost taxpayers billions of dollars addressing an issue that may not exist. Congress, therefore, should work with relevant stakeholders to ensure high-quality research is conducted. Based on the findings of such research, an evidence-based solution can be developed.
Dr. Karl Von Batten is a commissioner to the District of Columbia’s Statewide Health Coordinating Council appointed by Mayor Murial Bowser and founder of Von Batten-Montague-York, L.C., a public health and government affairs consultancy
