Scientific evidence/uncertainty (Science and Health Communication)

Abstract

The presented scientific evidence and uncertainty in science communication can be achieved by either different variables (e.g., Brechman, Lee, & Cappella, 2009, 2011; Guenther, Bischoff, Löwe, Marzinkowski, & Voigt, 2019; Kessler, 2016) or identifying frames (for thematic frames, see Ruhrmann, Guenther, Kessler, & Milde, 2015; for formal-abstract frames, see Kessler, 2016).

Field of application/theoretical foundation:

Evidence and (un)certainty are integral components of scientific findings and science in general. Scientific evidence can be defined as a continuum, ranging from scientific uncertainty to certainty and from weak to strong evidence. Media content analyses are investigating the extent to which information is given in media articles that provide indications of the evidence or uncertainty of scientific findings. Content Analyses also measure how evident scientific findings are presented in the media.

References/combination with other methods of data collection:

In some cases, the effects of different uncertainty depiction styles (Retzbach & Maier, 2015) and frames of the depicted evidence (Kessler, 2016) are examined after the content-analytical identification in experiments.

Example studies:

Brechman, et al. (2009); Brechman et al. (2011); Guenther et al. (2019); Kessler (2016); Retzbach & Maier (2015); Ruhrmann et al. (2015)

Information on Guenther et al., 2019

Authors: Lars Guenther, Jenny Bischoff, Anna Löwe, Hanna Marzinkowski, & Marcus Voigt

Research question: When they represent research results, how do German print and online media report on (a) relevant criteria to assess scientific evidence and (b) scientific (un)certainty?

Object of analysis: The study was based on a randomly selected artificial week to obtain a representative sample of German print and online media reports on science (N = 128 articles).

Time frame of analysis: July 6, 2015 to August 23, 2015

Info about variables

Variables: “For each represented research result, a variable collected the main (hypo-)thesis of the research study, the direction of the result (for or against the thesis), as well as the relevant criteria to assess evidence. […] For each result, it was also relevant to collect to which extent scientific certainty or scientific uncertainty was discussed. In the current study, an explicit statement referring to (un)certainty was differentiated from an implicit statement (subjunctive, speculative language as an indicator of uncertainty versus imperative as an indicator for certainty). This was supplemented by collecting the justifications for (un)certainty that were given for the scientific results.” (p. 10)

Level of analysis: news article

Variables and values:

• reported relevant criteria to assess scientific evidence: theoretical assumptions/(hypo-)theses; pilot study/a study never done before; research design: experiment, case study, etc.; research and measurement instruments; quality criteria, such as reliability; quality criteria, such as validity; references to significance (statistic values); objectivity; information about sample (size); time of study; explicit depiction of the research setting; number of studies done; information about how results were obtained; limitations, such as knowledge gaps; comparisons to other studies; funding source(s); reference to the investigating researcher(s); reference to the publication/ journal/ conference; future scenarios, specific applications
• reported explicit justifications for scientific (un)certainty: preliminary data, knowledge gap(s); (poor) methodological quality; contrasting findings of research; contrasting interpretation of same dataset; conflicting viewpoints of researchers; doubt whether data can be applied to humans; effect on humans not clear; effect on nature not clear; lack of technical/scientific opportunities; justifications for certainty; certain single result(s); sufficient data; (strong) methodological quality; results pointing in the same direction; successfully replicated findings; application for humans clear; effect on humans clear; effect on nature clear; highly experienced researcher(s)
• implicit statement referring to (un)certainty: no implicit representation vs. implicit representation

Reliability: “Four experienced coders coded the articles of the sample after several intensive training sessions. Intercoder reliability was calculated according to Holsti for 26 articles (20 percent of the sample) and the following satisfactory results were obtained: formal variables: 0.97; criteria relevant to assess evidence: 0.92; uncertainty (explicit and implicit): 0.95; certainty (explicit and implicit): 0.92.” (p. 10)

Codebook: in the appendix (in German)

Information on Kessler, 2016

Author: Sabrina Heike Kessler                 

Research questions: How evident are medical issues presented in science TV programs? Are there any relationship between the individual types of evidence sources and the way they are presented? Can constant formal-abstract patterns/frames of presented evidence be identified?   

Object of analysis:  There was a full-sample content analysis of science TV programs about scientific and medical issues (N = 321, with N = 851 evidence source argumentations).

Three frames of evidence identified via a cluster analysis. The frames differed significantly in their degree of depicting belief, doubt, and uncertainty, which were defined as the core elements of a frame of evidence.

Timeframe of analysis: August 1, 2011 to July 31, 2012 

Info about variables

Variables: variables that measure the represented uncertainty in the argumentations of evidence sources and variables that determine the formal-abstract evidence frames.

Level of analysis: Science TV programs and evidence source arguments

Variables, values and reliability:

Intercoder reliability values of the coding separated by variables

Variable	Number of Possible Values	Number of Codings	Holsti Reliability Coefficient	Cohen's Kappa
V5 (specific topic)	1 to x	30	.93	.92
V6 (general topic)	1 to x	30	.91	.90
V7b (main thesis)	x	30	.93	.92
V9 (number of evidence sources)	1 to x	30	.98	.82
V10 (type of evidence source)	6	57	.93	.91
V11 (validity of the evidence source)	4	52	.90	.85
V12 (arguments for)	3	52	.81	.63
V13 (arguments against)	3	52	.92	.73
V14 (polarity)	3	52	.99	.92
V15 (weighting)	2	52	.96	.58
V16 (actuality)	2	52	.95	.54
V17 (uncertainty explicit)	3	52	.86	.39
V18 (implicit uncertainty)	3	52	.79	.45
V19 (homogeneity)	2	52	.95	.51
V20 (detailing)	2	52	.92	.41
V21 (constancy)	3	52	.93	.71
V22 (secondary evaluation)	3	52	.67	.42

Codebook: in the appendix (in German)

References

Brechman, J. M., Lee, C., & Cappella, J. N. (2009), Lost in translation?: A comparison of cancer-genetics reporting in the press release and its subsequent coverage in the press. Science Communication, 30(4), 453-474. DOI: 10.1177/1075547009332649

Brechman, J. M., Lee, C., & Cappella, J. N. (2011), Distorting genetic research about cancer: from bench science to press release to published news. Journal of Communication, 61(3), 496-513. DOI: 10.1111/j.1460-2466.2011.01550.x

Guenther, L., Bischoff, J., Löwe, A., Marzinkowski, H., & Voigt, M. (2019). Scientific evidence and science journalism: Analysing the representation of (un)certainty in German print and online media. Journalism Studies, 20(1), 40-59.

Kessler, S. H. (2016). Das ist doch evident! Eine Analyse dargestellter Evidenzframes und deren Wirkung am Beispiel von TV-Wissenschaftsbeiträgen (Reihe Medien + Gesundheit, Band 12). Baden-Baden: Nomos. DOI: 10.5771/9783845275468

Retzbach, A., & Maier, M. (2015), Communicating scientific uncertainty: Media effects on public engagement with science. Communication Research, 42(3), 429-456. DOI: 10.1177/0093650214534967

Ruhrmann, G., Guenther, L., Kessler, S. H. & Milde, J. (2015). Frames of scientific evidence: How journalists represent the (un)certainty of molecular medicine in science television programs. Public Understanding of Science, 24(6), 681-696. DOI: 10.1177/096366251351064