We collect and analyze data in order to make inferences. Sure, there are other reasons to collect data.  For example, every year I write down the date that the forsythias come into bloom in our backyard. No inferences there … just keeping a record and remarking on changes. But justifications for time spent in school learning about statistics and graphing typically note that life is full of uncertainty and we can use data and statistical inferences to make better decisions and begin to understand how things work. Here’s the catch: making statistical inferences requires mathematics beyond what many people learn in school. Still, there is hope. New work on “informal inference” hints at the possibility of providing disciplined, yet broadly accessible ways to use data to make decisions. If we can really do this, the implications for how we help people develop data literacy are far-reaching.

Informal AND Disciplined Inference

“Random” is not a synonym for “haphazard” but a description of a kind of order different from the deterministic one that is popularly associated with science and mathematics.  (Moore, 1990, p. 98).

One of the beliefs that drives the work I do is that it should be possible to increase the application of scientific knowledge and practices to local decision-making. Consequently, I am often in meetings where local stakeholders without scientific training consider scientific data collected by local researchers, students, and others. These meetings almost always center on making inferences from data.  It is not enough to just say, “Here is what we found.” The stakeholders also want to know, “What does it mean?” and “What should we do?”  It is in this inference-making that these meetings sometimes take a turn for the worse. This can happen in different ways and for different reasons.

Sometimes the data are disregarded as not being generalizable. “Well, that’s just what happened where you collected your samples.  But I’ve seen things turn out differently in other places.”  Or, “Well, it’s different every year, so what you found last year won’t happen next year.”

Other times, the problem is one of too much generalization, as when one or two participants in the conversation see a trend, perhaps looking at just a part of the dataset, and begin to draw firm conclusions without acknowledging uncertainty.

Yet other times, participants will just use the data as a jumping-off point for talking about their own experiences, views on what is happening, and ideas about what should be done. When this happens, it is as if the data are just one story among many.

The quotation from David Moore at the beginning of this section speaks to these examples of inference-making gone wrong. “Informal inference” is not haphazard inference.  It acknowledges that there is variation within the data that were collected and that there will be variation between those samples and others collected at other times and places. However, within that variation there are patterns and order that can be used to make inferences in a disciplined way that supports reasoned decision-making. Maker and Rubin (2018), drawing upon more than a decade of work by a variety of researchers, note that disciplined approaches to informal inference generally share five characteristics:

1. There is a claim being made that goes beyond the data at hand.
2. The claim is accompanied by an expression of uncertainty (which is not necessarily expressed in terms of formal probabilities).
3. Data are used as evidence for the claim.
4. The inferential reasoning considers the qualities of the aggregation of the data (for example, mean, median, variability, shape) rather than just individual data points.
5. Inferential reasoning builds on contextual knowledge such as the design of the study and the nature of the system being studied.

This list draws attention to what is missing in cases of inference-making gone wrong. It also suggests what we should be doing in school if we hope to improve our collective ability to make better use of scientific evidence in community decision-making. What is especially exciting is that, over the past decade, teachers and researchers have begun to show how it is possible to involve students, including children in the primary grades, in making inferences that give them practice in using these five elements. I will describe some of those studies and stories in future posts.

Why Haven’t We Been Teaching Informal Inference?

First, I do know teachers who regard the development of informal inference skills as a learning objective and who help students master those skills. Even so, this is not yet a common practice.  One reason this is so is that many science teachers are not, themselves, confident in their own ability to work with data, particularly when it comes to probability and inferential statistics.  A second impediment is the approach that statistics educators have traditionally taken in designing the K-12 curriculum for learning about measurement, data, and how to use them.  Konald (2007) described this as …

a top-down approach in which the college-level course—taken as the ultimate goal—was progressively stripped down for lower grades … as if they were dropping down a short rope from the college level to students in high school. When, more recently, educators began developing statistics units for middle and elementary school students, they continued lowering the rope, basically by removing from the college curricula the concepts and skills they considered too difficult for younger students. The objectives and content at a particular level are thus whatever was left over after subjecting the college course to this subtractive process. (p. 270)

The consequence of this approach to curriculum development is that any ideas and practices that are not aimed toward the ultimate goal of mastery of college-level statistics have not been part of the curriculum.

Attention to developing the ability to make well-constructed informal inferences changes that.  Success in college-level statistics is no longer the only goal.  Rather than a curriculum that narrows the number of participants as students proceed from primary school to college, we can now begin thinking in terms of a curriculum aimed at developing the broadly useful (I would call is “essential”) capacity to make reasoned judgments from data without having to use mathematical tools that most people don’t have.

To be clear: I am not diminishing the importance and utility of advanced statistical reasoning.  Personally, my own recent focus has been on getting a better understanding, just for myself, of Bayesian inference techniques, which is to say that I think that such knowledge is important and practically useful. But that is specialized knowledge rather than knowledge that everyone needs. What is exciting about the emergence of a focus on informal inference among statistics educators is that it aims at knowledge that can be used by people who are not specialists.  And THAT, I hope, will increase the overall capacity to use data and science in making important decisions.

References

Konold, C. (2007). Designing a data analysis tool for learners. In M. C. Lovett & P. Shah (Eds.), Thinking with data (pp. 267–291). New York: Lawrence Erlbaum Associates.