As I was reading this morning I ran across this:

As the great nineteenth-century mathematical physicist Lord Kelvin famously said, “If you cannot measure it, you cannot improve it.” We need real-time data to understand our performance: are we getting better or worse? And we can use quantitative benchmarks–specific numerical goals we want to achieve–to focus our efforts and motivate us to try harder.

Hogwash. Worse … dangerous hogwash. 

The quotation is from a book by Jane McGonigal titled Reality is Broken. She is making the argument that games provide feedback systems that can help us improve. My issue is not with the point she is wanting to make; it is with how she is making it. The logic here is: (1) quote a great scientist; (2) because the quote is from a great scientist, accept it without scrutiny; (3) generalize from it to show that what you want to say must be obviously true.

Lord Kelvin also famously said, “There is nothing new to be discovered in physics now, All that remains is more and more precise measurement,” and “X-rays will prove to be a hoax.”

We have gotten past those pronouncements without obvious damage. But the measurement one … it has been a real problem. Jane McGonigal’s quoting it without irony is evidence of its resonance.  A lot of people hear that idea and think, “Yeah, that’s right.”

I have an old sports car … a 1960 Morgan Plus 4. It has two carburetors … called “SU” carburetors …. that take time and care to adjust. Adjustment is partly a matter of inspection to make sure that everything is moving freely, but depends mostly on pushing up a little pin that lifts a piston inside the carburetor. If the engine speed increases briefly and then returns, you’re good. If it keeps running faster, you have got it adjusted to provide too much gas.  If the engine begins to die, you have it adjusted too “lean” — not enough gas. Adjustment takes time — especially because there are two of these things and they need to work in unison. Adjustment involves listening and paying attention. But it doesn’t involve measurement.

I write a lot.  My first drafts of everything always use too many words. Editing my own work involves going over what I have written to remove phrases that just make sentences more complicated without adding anything. Doing this improves my writing. It does not involve measurement.

Sometimes I watch teachers as they review videos of their teaching. They notice things in the video that they were not aware of as they were teaching. Sometimes they combine this noticing with an idea, such as the idea that they want to encourage the students to listen and respond to each other, rather than just paying attention to the teacher. This kind of review and reflection can help teachers improve. It does not involve measurement.

Nobody will be surprised by these examples. Still, when we hear Lord Kelvin’s pronouncement … “If you cannot measure it, you cannot improve it” … it has a ring of authority. That’s the problem.

I have nothing against measurement. I like it, and I like numbers and distributions of measurements. I am now working with Molly Schauffler and other colleagues to devise a way to create a measure of students’ understanding of variability. Similarly, I work a lot with teachers to look at quantitative measures of what students are good at and what they are weak. Working together and with the numbers, we often uncover insights that seem useful.

But I see two BIG problems that come from taking Lord Kelvin’s thinking about measurement any more seriously than we take his thinking about X-Rays or the future of physics.

The first problem is that we take Lord Kelvin at his word and believe that other ways of noticing and knowing things are somehow inferior to counting and scaling. That really is hogwash. If you have a measurement of some kind about a student, you should certainly consider how it fits together with what else you know from working with the student, but it is not TRUE in some sense that displaces the other things you have noticed.

I would be comfortable with “There is no improvement without noticing.”  Measurement can help in noticing, but it is not the only way.

The second problem — related to the first one —  is that when we have a quantitative measurement scheme, it seems that the focus is all on the measurements. It seems that when we have numbers, we start thinking only about “moving the needle” and changing the number, and we stop paying attention to other things.  Numbers seem so precise and simple. They are seductive.

What troubles me is that if I quoted Lord Kelvin’s pronouncement about measurement to many of the people who are making decisions about schools, they would nod solemnly in agreement.

Hogwash. How do we put this idea in the same category as his pronouncement about X-Rays?