Ed School TnT 3: Why yes Herr Professor, we must think for ourselves

Featuring research about as useful as a Minecraft Librarian that won’t sell Mending

[Part 3 of the Ed School Trials and Trivialities series, beginning here.]

I’m just going to jump right into this one because I haven’t written in months. Some parts of the field of education have a problem with science, coming from two distinct directions:

1. empty scientism
2. excessive skepticism

Let’s start with the scientism side for this post because it’s easier. I’m talking about stuff like this:

Look, it’s a picture of a brain! Three of them, actually. This must be SCIENCE.

Universal Design for Learning (UDL) is the core principle of my teacher training program. Every lesson we produce must demonstrate some principle of UDL and, indeed, the university has requested that we use a prescribed UDL lesson template build our entire submission for the EdTPA teacher qualifying assessment.

The problem with this framework though is that despite the colorful brain diagrams, I have no idea what UDL means beyond some basic platitudes about “multiple means” and “meeting the needs of every student” shoehorned into every lesson plan. The reader would be fair to question my own competence (because I am indeed no longer trying) or that of my professors (who have assigned nothing but blog posts or infographics on UDL), but fortunately my confusion is shared fairly widely in the UDL research literature.

There are some real zingers, in stale academic language. To usher in a landmark second decade of scholarship on UDL, Edyburn (2010) states my chief complaint clearly: “I am concerned about the ability of the profession to implement a construct that it cannot define.” I gave a sad “haha” for the academic researcher, but when pushed forward onto an aspiring practitioner, there’s a severe practical problem: if I don’t know what UDL means, how do I implement it?

The science-y-ish details of UDL don’t offer much guidance. Within the three brain networks that organize UDL frameworks, there are nine more subcategories containing about thirty additional implementation guidelines. Those sound excellently specific except that by piling on so many guidelines, the scope of the project balloons to the point of incomprehensibility rather than coming to a greater focus. For example, the expression and communication category for the action and expression network (oy redundancy…) includes the following points:

Use multiple media for communication

• Compose in multiple media such as text, speech, drawing, illustration, comics, storyboards, design, film, music, dance/movement, visual art, sculpture, or video
• Use physical manipulatives (e.g., blocks, 3D models, base-ten blocks)
• Use social media and interactive web tools (e.g., discussion forums, chats, web design, annotation tools, storyboards, comic strips, animation presentations)
• Solve problems using a variety of strategies

That covers… well, just about everything you could do in a classroom! How then do you distinguish a UDL approach from a non-UDL one? The potential scope is literally universal.

In turn, with such a wide target, the research agenda attempting to validate the effectiveness of the UDL framework is all over the place. Echoing Edyburn’s early concern about definitions, a literature review by Rao et al. (2014) find that studies on UDL tend to lack “standard formats for describing how [UDL] is used,” making it difficult to compare results between studies. Though that might be expected in a new field, the rigor of the analytical methods used in those studies also tends to come up lacking, with few of them attempting to establish causality. Instead, many of them were simple descriptive studies of, to cut the academic jargon, whether or not teachers or students liked or used UDL. But of course, liking and using an idea says nothing about its effectiveness or else my preferred activities of playing video games and watching anime would become an optimal pedagogy. A later literature review from Capp (2017) comes out with my favorite expression of the general weakness of UDL research:

“Results from this analysis suggest that UDL is an effective teaching methodology for improving the learning process for all students. The impact on educational outcomes has not been demonstrated.”

Effective teaching method… …impact on outcomes not demonstrated. Quite a puzzler, but I think I’ve got it. =P

The solution to that puzzle, which the literature review authors are all too polite to say, is that though somewhere around 800 academic studies on UDL presently exist (according to Maguire and Hall (2018)), most of those studies probably suck.

I am sad to report that this is true of much of social science.

It’s a bit of an open secret now that certain fields in the social sciences have a bit of a problem with the Replication Crisis. Fancy studies that have studded the headlines of NPR, the New York Times, and too many TED Talks about Power Poses and other dubious social and psychological phenomenon have, upon closer examination, proven false. Even titans of the field, like Zimbardo’s Stanford Prison Experiment, have fallen under the scrutiny of methodologists. Much of the problem lies in the unintentional misapplication of statistics (p-hacking, poor sample selection, inappropriate modeling techniques, etc.), resulting in a positive-results bias encouraged by the cutthroat “publish or perish” dynamic in the academic job market. Without the methodological jargon, it’s pretty simple really: researchers want to find cool results. Publishers want to report them. So, they tend to discount opposing or ambiguous data and focus on the big scoops, even if they might not quite have good evidence or need to squeeze the statistics to get there.

The field of psychology is considered in “crisis” because only about 1% of its studies are replications. For education, one review says it’s just 0.13%. Now, I know part of the point of this post is cultivate a healthy sense of scientific skepticism and not trust any single study. But that’s an order of magnitude worse.

That brings me back to my complaint about my teacher training program: the stupid scientism of professors parroting some variation of “science says” without citing any rigorous literature or examining any of the evidence or arguments behind a concept as a class. In such a context, “science says” becomes an empty, faith-based argument from authority, the very vice that the scientific method seeks to resist. It really doesn’t build confidence when a professor (who, by the way, has assigned readings on the debunked theory of learning styles) answers my concerns with “UDL is science, not a fad” and leaves it at that. But why, exactly, am I supposed to “trust the science” by simple fiat of professorial authority, especially in a field so afflicted by obviously weak methodologies, to say nothing of replication rates?

Our main textbook, Is Everyone Really Equal, devotes much of its first chapter to a condescending argument about why undergraduate students should defer to peer-reviewed research and their professors’ expertise. And sure, peer-review is a useful exercise in basic quality control. But again, the argument from authority irks me – “peer-reviewed” does not mean true, or even good, or else the fraudulent idea that vaccines cause autism would still be an active research topic. Empty repetition of the phrases “believe science” or “peer-reviewed research” capitulate the values behind good science in exchange for stupid scientistic talismans based on credentials rather than careful argumentation.

And ho boy, I’ve come across some atrocious peer-reviewed work while browsing through education journals in my spare time. I’ll remained focused on UDL because I started with it to frame this post, but the principles could apply to any topic:

1. Confirmation bias. I worry about this one the most, though I can’t demonstrate it well. Education is an overwhelmingly progressive field and its research agenda on diversity and inclusion topics like UDL reflects those values. However, are researchers finding real, useful evidence or merely confirming their prior ideological commitments?
2. Unrepresentative samples. I see a lot of convenience samples in education studies, like one that examined faculty and graduate students at the university where the researchers happened to have themselves been faculty. And if that sample size of n = 9 looks bad, I’ve stumbled across studies with a sample size of three. THREE! Why even bother reporting those results? Even then, three is better than one; I am continually baffled by the prominence of “autoethnography” (read: personal essays) in education research. Anecdotes are not evidence!
3. Too much descriptive work. Descriptive studies are interesting but they don’t do much to show cause and effect (important when assessing the effectiveness of a method). Going back to the Rao et al. review, plenty studies show that people like and use UDL. That tells us nothing about if it works.
4. Poor operationalization. Does the study examine what it intends to? Returning to Edyburn’s concerns, are concepts well defined? In a survey, a teacher says they use UDL. Are they really? Or from Rao again: two studies use different criteria for identifying UDL. Will their results be comparable? This is especially important when a topic is as broad as UDL.
5. Conflicts of interest. Education is a big money industry. If you can convince a school district to buy your curriculum, method, or textbook (maybe with some flashy lines like “studies show…”), you could land a few million dollars, easy. Think back to the citation about the bottom of the UDL brain picture. The author, CAST, INC. is the non-profit organization that initially developed and promoted UDL. Nice and cozy, right? However, a non-profit designation can be deceptive. Though the organization itself does not aim to “profit” off the promotion and sale of its products, it still needs to pay the wages of its employees. As constituent members of the non-profit then, those employees have a financial interest in the success of their projects, creating similar incentives for good marketing as a regular for-profit corporation. So, when CAST sells consulting services, $1000 courses, and other resources, the possibility of positive-results bias could creep in.
6. Shaky Foundations. A lot of education research has interdisciplinary links with psychology. As previously mentioned, psychology (including neuroscience!) is experiencing a bit of a replication crisis. Are those brain network theories that inform UDL still valid after further study?

More than anything, my teacher training program infuriates me for its hypocrisy. Professors tell us to use critical thinking and in turn teach it to our own students. Oh, and that buzzword “inquiry” too! The basis of science! We should encourage students to interrogate the topic as rational actors, not simply tell them a truth. But in class, we don’t do that. Professors say “the science says,” plop some unreadable infographic lifted off Google onto their slideshows, and rest on that institutional authority – not empirical evidence, sound methods, rigorous reasoning, or even just a persuasive opinion – but something like “check out this pretty picture — a brain, a pyramid!” It’s so stupid, infantile even!

To return to UDL one last time, Edyburn closes his article with caution: “Unless serious intellectual energy is devoted to addressing the current shortcomings of the UDL construct, within the next 10 years we may be commemorating the passing of another education fad.” From my perspective 10 years later, the UDL fad hasn’t passed. There may be good reason for that: it offers an attractive model for addressing equity and inclusion in education. It is premature to reject the model in its entirety. But the science hasn’t much improved either.

I think the unfortunate reason for that lack is progress is that the field of education suffers from a crisis of authority even worse than its replication woes. Under the thrall of scientism, educational researchers attempt to justify their professional value by producing “best practice” guidelines analogous to those in medicine, despite weaker evidence. However, the weakness of that evidence in turn affords the research itself negligible persuasive power, leaving educationalists little recourse but to appeal to their credentialed “expertise” in “the science” – in other words, their mere authority. Trust us, we’re the (Ed) doctors.

They can get away with it because even if they understand sound social science methodology (which, with some of my professors, I doubt), they don’t teach it often in teacher training programs. Thus, without a skeptical audience, they can continue to make their arguments from authority unchallenged, all the long patting themselves on the back for teaching critical thinking skills. For the students’ part, they can feel good about their induction into a professional community so ordained by an “expert” — Why yes Herr Professor, we must think for ourselves! – even if they haven’t learned anything of demonstrable value.

I’m just ranting now. But I wonder: by the non-standards of autoethnography (n = 1), does this post qualify as research?

Ugh. If you can’t tell, I’m still angry.