Fools Gold? Why you shouldn’t fund an EdTech RCT (yet)
Randomised Controlled Trials are the “gold standard” of efficacy. Here’s why buying one is the wrong approach.
If randomised controlled trials (RCTs) are the “Gold Standard” of edtech evaluation, once you have a working product - or it’s a blocker to sales - you should probably have one… right?
Not so fast.
All that glitters is not gold - today we’re diving into:
Subscribers get 10-minute nuggets dispelling myths about evidencing educational impact in straightforward language, straight to your inbox.
This is Part 3 - here’s Part 1 and Part 2 if you want to catch up first.
What is a randomised control trial in education?
(Let’s start with what an RCT is → skip ahead if you’re already familiar.)
I used to be a Science teacher, and one of my favourite things was applying what we were learning to what in England is called Working Scientifically.
Basically, experiments and stuff!
But there’s a logic to how experiments are set up - agreed principles for how we do things.
That way, others can look at what we’ve done, see why we’ve made certain choices, and repeat what we did to check the results are similar.
If it’s replicable, it’s reliable.
If we want to know how well something works - its efficacy - then we need to control all the other factors that might influence it.
Simple RCT example: testing the efficacy of plant feeds.
Plants are a lot simpler than people, but there are still some things that affect plant growth that we can’t control, like how good the seeds are.
So we randomly assign a large number of seeds to the different groups, which spreads the variation among the groups, and weigh each group’s seeds to check they’re similar.
We also need a control group where nothing changes because it could be that plant growth was actually better without any feed!
If the seeds in one group were wayyy heavier, they had more nutrients = a better start in life.
They’d be more likely to grow taller, regardless of the feed.
RCTs in education
School-based RCTs control as much as possible by:
Selecting the system, sector, age range of pupils etc, with large enough numbers to see statistically significant effects.
Randomly assigning classes or schools that meet the criteria to receive either the intervention (your product), or the control (alternative or business-as-usual).
Checking the groups are statistically similar in terms of:
Student or teacher characteristics that we know are likely to affect the results e.g. socioeconomic status, ethnicity, prior attainment, teacher experience.
Baseline assessment results - which get repeated later to compare results.
Why are RCTs the “Gold Standard” of education evidence?
A. Million. Pounds. 🤑
That’s how much the UK government estimated it would cost to run an RCT of ONE PRODCUT in their testbed scoping report.
Okay, it’s not really because the cost should be given in gold bars…
The concept of measuring efficacy in education came from fields where it’s easier to run RCTs, or where the margins make funding trials much more profitable1.
RCTs are the most valid in terms of causality i.e. that the intervention was the reason for the treatment group’s higher outcomes, not something else.
They’re designed to minimise sources of bias and error.
EduEvidence’s Gold Certificate for efficacy is only awarded to products/programmes that have shown, using an RCT, statistically significant positive outcomes compared to the control group.
As an education/edtech founder, why shouldn’t you do an RCT?
Besides the cost, “doing an RCT” is putting your coat on with nothing on underneath - it looks okay at first glance, but it’s pretty suspicious if you refuse to take your jacket off!
I’m talking about academic integrity and replicability.
If you had the cash to splash, jumping into an expensive trial is a massive risk if the results are poor.
And even if they were great, what then?
One RCT doesn’t show results are replicable in different countries or in different types of schools.
If the results of a single RCT were statistically significantly positive, I’d be exceedingly curious about what they did.
Basically, I’d want to know if they rigged it or were making the kind of baseless marketing claims we looked at last time.
ICEI (WiKIT)’s newsletter this week was about whether research funded by companies should be written-off because it’s not fully independent.
Their contract terms ensure academic integrity, but crucially, they also make sure the type of study is right for the stage of product development and existing evidence.
They call this “right-sizing” research.
A good RCT builds on findings of smaller studies using more accessible methodologies.
To be blunt, if you still think developing a portfolio of evidence is just to identify appropriate constraints for an RCT, you’re missing the point.
An evidence mindset is seeing research as a way to improve efficacy and make better business decisions.
I came across this EdSurge article on “Debunking the Gold Standard” quite late on in writing this, but it’s a real gem!
Thank you for this post, it offers a valuable perspective on the complexities of RCTs in the EdTech space. From my experience, sustainable improvement in education often comes from thoughtful, incremental changes to systems, rather than immediate, large-scale trials. The challenge lies in how we balance evidence with ongoing development. How do we ensure that we’re building on small, meaningful studies before diving into something as high-stakes as an RCT?