Royal Grammar School


Selective with statistics

Selective with statistics

I used to have a habit of airily quoting statistics, usually superficially and frequently incorrectly. I was weaned off that tendency mainly by our own Mr Bill Gibson who has the real statistician’s horror of loose or sloppy use of figures and used to put me right, tactfully but firmly, when I’d finished speaking. I think I’m now cured.

The temptation is great and, as a former addict, I’m nowadays sensitised to facile statistical assertions when I hear others make them. The thought returned to me when, on Tuesday evening, I found my way through the maze of medieval passages that is London’s Middle Temple to hear Sir Michael Barber give the National Education Trust’s Mike Barber Memorial Lecture 2014.

Sir Michael has recently joined global education business Pearson as its Chief Education Advisor, having spent some years as Head of McKinsey’s global education practice. Those of us working in schools remember him better as 1997-2001 Chief Adviser to the Secretary of State for Education, when David Blunkett held that post, and after that as Head of Tony Blair’s Prime Minister’s Delivery Unit.

He gave a fascinating overview of education policy from 1944 to 2024 [sic]. I confess I recall the Blair years as a period when schools, teachers and their leaders were bullied and driven more than ever by a new government, a trend that shows no sign of slowing under the Coalition (or its potential successors). But hindsight is a wonderful thing. Sir Michael now sees “pressure” and targets as necessary means for achieving only the first stage of consistent systemic improvement. Pressure and targets, he said, won’t bring about greatness, the next desirable stage: no, greatness has to be “unleashed”.

Wow! He’s starting to talk my language, at last. But then, he’s comfortably on the sidelines now, and sees things from a very different angle. Something that struck me was his assertion that he has always been (rightly) keen that any change or initiative should be supported by an evidence base. Then he introduced his audience to something that I always suspected, but never actually knew about – the McNamara fallacy.

This is how Wikipedia defines it:

The McNamara fallacy, named for Robert McNamara, the United States Secretary of Defense from 1961 to 1968, involves making a decision based solely on quantitative observations and ignoring all others. The reason given is often that these other observations cannot be proven. (See the example below.)

It refers to McNamara's belief as to what led the United States to defeat in the Vietnam War - specifically, his quantification of success in the war (e.g. in terms of enemy body count), ignoring other variables.

The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide.

My memory over more than two decades of headship suggests to me that successive governments – including the one Sir, then plain Michael, Barber served included – rarely if ever avoid the McNamara Fallacy. Someone who was for me an educational guru and sadly died in January, Professor Roland Meighan, used to sum up the result of these blindspots: “We’re getting better and better,” he’d say, “At doing the wrong thing.” I miss him.

So many wrong educational turnings (some but not all of which we independents can fortunately ignore) were and are taken by policy-makers who are influenced or claim justification by selectively quoted statistics – or bogus figures, as my friendly statistician colleagues might say. There was a good example of that, just this week.

The Sunday Times got the story first, under the headline State pupils get university leg-up. The sizeable lobby that claims independent schools hog all the best university places were cock-a-hoop when research by the government’s own university funding body reported that at degree level former state school pupils were outperforming those who had been privately educated by 8%. It was, of course, grist to the mill for those (increasing numbers even in the Tory party) who seek to deride the “unfairness” of independent schools and of the success of the boys and girls who go to them.

Only that wasn’t quite what the research said. Those who shouted loudest either didn’t read the small print or willfully chose to ignore it. The detail revealed that, where candidates were admitted to university with grades around the BCC mark, those coming from state schools did indeed get better degrees by a percentage of 8.

But at the A*/A level there was no difference. None at all.

That suggests to me a rather different interpretation.  In my view (I fear I’m about to indulge in fairly wild generalisations) the research demonstrates that less strong A level candidates tend to be extremely well coached in independent schools thanks to whose efforts and expertise they gain university entrance grades. I find it entirely credible that candidates who haven’t benefited from such intense support (though let’s not start running down state schools here!) may gain similar grades but have a greater level of innate potential/ability and thus progress 8% further on average at university.

And what of the A*/A candidates? The strongest students tend to thrive in most settings. Moreover, top grades don’t grow on trees, and I don’t believe that even the most ruthless or intensive coaching (which is how independent school A level teaching is sometimes caricatured) can help a middle-ability candidate to perform as if they are top-level (sorry about those rather coy descriptors). 

Independent schools – the RGS and lots like us – tend to have such expertise at A level teaching that A-grade candidates can be well assured of gaining the grades they should. We don’t pretend to do the impossible, but we do assure students and their parents that their talents will be developed to the full and that preparation for the exams will be painstakingly planned and professionally executed: that way lies the dependable success that sometimes looks like a miracle but is in fact the predictable fruit of a professional job excellently done. At the same time, we provide a fantastic all-round educational and social experience on top of those results.

So will this allegedly ground-breaking research change the university landscape for RGS students? I don’t believe so.

We’re fortunate at the RGS: our students tend to be applying in the region of those top grades where the research found no difference in performance. Moreover, while the “recruiter” universities may use that research in justification of differentiating the offers they make between state and independent schools, the “selector” institutions (Oxbridge, the Russell group, medical schools) will always seek to attract the brightest and best, wherever they come from. It’s what they’ve always done, and what defines them. They dilute that at their peril. Indeed, I was reminded at a meeting of leading independent day schools yesterday that the Russell Group universities really are our friends: they desperately want our excellently-prepared, bright and highly-motivated students, and keep telling us so.

We independent school heads, usually speaking through our professional associations such as HMC, have never quibbled over universities making lower offers to candidates who demonstrate great potential to them but whom they feel circumstances (including current school) might cause to underperform at A level. We only protest when a blanket prescription of lower grade requirements is proposed for all candidates from a particular type of school or social background without the proper interview or selection process to inform such a judgment (we might ourselves make a similarly informed decision on a less well prepared candidate for entry to our own schools),  

Such proposals generally stem from an ideological dislike of our sector (what the Master of Wellington, Dr Anthony Seldon, described last year as the hatred that dare not speak its name), masquerading as affirmative action for social mobility. You’ll see I’m unconvinced. Given that hostility, though, should parents play smart and send their children to state sixth forms in order to avoid universities’ anti-independent bias.

No. despite all the rhetoric and hot air, that bias simply doesn’t exist in the top institutions, though competition for places at them will remain as tough as ever, if not tougher. And, given that the grades demanded will remain high (if not higher), parents and students will be best advised to stick with a school that they are confident will ensure that they attain those grades.

Well, you might say, he would say that. Yes, I would: it helps that it’s true, though, and not based on selectively plucked headlines or incomplete data that just happens to fit a particular agenda of the moment.

No dodgy statistics involved in our claim, and certainly no risk of a McNamara Fallacy. We publish the grades our candidates achieve, their university destinations and the courses they follow. Nothing selective or glossed over there. I just wish there was more integrity in the tactics (and suggested statistics) employed by our opponents.

Bernard Trafford