Stephen Downes points to Chad Adelman posting on Crossing the Finish Line – a recent book about university graduation rates by William G. Bowen, Matthew M. Chingos & Michael S. McPherson.
I haven’t read the book but am suspicious of any attempt to draw conclusions about social policy from statistical analysis – especially in reviews and commentaries that isolate particular statements about how variables are correlated (and even more so if they include references to “predictive power” going “below zero”, since statistical power is defined as a probability and a correlation of minus one has a very strong predictive power in any reasonable sense of the term).
A common “paradox” pointed out to students in an introductory statistics course is that it is possible to have a variable S (for, say, SAT score) that is positively correlated with some measure, say G, of success (eg graduation) in each of several subsets making up the whole of a population – while being negatively correlated in the population as a whole.
One way this might happen, for example, would be if there was a characteristic I (for, say, Inspiration) which was very highly correlated with G, and such that among the high I part of the population S was only weakly correlated with G but in the low I population S was very strongly correlated with G.
If among the population as a whole (in this case university entrants) low I was correlated with high S, then entrants with high S would be more likely to be in the low I group and so less likely to graduate and so S might be negatively correlated with G – even though in each of the low and high I groups separately, higher S does contribute to increasing G.
Of course, many readers of this (if in fact there were any) might then say “but if I is the best predictor of G, let’s just use it and forget about S”.
And maybe they are right. At least if the goal is soley to maximize the G rate we should just ignore the low I group and concentrate all of our efforts on those with the magic I factor.
If only we could identify it we could do away with all that “high stakes testing” and give our attention to those who deserve it.
Well the good news is that I have found I.
The bad news is that it is not Inspiration.
It is parental Income.
(And the reason S and I are negatively correlated is because low S students have more chance of getting into the population of university entrants if they are lucky enough to have high I)
My reason for posting this is just to inject a note of caution about throwing out those “horrid high stakes tests” without being clear as to how they might be used more positively. For example, if they were used as an alternative admission path as opposed to an additional requirement, then they might be providing an entry option to students with high talent but low high school gpa due to poor family circumstances.