The paper “The Problem With “Proficiency”: Limitations of Statistics and Policy Under No Child Left Behind” by Andrew Dean Ho published in Educational Researcher, Vol. 37, No. 6, pp. 351–360 (August/September 2008), makes the salient point that interpretation and use of cut scores, in particular the percentage Proficiency scores (PPS), gives inherently misleading accounts of progress and comparisons among groups.
The problem is not caused by intentional bias on the part of psychometricians or those determining the cut scores, though Ho does imply it could be used, and perhaps has been used, by those knowledgeable of this problem to give biased results.
In the following I summarize the problem he describes and later reference a Mathematica CDF file I will design and make available that simulates, dynamically, the effects of cut score placement and test score distribution changes to illustrate the principles discussed.
Assume Cut1 is at -1.5, and Cut2 is at 0.5. Then imagine two different scenarios. In the first scenario, assume the student 4th grade math WKCE scores fall at Cut1 for first year 1, and in year 2 the 4th grade class scores are 0.5 stdev higher. Visually, the cut score holds its position on the horizontal line while the whole bell curve shifts to the right by 0.5 stdev. A comparison between year 1 and year 2 would show a 4.4% improvement as this group would move into the next category. Cut1 would now strike at the -2.0 stdev point. Note also that no improvements will be recognized for about 95% of the students, yet all improved by the same amount.
In the second scenario, the students scores fall instead at Cut2, the first year, and in the second year, Cut2 falls at the 0.0 stdev point due to the 0.5 stdev improvement. In this case the year 1 and year 2 comparison would show a 19.1% improvement, 81% showing no improvement.
These scenarios show that for the identical improvement, that is all kids improved by 0.5 stdev in both scenarios, the perceived improvements show mediocre in scenario 1 and spectacular in scenario 2. Neither interpretation is wrong, but the results are grossly misleading. If one makes teacher evaluations, or AYP decisions based on either scenario, then the resulting consequences will be wrong in both scenarios.
The key understanding one needs to take away from these scenarios is for any cut score like Cut2 sitting right of the bell curve peak (the mean, median and mode), as the bell curve shifts smoothly right comparisons will make it seem that the schooling is getting better by the year, that the schools and teachers have found the magic solution. This will be a false conclusion. Once the bell curve moves to the point where the cut score is to the left of the peak, the rate of improvements will decrease, rapidly at first, decreasing less rapidly as the bell curve shifts further right. Any conclusion that the teachers are losing their edge, the curriculum needs to be changed or some heads must roll would be wrong. All such effects seen are an artifact of cut scores’ interaction with a bell curve, and nothing more.
The above logic applies to every cut score and every demographic subpopulation. Basic, minimum, proficient and advanced will be different as will rates for different ethnicities as will rates for different schools and school districts. Without more, interpretations are guaranteed to be wrong.
There is a rule here that must be exercised. No statistic can be understood unless and until it is related back to the original data. That is, in order to make sense of any statistic, and in particular, test outcomes, it is necessary to have either the full distributional information (the original scores) or the basic distribution statistics such as counts, mean, median, variance, skewness and kurtosis for each category and subgroup that would allow each of the distributions to be simulated.
TO BE CONTINUED
TJ Mertz Facebook linked to a Diane Ravitch post accusing the President and Arne Duncan of aligning themselves with right wing conservatives in their goal of destroying the public school system. See DianeRavitchWhoseSide. I disagree with Ravitch’s and Mertz’s suggestions.
I do assume Obama wants and expects Blacks and other minorities to get a good education. If one isn’t assuming that and considers oneself liberal or progressive, then you’re missing the important issue. When the President backs positions of the reform movement, even the right-wing allies, I have no doubt that he is not doing so because his strings are being pulled by his puppet masters. I certainly do feel that way about Republicans or other corporate types who take the same positions, but that is a different question — I assume evil intent and I think there is far more evidence than needed to support that position. However, the President’s positions are a different matter entirely.
The President does send his kids to private schools; he knows that schools can teach and kids can learn at a high level. He knows what it took for him to get an education. His mother homeschooled him when they were living in Thailand and his mother and especially his grandparents sent him to elite schools in Hawaii, and sacrificed to do it. He is also assuming that if he could get a good education, so can other Blacks.
Let’s put Obama, the man, into some context. We start by admitting to the obvious. Obama is self-assured, and has a healthy ego; no one has the temerity to run for President without an incredible ego, especially a Black man with a Kenyan name. Unlike a Romney, born into a extreme wealth and a warped ego who believes this raises him above the common folk, Obama believes others can rise to similar heights as he if given the opportunity. What is lacking is opportunity, Obama believes. If you do not agree with Obama’s sentiments on this issue, I think you have a lot of explaining to do as a liberal or progressive. Obama’s passion and belief does not come through as President but his writing, his biographers and those who know him as the man, have said as much. He also knows that most minorities do not have the options for a good education that he and his kids had (and Michelle). He knows that it can be done, he experienced it, and he knows that most public schools are not getting it done.
However, I have no doubt that the private school movement, charter schools, and vouchers will not work either; there is no question that the corporations want to milk the system for profit. There also no question that no one who is or has been involved in making education decisions knows what they are doing. That should be obvious, otherwise the failure to educate Blacks and other minorities would be rare, not typical.
We have had at least 45 years of consistent failure, at least nine 5-year plans that had promised to turn it around have passed without success. Why should we believe the tenth 5-year plan will succeed, whether charter, private, public, or vouchered.
I take it for granted that the President doesn’t know what the solution is; he is relying on Duncan, and Duncan doesn’t know. But the President does know, as we all must, that what is being done is not working. Public schools are simply not getting it done, and history shows they never have. Yes, the attacks on the public schools are making it less likely that they can turn it around, but where is the evidence that but for the attacks, they would be doing a good job? There is none. Should we then send our hard-earned tax dollars to private or charter or vouchers? No, and for the same reasons. There has been no evidence that the educational establishment or any implementation has made a difference, at least any that can be replicated at the scale necessary to raise the level of educational attainment that is expected and needed.
So let me answer the question which Mertz and Ravitch raised as to whose side the President and Duncan is on. Pubic Education. That is “Education of the Public”, which I suggest Thomas Jefferson espoused. And no, support for “Public Schools” is not the same as support for “Public Education”.
Sadly, however, the President will be disappointed because there are no viable institutions that will successfully support public education for all. Those who spend their time in political and financial posturing, like Ravitch especially, are not creating institutions that will succeed. Politics is the game that is being played, not education.
Click the following link to display the graphics in PDF format. Click here
Clicking the following link will open a new window containing the dynamic graphics discussed. The graphics require the Mathematica CDF player browser plugin to be installed. If you have not done so, go to the Downloads page to install. Then return here and follow this link. The process may take some time. Years In MMSD CDF
Comments on Years in MMSD Data
That data on which this post is based was first posted on Jim Zellmer’s School Info Systems at Madison’s Transfer Students and the Achievement Gap. The writer, Andrew Statz, Executive Director of Information Services for MMSD, concluded the results were largely ambiguous as to whether years in MMSD schools had a positive impact and illustrated the data using stacked bar charts. I used his data to create different graphics that I would more easily understand. As a consequence, I found the results less ambiguous, as I explain below.
I have to admit that I cannot read stacked bar charts, and I cannot see patterns in them that might be present. Instead, I have represented the same data as a sequence of bar charts for each year students spent in MMSD schools displayed as percentages within the year.
In these histograms, I was able to see some non-ambiguities that I could not see using the stacked histograms. I make some observations using the percentage histograms, understanding, of course, that the significance of these observations might be unstable given the relatively few students who transfer into MMSD as compared to those who enter school as soon as they come of age. A further problem is the data is not longitudinal — the students represented in each year are different; we are not seeing how the set of students progress in their education as the years go by.
What We Would Like to See in these Histograms
Though the students depicted within each row of histograms are different students, at some risk, we might want to assume they represent the same sample population over time. Making this assumption, the pattern we hope to see is a distribution skewed left; that is, the histogram will show few at the Minimum, a little more at Basic, significantly higher percentage of students in Proficient, and higher still at Advanced.
Examples of the acceptable skewness are shown in the 4th Grade Reading Percent histograms for MMSD Years 3 and 4+. In both these cases, the percentages are approximately Minimum at 10%, Basic at 20%, Proficient at 30%, and Advanced at 40%.
Of course, in a more perfect world, the histogram for Minimum and Basic would together total 2%, representing that we may not be successful those students starting more than two(2) standard deviations below the mean, Proficient at 13% and Advanced at 85%, representing that we can educate students to Proficiency if they lie between one(1) and two(2) standard deviations below the mean, and we can educate to Advanced for everyone else.
In order to demonstrate students are being well-educated in MMSD schools compared to other school districts, the overall pattern we would expect to see is students coming into the district (at MMSD Year 0) with a relatively nondescript distribution, and as the years in MMSD increase, the distribution becomes increasingly skewed left, as described above.
Comment on the Histograms
The Histograms show that students at MMSD demonstrate much less progress in Math than in Reading as measured by the WKCE. This should not be surprising given the general math and science illiteracy in the US (approaching 95% by some measures) in the adult population and the TIMSS and PISA score comparisons of American students with international students over many years. But, further, members of the administration I have talked to indicate that MMSD is focused mostly on reading and putting much less emphasis on mathematics.
The second impression is that students scoring at Minimum or Basic in Math remain in these two subgroups throughout their years at MMSD. Likewise, students scoring Proficient or Advanced in Math remain in these subgroups. Further, MMSD students seem not to be progressing from Proficient to Advanced; the percentage of students at the Advanced level generally remains lower than the percentage at Proficient. Both these observations perhaps illustrate the current and damaging attitude that in Math, you either have it or you don’t.
4th Grade WKCE Scores
In Math, we see an increasingly skewed left distribution involving Basic, Proficient and Advanced. This is good, showing that these kids are moving up the scale toward competence. However, students at Minimal remain stuck; the histograms show MMSD is making little progress for kids at the Minimal competency level and this remains true regardless of how many years the students are attending MMSD schools.
In reading, MMSD’s data looks good. However, those entering MMSD for the first time (MMSD Years = 0) in 4th grade already show a reasonably skewed left distribution, implying to me that reading is being focused on relatively successfully in other districts. There is an anomaly for MMSD Years = 2. These kids entered MMSD in the 2nd grade and had only 20% scoring Advanced and 45% scoring Proficient; this is somewhat reversed for the others years. These students seem to have come from a different population.
8th Grade WKCE Scores
The 8th Grade Histogram sequence contains MMSD Years = 2 which are students entering MMSD in the 6th grade, the beginning of middle school. The histogram shows an almost Uniform distribution in both Math and Reading, with an ever so slight left skew involving Minimal, Basic and Proficient in Math. Not knowing what these former 6th graders’ scoring distribution was entering MMSD, it’s hard to determine MMSD’s contribution to these scores.
In Math, there seems to be nothing ambiguous about the trends over time spent at MMSD schools in how well such students perform on the WKCE – there seems to be no MMSD contribution. Those students in the Proficient and Advanced levels fluctuate randomly between each other over Years in MMSD, those in Minimum and Basic also fluctuate randomly between each other. What the data seems to show is that those students stuck in the lower two levels stay there, and those in the upper two levels stay there, with the variation we see perhaps little more than students at the cusps between Minimum and Basic or Proficient and Advanced scoring just a little better or a little worse.
In Reading, there is no indication of progressive improvement to scores based on the number of years in MMSD, though the reading distributions look better than the math distributions; that is, there is a faint tendency to left skewness over time.
10th Grade WKCE Scores
In Math, except for students entering MMSD in the 9th grade (MMSD years 1), the distributions show right skewness, opposite of what we want to see. We don’t see improvements until the students have been in MMSD schools 5 or more years, then the change is modest at best. Students in the Minimum and Basic groups show no consistent improvements.
In reading, we see left skewness consistently only for those in MMSD since the 3rd grade. Those students entering MMSD between the 4th grade and 8th grade don’t show progress. Those entering MMSD in the 9th grade are from a different population of students than those entering at other times.
Lack of progress in math is disturbing. Worse still, is that MMSD’s focus on reading alone, I claim, is a substantial cause. The way math in general is taught focuses on “genuine” problems. Genuine problems heavily emphasize solving problems written in English and therefore the focus on reading English is made the first goal. This is a problem. Mathematics is its own language and the focus on teaching mathematics should be to teach that language. I contend that only basic competence in English (or any language) is required to teach and understand the language of math.
This belief is bolstered by the following. In college in the 1960′s, it was expected that those majoring in the sciences or math would learn either technical French or technical German. I still have my technical German language book that allowed me to read, quite slowly, computer science research papers written in German. There was never an emphasis to learn to speak the language. A number of those in computer science studying for their PhDs at the time were learning French for the same purpose.
Finally, a number of friends and colleagues in my math and science classes were from China, Taiwan, Korea, and India. Their proficiency in English tended to be basic but they were acing these technical classes. These technical classes were taught with emphasis on the math or science, not “genuine” problems requiring English proficiency. Of course, these foreign students by necessity were the cream of their cultures, so none were your average college students.
Nonetheless, at the level of elementary, middle and high school math and science courses, the content of which is in reach by almost any student, there is little evidence to suggest that this content can only be learned by focusing on “genuine” problems using verbose academic English. When taught by instructors who themselves are proficient in math and science, brevity of word is a plus and use of the language and symbols of math and science will allow even those with limited English proficiency to succeed.
In 1999, I ran across an article in AFT’s American Educator magazine by Louisa Moats entitled “Teaching Reading IS Rocket Science”. It can still be found on the AFT site. The title and the article accompanying it struck me as self-serving, creating an expertise requirement for teaching a skill which is as common as sitting on Grandma’s lap while she reads to you, and you read back to her. Grandma was never an expert, but we learned to read from her, and Grandpa, and Mom, and Dad, and Older Brother and Sister. Miraculous as it may seem (and the human brain is truly miraculous), slowly but surely, without pressure, we all learned to read. It came as naturally as learning to talk.
Why this pronouncement from Moats? The answer might lie in her early work, skill and training being in Special Education! I can agree that there are some special cases where failure or delay in learning to read might be symptomatic of some underlying physical or mental problem requiring a specialist trained in the area. However, for normal students the old fashioned sit-on-lap-with-grandma is all that is necessary. So, why extend the need for “rocket science” experts to teaching normal children to read? My guess it’s an example of general societal sickness that now requires the creation of a “client class” for “professionals” to treat, like bereavement specialists. Yes, therefore, we need rocket science reading specialists.
So, the secondary title of my blog of “Education is not rocket science” is a direct consequence of my strong disagreement that what needs to be learned in elementary and middle and even high schools is so complex and beyond the ability of any normal student “rocket science” specialists are required.
What about math and science? Doesn’t that require expertise in math and science? Answer: It shouldn’t. It’s only because 95% of adult Americans are mathematically and scientifically illiterate, by one survey, that we have to rely on “experts”. It is simply a sad commentary on the thoroughly inadequate education system (really the US culture — let’s not put the blame on the schools), that common reckoning (as Thomas Jefferson might have said) and science have not been learned by parents and grandparents and older brothers and sisters such that the material taught in elementary and middle schools seem to require expertise rather than common knowledge.
Likewise for history and biology (life?) at the grade and middle school levels. These topics can and should be so commonly known that any adult could impart this knowledge.
None of this is Rocket Science.