by Stuart Singer, The Teacher Leader

Educational reform has always been popular cocktail conversation.  While most people do not understand Wall Street derivatives and very few have expertise on medical science or the cause of automobile accelerator problems, everyone has been a student and they all have an opinion on what is right and wrong with our schools.  Recently, however, this discussion has shifted from casual to red-hot.  Normally a campaign issue that fades after the votes are counted, educational reform is now being debated on high-profile opinion shows and the front pages of newspapers throughout the country. The development of the “Common Core State Standards Initiative” (CCSSI), the new “Race to the Top” Obama initiative which ties funding to student performance and the drama of firing teachers by the scores have all brought the analysis of school data to the forefront.

Lies, Damn Lies, and Statistics

In virtually every new plan under consideration the use of statistics is the centerpiece for evaluating students, teachers, schools and districts.  As a math teacher for forty years, I find myself simultaneously applauding this approach while cautioning its utilization.  Far too often, such numbers have been the fool’s gold of education.  The recent firing of the entire staff at Rhode Island’s Central Falls High School demonstrates the danger inherent in the use of statistics as an evaluative tool.  As Valerie Strauss documented in her recent post, “Teacher firings and Obama comments stir serious backlash” the data used to condemn the school were misleading.  One of the primary complaints against the personnel was the low graduation rate.  The methodology of computing this number included counting as “dropouts” a significant number of individuals who were deported as illegal immigrants.  Also included in this category were students who transferred to other schools in order to enroll in honor programs after such courses were discontinued at Central Falls.  Not included in the analysis was an assessment of the demographics of the school’s student body compared to others in the state.  This staff was working with a significantly higher rate of free and reduced lunch, ELL, and special needs students than the norm in the state.  Mix in seven principals in the past six years and there is little wonder that the school struggled.  While this additional information does not by itself change the belief that Central Falls has serious problems, the greater concern is that, if future plans are to use data in more sweeping evaluations, such measurements must consistently and accurately reflect the performance of a school.

Evaluating Statistics is Not Intuitive 

I recently asked my son who is a senior vice-president of marketing for a large international bank about the use of student performance statistics in education.  His principle warning was to recognize that raw data by itself is generally useless.  He added that the true value of such numbers is when they are placed into context with multiple measures, and, even better, when used in regression analysis to truly understand what the data is saying.  His advice reinforced what I had discovered when I created a statistical review during the 1999-2000 school year.  It was the fourth year of state barrier exams and the basic numbers were not particularly favorable to our school.  The county ranked the twenty-four schools in the system based on the total score created by adding the pass rates in the four subject areas (Math, Science, Social Studies and English).  Not surprisingly my school with the highest rate of free or reduced lunch (54%) was positioned at number seventeen while the school with the lowest rate (1%) was at the top.  Staff morale was low—the teachers believed they were doing far better than that placement would indicate and I often heard derogatory comments about the top-ranked school—“Of course they were number one.  What would you expect with that student body.”

A Colorful Comparison

With the prodding of my principal I took a look “inside” the numbers.  I began by color-coding in red the schools with the twelve lowest free and reduced lunch rates and the highest dozen in blue.  Not surprisingly, the top half of the county score chart was entirely red and the bottom completely blue.  Clearly success on these tests was directly correlated to the financial status of the student body.  But the goal of my research was not to prove the truth of this long held belief; the task was to determine how well each school was performing based on their unique circumstances.  A formula was developed comparing the relationship between the raw scores and free/reduced lunch levels.  A line of regression was created and the equation of that line would produce a predicted total score for a school based on its demographics.  The correlation coefficient (basically its accuracy reliability) of this tool was determined to be 0.9.  Since 1.0 represents perfect correlation (i.e. the sun rises in the east) this value represents a very high degree of accuracy. Using this information it was determined that a school with a 6% free/reduced rate “should” score 354 while one with 26% would be expected to have a 335.  The rest of the analysis was simple.  Every school’s “expected” score was computed, then that number was compared to their actual result.  If the school at 6% had a total of 351 it would be given a –3.  The 26% had a 341 that translated to a +6.

A Beautiful (and More Meaningful) Blend

The twenty-four schools were ranked with these new numbers.  The revised chart was a montage of colors.  The top half now had five blues and seven reds.  My school was ranked number one and the previous leader was second.  I gave a thirty-minute presentation on this process at a faculty meeting.  Despite the density of the math, the audience was unusually attentive because of the relevance of the topic.  More importantly, two critical conclusions were clearly ascertained.  Based on this analysis of multiple, relevant sets of data, our staff was doing extremely effective work with our unique student body and that other school, although possessing a wealthy student body, was also performing at a very high level.  Validation was given to two schools at opposite ends of the demographic spectrum. 

This particular study represented one small statistical interpretation of educational scores.  It was neither sophisticated nor broadly based.  But it did have the capacity when used over a period of time to more accurately demonstrate a school’s actual trajectory in terms of this set of exams.  This process was only presented to the faculty for one more year.  It was, however, a clear demonstration of the power of precise mathematical analysis.  The staff gained significant reassurance and confidence from these results and quickly elevated their goal to having the best scores in the county regardless of socio-economic level.  Likewise, the administrative team acquired a better understanding of the success being attained within the school.

If comprehensive data analysis is to become a critical component in crafting future educational policy extreme care must be taken to ensure that these powerful tools are used correctly.  Procedures should be in place requiring that context, consistency, and research are utilized extensively when creating any measurement.

Next:  A Plan for Standardizing Data