by Stuart Singer

Many times in a variety of venues I have expressed my firm belief that end-of-course barrier exams are a step in the right direction for education.  In a previous post on this website, A Good Starting Point But Not Quite the Finish Line, I wrote that while such tests have structural flaws, their implementation has created positive outcomes that include establishing uniformity in many curricula, creating a standardized evaluation of student comprehension, aiding in the evaluation of teacher performance and as a component in awarding federal funds for systems that demonstrate significant and measurable improvement on these tools (U.S. devises scoring system for school reform contest).  Even with all of those constructive improvements in mind, I was stunned when I read a quote from the Virginia Superintendent of Public Instruction Patricia I. Wright in the February 10 Washington Post article, Md. tops U.S. in one measure of AP test performance.  Referring to the improvement in AP tests scores in her state Ms. Wright said:

“More of our young people are ready for the challenge of AP courses because of the Standards of Learning program (Virginia’s end-of-course exams) which has raised the instructional floor for all students."

How can osuch lofty accomplishments be ascribed to a fifty multiple-choice question test that is designed to ascertain the level of understanding obtained from 160 hours of classroom instruction in basic core subjects? It is obvious to me that Ms. Wright needs to curb her enthusiasm a bit.

You Say Mastery, I Say Minimal Competency

Apparently Ms. Wright is among the group of educators who believe that passing the Virginia SOL Exam demonstrates competency in a course and passing with a relatively high score indicates mastery.  While it would be wonderful if that interpretation were accurate, the facts reveal a very different story.  The Virginia Standards of Learning Exam is a collection of fifty, four-option, multiple-choice questions, in which the results are “normed” in much the same manner as the SAT.  This norming is based on the number of correct responses (no penalty for incorrectly guessing) and translates into SOL scores ranging from 200 to 600.  Four hundred or more earns one a “pass”; five hundred and above delineates advanced status.  To graduate, a student is required to pass the two English components and an additional four tests from the other three curricular areas including at least one from each discipline.   To be awarded an “Advanced Diploma” one must pass a minimum of nine.  But does this system of accountability really demonstrate what its advocates claim—a mastery of the subject matter being covered?

It Is All in the Math

Students, parents, teachers and administrators have continually misinterpreted SOL results. Understandably, when looking at a scoring range of 200-600 a result of 450 is viewed with great favor.  Consequently, when I refer to the SOL program as a measure of minimum competency rather than any demonstration of deep understanding, I have raised the blood pressure of many of my colleagues and supervisors.The strength of my argument rests squarely on the principles of mathematical probability.  The major weaknesses in the SOL testing system are the exclusive use of multiple-choice questions and the grading scale being employed.  A multiple-choice format without penalties for guessing has a strong propensity for misleading results.  This dilemma can be illustrated by a demonstration I give on this subject.  The process is simple—an individual is given a blank bubble sheet and then instructed to randomly fill in fifty answers with an A, B, C or D.  Then any one of the SOL tests is selected.  When the “blind” answer sheet is compared with the correct answers for the chosen test it will on average generate slightly more than twelve correct answers.  This result of twenty-five percent accuracy regardless of the questions or the answers is to be expected based on the laws of probability (note key word—laws).   Consequently, every SOL answer sheet will have about twelve correct answers even if the test-taker’s knowledge of the subject is zero.  Now the first step on the slippery slope of interpreting scores is about to be taken.  A dozen “correct” responses on a typical SOL exam translates into a score of about 325.  Remember that range of 200-600?  In reality the actual numbers should be 325-600.  Suddenly the use of 400 as a “passing” score becomes less credible.  Unfortunately, the process gets worse.  

On many SOL exams thirty correct responses equals a score above 400. Some require a few more, others a bit less.  So how much “mastery” of a subject is required to get to thirty correct answers?  If a student knows only half of the required curriculum, success is easy.  These students will accurately answer only 25 of the 50 questions.  Of the remaining 25 questions probability predicts at least six additional correct answers giving a raw score of 31.   On the 2006 Algebra 1 SOL that score would earn the student a result of 422.  Ironically, a score of 50% on virtually any classroom endeavor would be considered a failure.  (For additional scenarios see attached “How SOL Scores are Created”)  In reality even this example is understating the final score.  Placing the correct answer in the array of choices will skew the results slightly upward.  Of greater significance, if the test taker can eliminate one or more of the wrong answers the probability of a correct guess increases significantly.  Working backwards with the four answers that have been presented can often solve questions involving computation.  

After I gave this presentation at a PTSA meeting a parent approached me to say, “No wonder my son struggled so much in Honors Geometry.  I did not realize that his Algebra 1 SOL score of 460 was not what I thought it was.”  She was not alone in her misunderstanding of the numbers.  Virginia is not the only state with a “barrier-testing” program beset with these problems and comments like the one made by Ms. Wright add to the confusion.  This lack of transparency and misleading comments can lead to decisions resulting in significant future academic damage to students.

Doing It Right

What set of circumstances would allow Ms. Wright and others to use the word “mastery” when discussing these tests?  How can a state create a system of exams that would actually lead to more college-level thinking?  Here are some suggestions:

- Devote the resources necessary to upgrade the quality of the testing process.

- Be transparent and realistic.  Include input from teachers, administrators, and college admission officials.  Educate the community concerning the intent of the program. 

- Be wary of No Child Left Behind—any program that mandates a 100% pass rate in 2014 should not be a factor.

- Create tests that parallel AP and IB exams.  Replace multiple-choice questions with short-answer inquiries.   Have a portion dedicated to essay or multi-step problem solving.  Eliminate any advantage for guessing. 

- Do not norm the results.  Establish a criterion-referenced test with realistic scores and produce students who can achieve them.  For starters establish a score of 95% for mastery, 80% is competent, 66% will be passing and anything less is failure.  In the real world being wrong one third of the time is not a formula for success. If 95% of the test takers pass under these criteria everyone can be happy.  If 65% fail, do not lower the standards.  Instead, find ways to improve the knowledge base of your students. 

While it is wonderful that AP and IB test scores are improving in Maryland and Virginia, the root cause of this improvement cannot be attributed to Standards of Learning Program.