Collegeboard is about to introduce its New SAT in just one month.

These recent events --

• misprinted test booklets

• “anomaly” tests with unusual scales, and

• the past month’s PSAT percentile reporting

-- suggest some opacity and randomness in the SAT scoring and reporting process. Will we be seeing even more of this phenomenon with the roll-out of the completely redesigned SAT next month?

The Misprinted Test booklet scandal(s)

The Known Scandal of June 2015

Many parents of seniors graduating this year heard about or may be aware about the June 2015 misprinted test booklet scandal. There were articles everywhere, from the WSJ to the NY Times. A large number of test booklets were incorrectly printed with the timing for either Section 8 or Section 9; random test booklets stated that these sections were 25-minute sections when they were really 20-minute sections. The problem became one of inconsistent proctoring: some proctors stuck firmly to their cue sheets and dismissed the student who raised his hand to tell the class that his test booklet said that the section was supposed to be 25 minutes, while others gave in to the timing stated on the misprinted test booklets (not knowing about the printing error).

Collegeboard could not fairly score this test when some students were given 5 extra minutes on one of the test sections. CB ultimately decided to keep the test but score it differently. CB chose to score only the first two 25-minute CR sections (48 questions instead of 67) and only the first two 25-minute math sections (38 questions instead of 54).

So what did this mean for student scores? Well, with fewer total questions per section, each miss and/or omit subtracts more points from the top scaled score than it would if there had been the “normal” number of questions scored. This is similar to what occurred on the “old” PSAT. Most of my top tutoring students who were already in the 90th+ percentiles were flat-to-down on this test. I had one fortuitous student who improved almost 200 points, but he was coming from a slightly lower baseline. He was thrilled, but he was clearly the outlier.

There is no question that the students who took the June 2015 SAT were disadvantaged. CB made up for this debacle by offering a free test – the October 2015 test – for juniors who wanted a “re-do”. Little did these juniors know that another scandal awaited… 

The Unknown Scandal of October 2015

What most parents have not yet heard about is the likely misprinted test booklet scandal of October 2015.

I took this test. When I took the test, I noted that the first CR Section 2 had only 23 questions. CB designs some tests to have one CR section with 23 questions and one with 25. This can be a little tighter on the pacing since the 25-question section would also have the back-to-back long passages. But I was prepared after finishing and reviewing Section 2. I was thinking ahead.

When I got to my Section 5 CR, I was very surprised to see that it, too, had only 23 questions. I had never before seen an SAT test with only 65, instead of 67, questions in CR. So I thought, “maybe CB will have 21 questions instead of 19 in the 20-minute final CR section?” But again, that did not happen.

I was mystified when I left the test room, wondering what a 65-question CR test would like for scaled scores.

I received my online scores in 19 days, and when I opened the details, I was shocked to see that there were 67 questions reported on CR, and I was reported to have omitted two of these questions. I called Collegeboard that day for the first of my TEN phone calls over the next two months!

I thought that CB would want to research my complaint about a misprinted test booklet, because, as I told them, I was concerned that if my test booklet was missing questions and printed incorrectly, it was highly likely that this random test booklet appeared across the country, just as the test booklets had in June. CB Customer Service assured me that they could easily find my test booklet to confirm whether my test booklet had missing questions. They told me that they had escalated my concern and that I would hear back from a manager within 7 days.

Did I hear back? No. I called again, and I opened a Case Number with my complaint. I was told that I would hear back from a manager within 48 hours. I received my QAS report in the mail and realized that I had never seen some of these questions. This Section 2 was not the same as the Section 2 of the test that I had taken.

Again, I heard nothing back from CB. One would think that CB would be invested in making sure that it was aware of another misprinted test booklet issue if indeed another scandal had occurred. But one can also see how it would be in CB’s best interests to keep this latest error under wraps. It was to CB’s advantage that the October test is primarily a “senior” test; most seniors don’t care about the details of their score misses or go carefully through the QAS. This potential problem would not even occur to most students.

Eight more phone calls provided no resolution to my complaint. No CB representative could confirm that my test booklet had been found or analyzed. Since I had a perfect 800 score in Math and a perfect 800 score in Writing with a 12 Essay, I think I had been “on my game” that day. The fact that CB knowingly chose to never acknowledge my concerns seems strong evidence that, indeed, there was another misprinted test booklet scandal that took place in October, 2015.

I am not sure if my test booklet was only missing two questions. I believe that the questions that I had were not in the same order as those in the booklet. So, this misprint was likely to cause not only two omissions but also a couple of other random misses in the long passage because the Answer Key did not correspond to the questions in the misprinted booklet.

How many points was this misprint worth? My guess is somewhere between 40 and 80 points.

How significant is this? I can tell you that I have students who take another test to improve a mere 30 points. So, yes, this would be very significant to most of my students.

So, if you are a senior graduating in the Class of 2016, you likely took a Random Walk down the SAT trail. If you were a lucky one, you avoided taking the June or October 2015 tests. If you were not so lucky, you were unknowingly disadvantaged by having taken either or both the June or October 2015 tests.

More on the randomness of CB test scores in my next blog on Anomaly test scales.

I’ll also be addressing the opacity of the recently reported percentile scores on the New PSAT.