The Tests and the Brightest - 80.06       F E B R U A R Y 1 9 8 0 Each year some 2.5 million high school students match wits with the Scholastic Aptitude Tests. The results go a long way toward determining who gets into the most selective colleges. The tests are the subject of a growing debate. Do they really discover the best and the brightest? Or do they chiefly identify the richest and the most expensively educated?by James Fallows ON a bright weekday morning last October, with the school year just begun,several dozen fresh-faced boys sat crouched over little desks. The boys werefifth-formers -- high school juniors -- at St. Albans, a private Episcopal schoolin Washington, D.C.; the object of their attention was the PreliminaryScholastic Aptitude Test, the first stage in the examinations known to millionsas "the College Boards."Like other test-takers, they looked tense and worried; as the examination bookswere collected and the No. 2 pencils put away, they gathered in clusters todiscuss their answers. "How many intersections are there between three parallellines and three non-parallel lines? You said twelve? God, I could only findeleven." But their worry was within bounds; St. Albans boys always did well, ontests and in college admissions. The year before, half of the St. Albans seniorclass had scored in the mid-600s or above on the verbal portion of theScholastic Aptitude Test. On a scale running from 200 to 800, this placed themabove 97 percent of their contemporaries. Their scores on the mathematicalportion were even better. The richness of their education gave them SAT-styleskills -- close reading, puzzle-solving, facility with the nuance oflanguage -- almost as an incidental benefit. In one room, seniors would discussthe mathematical formulae of Leibniz while, next door, juniors analyzedVoltaire's parody of Leibnizian optimism in Candide. By comparison, antonymquestions and reading-comprehension passages were a piece of cake.Return to Flashback: Educational MeasuresDiscuss this article in The Body Politic forum of Post & Riposte. Shortly after the fifth-formers finished their examination, a calculus teacherjoked with one of the sixth-form (senior) boys. "Three touchdowns in thefootball game, straight A in English class, 750 on the SAT -- Jones, you'reHarvard's dream." "Seven-eighty," the boy, smiling modestly, replied.A few weeks later, elsewhere in the District of Columbia, students from a largepublic high school took the SAT. The city's high schools have beenoverwhelmingly black and lower-class since the court decisions orderingdesegregation, busing, and an end to the "tracking" system; many affluentfamilies, black and white, have moved to the suburbs or send their children toprivate schools. Median SAT scores in the District -- when the school systemchooses to reveal them -- are 200 to 300 points lower than those at St. Albans.Several years ago, there was a minor scandal when the valedictorian of a localhigh school, a straight-A student admired by all he met, was refused admissionnearly everywhere he applied to college, so miserable were his SAT scores.A number of students left this test center early, ignoring the test booklet'sadmonition to use extra time to recheck answers. Large portions of their answersheets were blank. One black teenager pulled on his Washington Redskins jacket,lit a cigarette, and joined a group of friends. "Shit, man," he said, "what'sthe point?"Each year, more than 2.5 million students sit hunched in testing centers,taking the PSAT, the SAT, "Achievement" and "Advanced Placement" tests to giveadvanced standing in college, admissions tests for law and medical schools, theGraduate Record Exam (GRE), and dozens of smaller examinations. Most of theseare written, scored, and controlled by the Educational Testing Service ofPrinceton, New Jersey (which, contrary to popular understanding, is notsynonymous with The College Board. The Board is an organization of more than2500 colleges, schools, school systems, and education associations, whosemembers hire the ETS to write their tests). For its work, the nonprofit ETStakes in about $80 million a year.The SAT has been part of American education for more than fifty years; the ETS,for more than thirty. But never have the testers been more under siege than inthe past twelve months. Last November, a few representatives of the ETS and TheCollege Board, standing incognito in the back of the crowd, heard one speakerafter another lambaste them at the National Conference on Testing inWashington, D.C. "The dishonesty of the ETS reminds me of the armsmanufacturers," said Terry Herndon, executive director of the NationalEducation Association. "They say guns don't kill people, people kill people.The ETS says it's not their fault if the tests are abused -- someone else is toblame.""We've always been told that the tests might have problems, but we had to livewith them, because they're the best we have," said Anna Kahn, nationalsecretary of the PTA. "Now we're beginning to question that.""Standardized tests are used from the cradle to the grave, to select, reject,stratify, classify, and sort people," said Gerda Steele of the NAACP, "and theyare used in ways that keep certain segments of the population from realizingtheir aspirations. Most of all they limit the access of blacks and otherminorities to higher education.""Ninety million lives have been affected by data collected by the ETS,"concluded Herndon, building toward a crescendo that brought a roar from thecrowd. "They're unaccountable to the political community, unaccountable to theeducational community, unaccountable to the legal community. Something must bedone!"The something in most minds that day was a national version of the "truth intesting" law (a nickname the testing industry abhors) passed in New York Statelast year. Under that law, students who take the SAT and similar tests in NewYork this spring will, for the first time, receive corrected answer sheets andcopies of the test booklet a few months after the test date. This conference,one of several organized by a group called Project DE-TEST (for DE-mystify TheEstablished Standardized Tests), was designed to build support for the effortsof Democratic Representatives Shirley Chisholm, Theodore Weiss, and GeorgeMiller to extend similar disclosure provisions nationwide.Even more, the speakers at that conference hoped for a basic challenge to therole of standardized tests in American education. They are saying, in effect,that the SAT -- like the IQ tests that precede it in elementary school, and thegraduate school admissions tests that follow later on -- is fundamentally unfair;far from serving as agents of diversity and social mobility, such testsreinforce and legitimize every inequality that now exists. As James Loewen, asociologist from Catholic University's Center for National Policy Review,concluded, "standardized tests are the greatest single barrier to equalopportunity, at least in the sphere of education." Such accusations, and thevitriol behind them, have wounded and mystified those at the ETS. Several daysafter the Washington conference, the ETS observers were back at headquarters,stunned by what they had seen. They knew that such critics existed, but to heartheir work described that way was a heavy blow. In a side office, a secretarytranscribed a tape of Ralph Nader's latest blast at the ETS. From the tone ofhis criticism, there could be no doubt: this was war.To the ETS, it is a particularly galling war, because it is an attack on thecompany's point of greatest pride. Those who have shaped the testing industrydo not think of themselves as guardians of privilege; quite the reverse. Theyare proud of the role they have played in opening up opportunities; secure inthat pride, they can understand the current criticism only as one moremanifestation of the mindless leveling impulses of the day. What else couldexplain such hostility to the tool that seeks out talent wherever it may befound? Why else does everyone hate us so?From the other side, Alan Nairn defines the challenge differently. For the lastfive years, Nairn has been working for Ralph Nader on a study of the ETS due tobe published this spring; at twenty-four, he resembles in appearance andintensity the Nader of twenty years past. "Social class is viewed as a sad factof life, but not an issue," he says. "The controversy over testing makes classan issue."ON a 400-acre tract of pleasant wooded land in Lawrenceville Township, a fewmiles from Princeton, the professionals of the ETS develop the tests, studytheir effects, and brood about challenges to their authority.The people at the ETS are not naive about testing, nor blind to the dark spotsin its past. Their profession is still a young one; the first IQ test wasdeveloped less than a century ago. Modern "psychometricians" (which is whatthose who test mental ability call themselves) know that, like otherstill-emerging sciences, theirs had rough moments in its clumsy, infant days.They know perfectly well that the first crude IQ tests were used mainly forracial and ethnic exclusion. In 1912, on the basis of tests run at EllisIsland, Henry Goddard scientifically proved that 83 percent of Jews were"feebleminded," along with 90 percent of Hungarians, 79 percent of Italians,and 81 percent of Russians (most of them Russian Jews). Modern ETS researchersrecall with sad smiles the miraculous finding, some years after the EllisIsland tests, that Jews and Italians improved dramatically in intelligenceafter they had lived in this country for a while, and that their children,raised as English-speakers, seemed somehow to have been spared their parents'feebleminded genes. At the ETS, they have seen drawings such as Figure I, whichis used in the current version of the Stanford Binet Intelligence Test forelementary school children and which has now become practically a logo forProject DE-TEST. They have heard the sample question from the Lorge-ThorndikeIQ test that John Weiss of Project DE-TEST uses to illustrate racialstereotyping and bias in standardized tests: When a dove begins to associate with crows, its feathers remain ______ butits heart grows black. A) Black B) White C) Dirty D) Spread E) GoodFor each outsider's story of a defective test, the insiders have a story oftheir own. Winton Manning, a bearded, reflective man in his late forties who isan ETS vice president and philosopher on matters of social equity, frequentlytells journalists the tale of the breech block test. During World War II, theNavy evaluated breech block assemblers not by how quickly they could put abreech block together but by their score on a written test about how it shouldbe done. We have seen it all, is the unspoken message of stories like this; wehave worked hard on our tests.They take most pride in the oldest and most famous test, the SAT, firstadministered in 1926. Before that time, all College Board exams were essays;the Board was established in 1900 to standardize the entrance exams thatHarvard, Princeton, Yale, and a few other elite universities had beenadministering separately. The essay test and the multiple-choice SAT coexisteduntil World War II, when the essay was suspended for the duration and neverresumed (except as a component in certain English proficiency exams). HunterBreland, an ETS research psychologist, explained that to get statisticallyreliable results from an essay exam, students had to write five separateessays, with five readers each. "We found we could do as well with fiftymultiple choice questions in a thirty-minute test," Breland said. "We got thesame people in the same order." When those fifty questions are expanded to afull three-hour SAT, he said, you end up with a far better system than anyfeasible essay test. This principle has been extended even to demonstrations ofwriting skill. The old way of testing writing was to have the student write.For the last five years, as part of the SAT, students have spent thirty minuteson the Test of Standard Written English. It consists of multiple-choicequestions like the following, in which the student is asked to choose the bestphrasing for the underlined portion: UNLESS THEY TAKE ACTION SOON, the world may face a power struggle for theresources of the sea.A) Unless they take actionB) Unless action is takenC) Unless action will be takenD) If it will not take actionE) If action will not be taken(The "right" choice is B, nicely illustrating the ETS concept of the "best"answer. None of the options is very good, and B is a lame passive; but brightstudents are expected to pick it out as the "best" of a bad lot.)Another breakthrough toward test "efficiency" -- finding the people you want withfewer questions -- came with the "quantitative comparison" section that hasreplaced much of the mathematics portions of the SAT, the law examination, theGRE, and other tests. Instead of being asked to compute answers, the studentsare told to compare two items and see which is the larger. An example: Number of minutes in 1 week. Number of seconds in 7 hoursThe student is asked to mark A on the answer sheet if the quantity on the leftis larger, B if the quantity on the right is larger, C if they are equal, and Dif there is not enough information to tell. (The complexity of the answersystem makes some at the ETS feel that this section is vulnerable to coaching;if you've seen this sort of question before, you are less likely to panic onthe test.)With "objective" tests seemingly necessary in practice and defensible intheory, the ETS develops new ones all the time. The first step is to establishthe instruct validity" of the test -- that is, whether what it measures has, atface value, some connection to the purposes of the test. If it's a screeningdevice for potential lawyers, there should be some test of logical reasoning;if it's screening future doctors, a test of the physical sciences. Unless it isan achievement test, the questions are not supposed to measure outside"knowledge," only how the students handle the information given them. The mostobvious exceptions to this are the word analogy and antonym sections of the SATand the GRE, which are essentially vocabulary tests. On the sample SAT passedout by the company, the verbal section requires knowing words such as"emanation," "aesthete," and "espouse"; the GRE includes "concatenation,""anodyne," and "preternatural."Questions are written by the ETS's own staff or by a sizable free-lance pool.To get a better bargain for its money, the ETS has begun holding "item-writingworkshops" to teach its contributors the rules of the game. In early November,I sat in on one of these workshops, conducted by Richard Evert, a young ETSstaffer with a professorial air. There were half-a-dozen free-lancers in theroom, all but one of them women. They were working on "Logical Reasoning"questions from the LSAT. "It is important to remember that those taking thetest are a very bright population," Evert said as the session began, "whoseablest members, though younger and less wise" (this with a smile) "are just asgood as any of us. It will tax your abilities to write difficult items -- partlybecause, as they work their way up through the review process, a lot of thedifficulty will be ground away."Among questions he discussed, some were thrown out for being too confusing intheir answers, others for lurking cultural or sexist bias ("Sir! You just spent$3000 on your daughter's wedding and you are still drinking an ordinary vodka?Now's the time for Tovarich"). The free-lancers were counseled not to be upsetif their questions were rejected, handed back to them with elaborate correctivesuggestions, or eviscerated of their trickiest parts as they made their waythrough the in-house review. They were told that a special "minority"representative would screen the questions for bias; they saw examples ofnit-picking comments from the endless reviewers and editors who look at eachquestion individually and then look once more at the overall mix of questionson each new test. By the time a question makes its way to the final test form,the ETS says, it will have been inspected nearly thirty times; the greatmajority of questions will perish along the way. This is why, they say, ittakes eighteen months to prepare a new test, and why a full GRE or LSAT maycost $100,000 or more to produce.For items that pass through this screening process, the most demanding standardis still to come: their statistical performance on the "pretest." This step isthe heart of the ETS's claim for consistency in its tests; it is also the rootof one of the most basic complaints.EVERY SAT is divided into six sections, five of which count for the student'sscore. The sixth -- which the student can't distinguish from the others -- is madeup of new questions that have passed the in-house reviews and are ready foruse. As soon as the score sheets get back to Princeton and are sent through thecomputerized scanners and scorers, statisticians work up response charts foreach of the new questions. These charts are long horizontal strips, dividedinto six sections for the six possible answers to the question (A through Eplus "omit"). The chart shows how many students chose each possible answer, andwhat the median score of those students was on the rest of the test. What theETS hopes to find, of course, is that those who chose the right answer (knownas the "key") had higher scores than those who did not; if so, the question"works." If the question does not work -- like the first one in Figure 2 -- it mustbe changed. Sometimes the whole item must be junked; sometimes it can be saved,by adjustments and rewording to make the key more obvious or the "distracters"less tempting. Then it must be pretested again, until the statistics come outright. The second question in Figure 2 shows an adjustment that made it"work."The importance of this step can hardly be overstated. From the ETS's point ofview, the statistical pretest is essential; how else can it guarantee, yearafter year, that the test will measure just the same things, in just the sameway? But this consistency also means that the ETS must be very sure that itstests are measuring the right qualities, because their focus will never change.If talents are diverse, if different groups display their abilities indifferent ways, this process will never reveal it, because the standard set inthe beginning is the standard it retains.Even then the ETS's work is not done. There is the awkward but necessarybusiness of "ethnic studies," which means a comparison of black and whitescores, and correlations with different social and economic groups. There arethe endless instructions to colleges and graduate schools about how the testsshould be used. "A GRE test should be used only if its limitations are known,"says the GRE handbook, in a typical warning. "A GRE test score should be usedas only one of several criteria, and should not be given undue weight solelybecause it is convenient."By and large, these warnings are more necessary for graduate schools than forcolleges. One of the minor ironies of the testing controversy is that it occursduring a relative lull in pressure for college admissions. Because there arefewer college-age students than there were ten or fifteen years ago, andbecause a smaller percentage of them apply to college, today's students have aneasier time of it. Lois D. Rice, a vice president of The College Board, is fondof pointing out that 90 percent of all college-bound students are accepted bythe college of their choice; the argument about the SAT, she says, is really anargument about who gets into a few such as Stanford and Yale. This glosses overthe fact that admission to the Stanfords and Yales still makes a difference;more generally, her figures may be only a sign that the tracking system hasgrown more refined.These days, the SAT's significance for college admissions is mainly to confirmthe judgments made by IQ and placement tests over the previous dozen years.There is a close connection between scores on elementary school IQ tests and onthe SAT; indeed, recent research shows that tests in the fourth grade indicatewhich students will go to college almost as reliably as the SAT does. By theage of nine or ten, students are getting the picture about which of them willbe the lawyers and which the plumbers.GRADUATE schools still feel the pressure of numbers, especially those thatcontrol entry to lucrative professions such as law and medicine. Althoughapplications have swelled in the last decade, admissions staffs remain small.As a result, many professional schools do end up using cutoff scores, and theETS experts consider this an abuse of their work. Winton Manning has put out apolicy paper explaining the difference between using test scores to determineadmissibility, which is good, and using them for selection, which is bad. YaleLaw School, for example, might decide that no one who scored below 500 on theLSAT could handle the work; that, in Manning's view, would be fine. But manybig law and medical schools have more applicants than they can handle who scoreover 700, and they end up choosing among students on the basis of scores alone,accepting one at 740 and rejecting another at 710, even though the standardmeasurement error of the test means that the difference between those scoresmight be due entirely to chance.Manning and the others know these things happen; but that does not finallyundermine their faith in what they do, for their ideology and their lifeexperience combine to convince them of the value of their work. One critic ofthe tests said he was first drawn to the subject because his grandparents,Eastern European immigrants, arrived at Ellis Island just when many of theircountrymen were being pronounced feebleminded. William Angoff, an ETS vicepresident who is the spiritual father of the SAT, has taken the opposite lessonfrom life.Angoff is an urbane, silver-haired figure; when we met, he was wearing a blueblazer, gray slacks, and a striped silk tie. He grew up in the tenements ofBoston, with his whole family chipping in to pay his way through school. Hewent to Boston Latin, from there to Harvard, and on to eminence in the world ofpsychometrics. " I consider that the tests have been a friend of Americansociety at all levels," Angoff says. "It certainly has been a boon to peoplelike me. It picks out people because of their individual likelihood to succeed.The person who is going to do well on that test at Boston Latin is going to dowell on it anywhere else."With this assertion, the question is joined: Are the tests fair? William Angoffis serene in his faith: "I believe that standardized tests have benefitedAmerican education, and have benefited all the classes, as a very importantpart of the American meritocratic philosophy." Among his colleagues, thatconfidence is shared; people speak without irony of "our mission," and have nodoubt that the mission is the smoother, more just working of the"meritocracy."Yet there are reasons to think their faith misplaced, their efforts, howeversincere, far more destructive than they imagine. The reasons begin with thepremises of the "meritocracy" whose cause they uphold.It is interesting to remember that when Michael Young, the British sociologist,invented the term "meritocracy" twenty years ago, he did so with satiricalintent. His point was that a system of rewards based on "ability" and "merit"would not necessarily be any fairer or more pleasant than other systems ofstratification the world has known. It would, he said, be a dull and dangeroussociety, run by single-minded technicians. So deep has been the American hungerfor a "fair" system of classification, one based on ability rather thanaccident of birth, that Young's term has been appropriated without its irony.The unspoken premises of our meritocracy are these: -- That there is such a thing as "intelligence" or "ability," and that it can bemeasured. -- That intelligence matters. It counts for more than most human qualities andprovides the fairest proxy for "merit" in discriminating among people. It ishateful to judge people by charm, lineage, beauty, wealth. There may be otherimportant qualities -- honor, imagination -- but they are "soft"; intelligence isquantifiable, "hard." Not incidentally, intelligence lends itself to objectivedistinctions among large numbers of people. -- That education is the engine of social progress, and intelligence is itsfuel. A fair -- and self-interested -- society will give the best opportunities tothe most deserving candidates. The best students will go to the best schools,where they will be trained for the highest responsibilities (for which theywill, in large measure, receive the highest rewards). By the same logic, itmakes sense to exclude, as early as possible, those who are not up to theseresponsibilities, through predictive testing and academic "tracks."Whether or not they admit it, those who defend the tests are defending thesepremises. Whether or not they know it, those who complain about the tests arechallenging the premises, beginning with the concept of measurableintelligence.Like most modern psychometrician ETS officials are careful to say, whenspeaking for the record, that native intelligence is not what their tests aredesigned to detect. Indeed, the main SAT fact sheet says that it is a "test ofdeveloped ability, not of innate intelligence; a test of abilities that aredeveloped slowly over time both through in-school and out-of-schoolexperience." They say it with feeling, and on one level I am sure they believeit. But they clearly believe something contrary, too, if they are sincere -- andthey are -- in their claims that the tests serve as agents of the meritocracy.They believe that the tests measure something fundamental. One does not have tocall it intelligence. Call it smarts, or the right stuff. By whatever name, itis a notion of intrinsic worth. How else could the tests be presumed toidentify the promising lad from the Boston slums and propel him upward throughBoston Latin and Harvard? How else could they provide a means of comparing asixth-former at St. Paul's with a senior at Muncie High? How else could they begood for "all the classes" in America? The test-makers seemingly want it bothways: they want to speak with scientific decorum about the limits of theirwork, and they want to say, We have the best tool for judging peopleobjectively. Alan Nairn puts the paradox this way: "With all the disclaimers,they are in effect saying that this is so important a piece of information, soterribly revealing about the student, that it must be handled with greatcare."I have yet to meet a high school student who did not take the tests as ameasure of how "smart" he was. Students were not allowed to see their own SATscores until 1958. Frank Bowles, president of The College Board when thatdecision was made, was prescient about its effects. He said in 1960: "There wasgreat fear that students would have their values warped by learning their ownscores, but I have learned from hearing my own children's conversations thatSAT scores have now become one of the peer group measuring devices. Oneunfortunate may be dismissed with the phrase, 'That jerk -- he only made 420!'The bright, steady student is appreciated with his high 600, and theunsuspected genius with his 700 is held in awe.""I have spent time among people at the pinnacle of the meritocracy," says onetest expert, "with people who can dissect very rationally all the shortcomingsof the tests. But they'll say, 'That guy's really smart -- he got 800s on hisSAT's.'" Of the sixty-odd people I spoke with while preparing this article,exactly one volunteered his SAT score -- a friend who has made a reputation as ananalyst and writer. He scored in the mid-300s, felt crushed by the experience,and finally found his way into college on his swimming skills.Nearly everyone else said something like, "Of course, I did very well on thetests myself," in the same tone a millionaire might use in saying, "Of course,my family had some money." To say just how much would be bragging; but to leaveany doubt about the general picture would be worse.And this is only the effect on those who succeed: the real damage is to thosewho are taught to expect to fail. Record amounts of mail poured in to theNational Education Association, the teachers' organization, after two articlesby Arlene Silberman appeared, in Reader's Digest and McCall's, about the waychildren were taught to think of themselves as "ordinary" or "slow average"because of elementary school IQ tests. This wounding effect is compounded by astatistical quirk of the test. Common sense would suggest that, if a scale runsfrom 200 to 800, 500 would be the "average" score. Indeed it was, forty yearsago, when the statistical norms for the SAT were established and 500 was set asthe median score. But as times have changed and the sample of students takingthe test has grown less select, the median has dropped nearly 100 points; thesedays, only about one quarter of all students score above 500 on the test. Theother three quarters think they are "below average." At any one time, 75percent of all teenagers probably also think that they are below average inpopularity, resistance to acne, or development of secondary sex traits. Thesethings pass, or most of them do; the verdict on intelligence remains.THE ETS's long denial that coaching could affect scores illustrates itsconviction that the tests plumb innate capabilities. The test-coaching disputehas been the subject of numerous articles (the best of them, although hotlycontested by the ETS, is Stephen Levy's "ETS and the Coaching Cover-up," in theMarch 1979 issue of New Jersey Monthly) and promises to occupy lawyers andresearchers for years, but its central point is simple. Any fool can look atone of the tests and see that preparation has to make a difference. The verbalsections of the tests are not only loaded with vocabulary but also containobscure kinds of items likely to confuse or panic those who have not seen thembefore. One example is the "data evaluation" section of the LSAT, whichconsists of a long passage describing a business decision followed by up totwenty separate elements, each of which the student must identify as "MajorObjective," "Major Factor," "Minor Factor," "Major Assumption," or "UnimportantIssue." The mathematical portions penalize those who have not brushed up onhigh school algebra and geometry. During the early 1970s, several versions ofthe LSAT included questions based on a "triangle chart" like the ones in Figure3. From the information given, the student would be asked questions such as,"What was the average number of men per city who preferred home improvements?"Once the chart has been explained, it is a snap. Without that explanation, mostpeople are left to guesswork and blind luck.The ETS more or less admits all this now. Its newly released handbook for thelaw school exam says that "vocabulary cards and exercises, in addition toextensive reading with the help of a dictionary, will be useful" for certainparts of the test. Its publications now instruct students in test-takingstrategy -- for example, that it always makes sense to guess if you can narrowthe answer down to two or three possibilities. But until very recently, the ETScould not bring itself to admit such a thing. It defined coaching aslast-minute cramming, said such hasty efforts would do no good, and extendedthe judgment to a blanket assertion that special preparation for the tests "islikely to yield insignificant increases in scores."Why such resistance to such a self-evident truth? Most likely it is becausecoaching, if effective, threatens to upset the whole applecart, by suggestingthat what the tests measure can be fairly quickly changed. If the whole subjectwere less heated, this finding would not be surprising. One year of highschool, with its courses in algebra and English, is expected to increase testscores, so why shouldn't a six-month coaching course have a similar effect?Stanley Kaplan, founder of one of the most popular chains of coaching schools,often says that parents pay him several hundred dollars to do what the publicschools should be doing -- and that schools such as Exeter and Andover are justgrander versions of the same idea.Coaching also undermines the foundations of the tests from another direction.If courses can be designed for the specific purpose of increasing scores on thetests, does that not suggest that the tests reveal, rather than "aptitude" or"achievement," only mastery of an unusual and specialized system of thought?Since long before the current testing controversy, the ETS and its allies havefought a running battle against those who claim test scores are poor indicatorsof "merit" because of the limits inherent in both the format and the content ofthe tests.Nearly twenty years ago, a mathematics professor at Queen's College namedBanesh Hoffmann published a book entitled The Tyranny of Testing; his argumentwas that multiple-choice tests reveal nothing about the student's reasoning andpenalize those with complex or creative styles of thought. One example, takennot from Hoffman but from an official testing manual, is shown in Figure 4. Themanual explains why D is the answer to the first question and H is the answerto the second. But what about the student who sees, in the first question, thatthe first three figures are alike in being four-sided figures with no rightangles, and chooses E as the only appropriate answer? Or who sees, in thesecond question, that the first three figures are all isosceles triangles, andthat F is the only isosceles triangle among the answers? That student will getthe question wrong, and have a lower IQ, unless he is enough of an old hand atmultiple-choice tests to know the kind of obvious thinking the test-writers areusually looking for. "Sometimes you hear of very bright students who do poorlyon these tests," says David Riesman of Harvard. "They don't exactly fail them,but their scores are not as spectacular as they should be. All you have to tellthem is that the questions are designed for l'homme moyen sensuel, that theyshould take it at face value. Then they do fine."While visiting the ETS, I spent evenings in my hotel room with a stopwatch onthe table and a No. 2 pencil in hand, taking sample versions of the SAT, LSAT,and GRE. Six or eight times in the verbal portion of each test, I foundquestions with the same problem as the one in Figure 4: there were two or threeplausible answers, depending on the logical course one chose. In most cases, itwas easy enough to guess the "right" answer -- not by means of superior logic,but by knowing the way the ETS thinks. Of all the forms of test bias, this maybe the most insidious and deepest rooted: the shared assumptions about whichlogic is "compelling" and which merely "superficial," assumptions that derivefrom the social experiences that shape judgment and taste. ETS officials wincedat the suggestion that there was a system to their thought, but any veterantest-taker recognizes it. Recently a young law school graduate, his life storyan unbroken series of successes on ETS exams, began boning up for the bar examwhile finishing his clerkship at the Supreme Court. "My self-confidence went up50 percent," he said, "when I discovered that the Multi-State Bar is an ETSexam."The complaint against multiple-choice tests is based less on their neglect ofindividual genius than on their bias against larger groups. This is the heartof the case against testing. For all its other claims, the testing industryfinally rests its defense on "equity." The tests were created to broaden thepool of talent open to the colleges, and that is what the testers say theystill do. When they've finished with the disclaimers, the stipulations about"developed ability" and "limited predictive validity," when they lean back andtalk about the meaning of it all, most of them speak as William Angoff does.The tests have been a friend to all classes. The system works. If their casewere true, it would excuse most other defects in the test. But it is impossibleto ignore the evidence that, in most instances, the tests simply ratify earlieradvantage -- that, as engines of mobility, they have sputtered and died.From the archives: Academic Ignorance and Black Intelligence by William Labov (June, 1972)"Inner-city children do not necessarily have inferior mothers, language, or experience, but the language, family style, and ways of living of inner-city children are significantly different from the standard culture of the classroom, and this difference is not always properly understood by teachers and psychologists."THROUGH all the controversies of the last decade about inherent racialdifferences in intelligence, many black groups have concluded that standardizedtests are the latest attempt to deny them what they deserve. Blackpsychologists' associations have called for a flat moratorium on standardizedtests; a journal called Measuring Cup, published in Savannah, reports on testsfrom the black point of view. James Loewen of Catholic University, who iswhite, has devised questions like the following (modified from RobertWilliams's Black Intelligence Test of Cultural Homogeneity, or BITCH), whichtest "verbal aptitude and reasoning," but are based on black vocabulary.Saturday Ajax got an LD:A) He had smoked too much grassB) He tripped out on drugsC) He brought her to his apartmentD) He showed it off to his foxE) He became "wised up" (less dense) Once you know that an LD is a Cadillac Eldorado, the reasoning can begin;without that, it's hard to make a start. Loewen spent several years in the1970s teaching at Tougaloo, a primarily black college in Mississippi, andconcluded that the SAT and the Graduate Record Exam were major practical andpsychological barriers to his students. Students who performed well in hisclasses scored in the mid-400s on the GRE; his very top student got a 565. "Welearned to help some students develop reasons why they had not taken the GRE,"Loewen told a congressional committee considering the "truth in testing" billlast fall, because "some schools found it possible to waive the GRE requirementfor a believable excuse, while they would not have overlooked a 400 score."Loewen went on to make a fundamental point: that the tests are not fairnational standards of comparison. Without one nationwide test, the testdesigners say, how can you compare the boy from Boston with the girl from GrandForks? "A GRE score of 500," the GRE booklet claims, "has the same meaningwhether earned by a student from a small private liberal arts college or astudent at a large public university." This is precisely what Loewen denies,calling it a "statement of arrogance." "If you have two kids who get 500," hesays, "one from Harvard and one from Tougaloo, you know that one of them ispretty dumb, and it's the one from Harvard."The bias is not racial so much as economic, and the overall point is so baldthat it can hardly be ignored. College Board data showed this relationshipbetween economic standing and test performance in 1974:Student's Score Student's Mean Family Income 750-800 $24,124 700-749 21,980 650-699 21,292 600-649 20,330 550-599 19,481 500-549 18,824 450-499 18,122 400-449 17,387 350-399 16,182 300-349 14,355 250-299 11,428 200-249 8,639Loewen has compared the median incomes of the fifty states and the District ofColumbia with the National Merit Scholarship "cutoff score" for each state.(Scholarships are awarded on the basis of scores on the Preliminary ScholasticAptitude Test, which is usually given in the junior year. Because NationalMerit does not want all its scholarships going to Connecticut and New Jersey,it sets a different cutoff score for each state.) "Connecticut has the highestcutoff, and Mississippi the lowest," Loewen says. "It is intriguing to notethat Connecticut is the richest state in the nation, and Mississippi thepoorest." All in all, he says, his list and The College Board's matchperfectly; there is a .83 correlation between them -- about as close arelationship as statisticians ever expect to find.These patterns strongly suggest that what the tests measure is exposure toupper middle class culture -- perhaps even the culture of the professional classof the east coast. (Is it entirely a coincidence, Loewen asks, that scores onthe American College Testing Exam, written in Iowa City, are consistentlybetter predictors of performance for students from all backgrounds than the ETStests written on the outskirts of Princeton?) Whatever the reason, the findingssimply blow apart the original precepts of the tests. One can of course arguethat intelligence is hereditary, as in part it is, and that intelligence earnsmoney, as to some extent it does. But these general tendencies do not explainthe lockstep correlation between parental income and student scores. Unless oneis willing to set aside the evidence of daily life and conclude that all smartpeople are rich, these results can mean only one thing: that standardizedtests, created to offset one kind of privilege, have merely enshrined adifferent kind. The tests measure something, probably something of value -- butwhatever it is, it's clearly a symptom of social advantage."Tests may measure aptitude of achievement within populations that sharebackgrounds," Loewen says, "but they do not measure accurately acrossbackgrounds." To illustrate, he prepared the "Loewen Low Aptitude Test," whichis "designed to show my urbane white students some of the forms of test biasand to give them the experience of 'flunking' an aptitude test." Here is one ofits questions:Spline is to mitre as _____is to ____. a) love . . . marriage b) straw . . . mud c) key . . . lock d) bond . . . bail e) bond . . . paperThe question is biased toward the working class, toward students whose"exposure" would have taught them that a spline is a small piece of woodinserted to keep a mortise joint tight. From there, reasoning takesover -- spline adds strength to a mitre as straw adds strength to mud. (If youchose "love ... marriage," you have not grasped the concept of the "best"answer. "Sometimes one answer will be 65 percent right, and the 'distracter'will be 60 percent right," one test designer explained. "The first one's the'best' answer -- but the distracter would be best if the other were removed."Here, any SAT veteran should recognize "straw ... mud" as the "best" because,like "spline ... mitre," it is tangible, not abstract.)THERE is an annoying trick element to questions like this, but the argumentbehind them is sound. If the meritocracy's aim is to reveal talent, shouldn'twe strive for ways to transcend boundaries of upbringing and taste? Is therenot some way to avoid falling into the mushiness of saying that Black Englishis "just as good" as standard English, or that "all forms of learning areequally valid," yet still discover able people who grew up knowing more aboutsplines than about concatenations? As a first step, shouldn't the ETS system ofselecting questions be turned on its head? Instead of choosing new questionsbecause they will yield exactly the same results as the old, the testers shoulddeliberately look for questions that break the pattern -- not because they wanteverybody to get the same score, but because people from different backgroundsshould have a chance to display their reasoning skills. It has been donebefore. The first widely used American version of the IQ test showed that, innative intelligence, women were significantly weaker than men. When the testwas restandardized in the 1930s, one of the specifications was that men andwomen should get equally high scores. This involved, among other things,removing sports-knowledge questions from the test, and inserting others onwhich women did particularly well. Is it so outrageous to contemplateadditional test items on which able poor or black children would excel?Many professors look with horror on such a dilution of the tests. It is thelevelers at work, they say: if these standards fall, nothing else will stand."If people are underprepared for tests, they will generally be underpreparedfor courses," says David Riesman; he makes the logical point that a universitycannot do its business without students qualified to meet its standards. "GoodGod," said one of my former professors when I mentioned the subject to him."Students have little enough asked of them as it is. Few enough real standardsto meet. It can only be worse if the everybody-is-equal view takes over."True, many in the anti-testing movement abhor standards of all kinds -- anydistinction, any measurement, any judgment that might make somebody feel bad.But one need not be among their number to feel uneasy about the tests and themeritocratic philosophy they represent. It is self-evident that not everyone issuited for freshman work at Harvard or at any other college. Not everyoneshould be a doctor. Some were born without the potential; others were illprepared along the way. The universities do have standards of scholarship toprotect. In a way, the universities have been unfairly trapped by a myth -- the myth ofthemselves as the forces for social change. The ultimate meritocraticjustification for selective education is that this privilege will go to themost deserving, who will be better trained for their future responsibilities,and who will provide surer, wiser leadership for all mankind. But mostsociological studies show that education does not work this way at all. Theamount of education you get makes a difference in success in later life, butperformance in school doesn't (except by entitling you to spend more years inschool). Instead of selecting and training leaders, education certifies them tohold positions of privilege. This confusion between the academic role ofeducation and its role as a granter of credentials may be the biggest threat toacademic standards of all. It leads to grade-grubbing, demands for "fairadmissions," and a view of liberal education as nothing snore than a ticket tobusiness school.The testers are aware of these complaints. But what, they say, do you expect usto do? "It would be tempting to say that the schools should be looking about tofind the people who will make the greatest contributions to society in theiroccupations and professions," says William Turnbull. But the schools have askedthe ETS to help with a more limited task, selecting "those who are ready forthe next step."The proper challenge to testing is to accept a higher modesty, one that beginswith the understanding that this "merit" system pretends to more fairness thanit delivers. All systems of selection are unfair; all are tempted to claim toomuch justice for their results. We recognize this about previous systems butare reluctant to face it in our own. Once that fact is recognized, it may bepossible to think again about the proper building blocks of ameritocracy -- measures that do not seal fate at an early age, that emphasizeperformance in specific areas, that expand the pool of talent in more than ahit-or-miss way, and whose limits are always visible to us, so that we are notagain deluded into thinking we have found a scientific basis for the order oflords, vassals, and serfs. Copyright © 1980 by James Fallows. All rightsreserved. The Atlantic Monthly; February, 1980; The Tests and the "Brightest"; Volume volume 245, No. 2; pages 37-48.         |
|
