Posted by: Patrick Allen Foster | January 15, 2014

Common Core Standards Attempt to Quantify Reading Complexity, with Results that are Simultaneously Hilarious and Tragic

Some months ago, Blaine Greteman, writing in the New Republic, had this story about an algorithm used to categorize and rate books for purposes of the Common Core (emphasis added):

Lexiles were developed in the 1980s by Malbert Smith and A. Jackson Stenner, the President and CEO of the MetaMetrics corporation, who decided that education, unlike science, lacked “what philosophers of science call unification of measurement,” and aimed to demonstrate that “common scales, like Fahrenheit and Celsius, could be built for reading.” Their final product is a proprietary algorithm that analyzes sentence length and vocabulary to assign a “Lexile” score from 0 to over 1,600 for the most complex texts. And now the new Common Core State Standards, the U.S. education initative that aims to standardize school curricula, have adopted Lexiles to determine what books are appropriate for students in each grade level. Publishers have also taken note: more than 200 now submit their books for measurement, and various apps and websites match students precisely to books on their personal Lexile level.

Greteman points to this piece in The Atlantic, which reports that a survey of 1,154 public school teachers showed a consistent “tendency to err on the side of lower-level books” — i.e., a solid majority of teachers regularly assigned books or texts to read that have Lexile scores lower than what the Common Core recommends for that grade level. The Atlantic piece in turn draws upon and points us to this report, “Common Core in the Schools: A First Look at Reading Assignments,” by Timothy Shanahan and Ann Duffett (pdf, 57 pages, is here). The report looks at whether English teachers are assigning texts with “language complexity appropriate to the grade level.” From the findings (p. 8):

The Common Core asks teachers to assign texts that provide language complexity appropriate to the grade level, but significant proportions of teachers—particularly in the elementary grades—are still assigning texts based on students’ present reading prowess. Specifically, the majority of elementary teachers (64 percent) choose to match students with books presumed to align with their instructional reading levels. This happens less often in middle and high school, with approximately two in five middle school teachers selecting texts this way. This means that many youngsters are not yet working with appropriately complex language in their schoolbooks.

I’m sure that there’s much that can be said about these findings, but that’s not why I am writing this post. Rather, the Fordham Institute report includes lists with the recommended texts for each grade level, along with the Lexile scores for each text. The lists and the scores make for excellent nit-picking, punditry, and general grousing.

According to the standards, middle-schoolers (grades 6-8) should be reading texts in the 950 to 1185 range. Here are some representative Common Core exemplar books, with their Lexile scores, for grades 6-8 (from Table 6, p. 30 of the report):

Anne Frank: The Diary of a Young Girl (Anne Frank) 1080
Little Women (Louisa May Alcott) 1230
Roll of Thunder, Hear My Cry (Mildred D. Taylor) 920
The Adventures of Tom Sawyer (Mark Twain) 970

~~~

Here are some books that did not make the Common Core exemplar list for grades 6-8:

The Pearl (John Steinbeck) 610
Johnny Tremain (Esther Forbes) 840
Let the Circle Be Unbroken (Mildred Taylor) 850
The Golden Compass (Philip Pullman) 930
A Separate Peace (John Knowles) 1110
Summer of My German Soldier (Bette Greene) 800

~~~

It is a mystery to me why Mildred Taylor’s Roll of Thunder, Hear My Cry (the first book in a trilogy) makes the list, but Let the Circle Be Unbroken (the second book in the trilogy) does not make the list.

Anne Frank’s Diary, of course, was originally written in Dutch. Wikipedia tells me that the first English translation of the Diary, by Barbara Mooyaart-Doubleday, was published in 1952; a second translation, by Susan Massotty, appeared in 1995. The report does not indicate which translation was used for the Lexile score, although if I had to guess, it was probably the 1952 version, if only because it’s been around longer and for that reason likely has the benefits of the first mover effect and greater market penetration. (Still, it might be nice if the report specified which translation is used to compute the score.)

The Common Core standards recommend that ninth-graders and tenth-graders should be reading texts with a Lexile range of 1050 to 1335. That said, here are some of the Common Core exemplars listed for grades 9-10 (from Table 7, p. 35 of the report):

The Metamorphosis (Franz Kafka) 1340
The Odyssey (Homer) 1130
Candide, or the Optimist (Voltaire) 1110
Pride and Prejudice (Jane Austen) 1100
Fathers and Sons (Ivan Turgenev) 980
In the Time of the Butterflies (Julia Alvarez) 910
Jane Eyre (Charlotte Bronte) 890
Fahrenheit 451 (Ray Bradbury) 890
Things Fall Apart (Chinua Achebe) 890
To Kill a Mockingbird (Harper Lee) 870
The Book Thief (Markus Zusak) 730
The Killer Angels (Michael Shaara) 610

~~~

Once again, for the non-English works, we have no indication of the translation in play. This is especially egregious in the case of the Odyssey, for which there are at least a dozen English translations in wide circulation, which vary greatly in their style and complexity.

And here are some texts that did not make the exemplar list for grades 9-10:

Brave New World (Aldous Huxley) 1360
Robinson Crusoe (Daniel Defoe) 1360
Gulliver’s Travels (Jonathan Swift) 1330
Silas Marner (George Eliot) 1330
The Crucible (Arthur Miller) 1320
1984 (George Orwell) 1090
A Tale of Two Cities (Charles Dickens) 990
Adventures of Huckleberry Finn (Mark Twain) 990
The Princess Bride (William Goldman) 870
The Fellowship of the Ring (J. R. R. Tolkien) 860

~~~

I am sympathetic to the desire for quantitative scales and measures for comparing complex items (like books) and for formulating more uniform curricula. And, to be fair, as Greteman notes, “both the creators of the Common Core and MetaMetrix admit these standards can’t stand as the final measure of complexity. As the Common Core Standards Initiative officially puts it, ‘until widely available quantitative tools can better account for factors recognized as making such texts challenging, including multiple levels of meaning and mature themes, preference should likely be given to qualitative measures of text complexity when evaluating narrative fiction intended for students in grade 6 and above.’ But even here, the final goal is a more complex algorithm; qualitative measurement fills in as a flawed stopgap.”

All that said, the list and some of the scores have me scratching my head.

The selection or inclusion of specific texts as exemplars is mysterious. How is that Oedipus Rex made it onto the list of Common Core exemplars (p. 36), but Antigone did not? Both are plays by Sophocles, but if anything I think that Antigone is a juicier text to dive into and one in which high school students might find more points of connection. Antigone deals with civil disobedience, sacrilege, the abuse of power, a person’s duties to the state and the law in conflict with duties to family, a young woman’s courage to obey the demands of conscience in the face of opposition from the powerful and abandonment by her allies — what exactly does Oedipus Rex have going for it, besides the protracted (and effective) use of dramatic irony?

Moreover, the opacity of the process is so very frustrating. I understand the concept of a proprietary formula, but when we’re using these scores to help craft public policy, I really have to wonder whether relying on such a secretive process is good practice. Especially when this formula apparently produces results such as these. (Is The Killer Angels really so much simpler as a text than To Kill a Mockingbird as their relative scores indicate?) A reality check would seem to be in order.

Once again, Greteman:

When Huckleberry Finn isn’t complex enough for our high-school students, I can’t help wondering if we need to change the way we conceptualize literary complexity. I’m an English professor, and I live in Iowa City, a UNESCO World City of Literature, but according to MetaMetrix my bookish hometown might as well go play patty cake. On my way to work I pass the House on Van Buren Street where Kurt Vonnegut began Slaughterhouse Five — but with a score of only 870, this book is only a fourth-grade read. By these standards Mr. Popper’s Penguins (weighing in at a respectable 910) is deemed more complex.

Indeed.

Meanwhile, Natasha Vargas-Cooper wonders if we should stop teaching novels to high school students altogether. (H/T: 3 Quarks Daily.) An excerpt:

Just maybe the novel is not the best device for transmitting ideas, grand themes, to hormonal, boisterous, easily distracted, immature teenagers. Maybe there is a better format and genre to spark a love of reading, engage a young mind, and maybe even teach them how to write in a coherent manner. Thankfully this genre exists: It’s called non-fiction.

Journalism, essay, memoir, creative nonfiction: These are only things I started reading as an adult because of how little I enjoyed reading novels in high school. Surely, the un-made-up stuff would be more of a bore, I thought. Yet when I finally read In Cold Blood, Into Thin Air, the works of Hunter S. Thompson and Joan Didion, I continually pleaded aloud to my friends in their twenties, “Why didn’t anyone make me read this in high school?!”

Vargas-Cooper’s basic argument is that teenagers necessarily lack the life experience to connect with and digest the novels of Hemingway and Steinbeck. Count me as unconvinced. As one of the commenters at 3 Quarks Daily says, “Here’s a radical thought: maybe great literature can have multiple themes that resonate on different levels in different periods of life.”

That said, some novels much more than others definitely lend themselves to pedagogical purposes. For more on this topic, see this post at Thus Blogged Anderson, along with the comments. I agree with Tom Freeland that The Sun Also Rises (an example used in Vargas-Cooper’s article) is a sub-optimal choice to assign to high school sophomores.

Advertisements

Responses

  1. Apparently this algorithm is supposed to be a substitute for people who have actually read the works in question. And, per the blog post of mine that you link to, note as well that Anne Frank’s diary is the only nonfiction work listed.

    • In fairness, if one digs deeps enough into the lists, one can find other non-fiction; on the Grades 6-8 list, for example, the Narrative of the Life of Frederick Douglass, an American Slave is one of the exemplar texts — filed under “Informational Texts” on p. 33. (Noted: only about 10% of middle school teachers assign that work. Maybe they’re saving it for high school.)

      That said, fictional works clearly outnumber non-fictional works on the list. No real surprise there; just a reflection of the large-picture status quo.

      As for the algorithm, I might not have quite so much of a problem with it if there were more transparency about the process, and maybe more than one measure. Heck, even the recently deceased BCS ranking system used an average of polls and several computer ranking systems — relying not on one single algorithm, but on the average of multiple algorithms.

      I understand why a company like US News wants to keep their college ranking algorithm secret and proprietary — the secret sauce is key to their business model. Same with Google’s search algorithm.

      But when governments are relying upon algorithms to select and assign texts for curricular purposes, I don’t think it’s out of line to want transparency. Had I the power, I might give NSF grants to several university teams to develop four or five competing algorithms — and make it a requirement of the grant that the resulting algorithms would be open source and transparent.

    • Agreed. The secrecy is silly.

  2. What is really tragic is that books like “Diary of a Wimpy Kid” has a score of 950 while Fellowship of the Ring scores 860.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: