Dana Goldstein

Realistic Expectations for New Teacher Evaluation Systems

I'm seeing some off-the-mark responses to the news--first reported by Education Week's Stephen Sawchuck, and then picked up by the New York Times--that many of the new, high-stakes teacher evaluation systems are rating only 2-6 percent of teachers ineffective. This is being greeted by some supporters of numbers-driven teacher reform as a disappointment, while skeptics, like American Federation of Teachers president Randi Weingarten, are suggesting this proves the vast majority of teachers are great performers, after all. 

I don't think we can jump to either conclusion. First of all, the goal of these systems is not necessarily to fire large numbers of teachers, it is to help them improve their practice, since previously, most American educators received little constructive feedback on their work. Most new evaluation plans include more classroom observations, which means teachers are not just receiving number ratings, but actual notes and suggestions on their instruction. Of course, whether that feedback is helpful or useless depends entirely on the quality of the administrator.

Contra Nicholas Beaudrot, it's not true that education reformers have a "hazy" idea of how many bad teachers they'd like to see lose their jobs after this overhaul. I've asked a number of prominent accountability hawks that question over the past six years and the answer I've heard most frequently is "5 to 10 percent." As Matthew DiCarlo explains, that estimate is culled from the research of the ubiquitous Stanford economist Eric Hanushek, and by that standard, these evaluation systems are already half way to where they are intended to be, a reasonable outcome for something so new.

That said, I'm still quite skeptical that the new evaluation plans will transfrom the teaching profession, in part because of the lessons from history I'm learning as I research my book. For over a century, school reformers have been dissatisfied with how teachers are evaluated, yet overhauling rating systems has not, historically, been an effective way to improve educational outcomes for kids. This is like hoping to lose weight by buying a new, high-tech scale, without changing your diet or exercise routines.

During the late nineteenth century, the New York City schools used an "excellent-good-fair-bad" rating system for teachers. When reformer William Maxwell became superintendent in 1898, he complained that 99.5 percent of teachers were rated "good" and instituted a plan to grade teachers on an A-D scale instead. The city distributed intricate tables for judging teachers’ output. First, teachers would be measured by evidence of their students’ learning, which could be demonstrated through test scores or examples of children’s essays, penmanship, and drawings. Teachers would also be judged on their personal characteristics and given numeric ratings in largely subjective categories, such as “obedience,” “honesty of work,” “dress,” “voice,” and “force of character.” A teacher’s command of classroom discipline could be assessed by counting the number of students who were late or unruly, and even by timing the number of seconds and minutes it took for a teacher to distribute or collect worksheets. 

By the late teens, the vast majorty of teachers were earning perennial ratings of B+, the exact sort of slightly-better-than-average rating that had predominated under the previous plan. In prominent education journals, dissident principals like Alexander Fichlander, a Brooklyn leftist, explained that the paperwork involved with implementing the system was so burdensome that administrators rushed through it; what's more, there was little incentive to spend a lot of time rating teachers if the district provided no extra funding or training to those who needed to improve. Additionally, when managers find it is difficult to replace low-performing teachers with workers who are more effective--another likely outcome--they may decide evaluation systems are not worth their time.

Because of these problems, by mid-century, detailed evaluation systems were being replaced by simpler "satisfactory-unsatisfactory" plans, which today are being replaced by value-added measurement and frequent observation notes. But if the new evaluation systems end up being more about paperwork than about improving practice, then they, too, will fail to improve instruction and will lose their political palatibility.

On "Innovation" in Social Policy, and "Solutionism" in Education

The following was written by the Princeton sociologist Melvin Tumin, in 1973. Tumin was thinking about the Teacher Corps, a Great Society program that was a sort of lefty precursor to Teach for America, in which intern teachers arrived in disadvantaged schools eager to close achievement gaps and -- because it was the sixties -- also build racial pride, improve student self-esteem, and overhaul the curriculum to better reflect African American history and culture. Most Teacher Corps interns grew frustrated by the slow pace of change within schools; most veteran educators were suspcious of the interns' intentions and ideology. As Tumin points out, this inevitable culture clash is a key finding of research in organizational theory, and helps explain why it is so difficult for small social programs to impact bureaucracies:

...one of the most important factors that made the Program difficult to implement was that it could not promise the members of agencies and institutions whose cooperation was needed that it would be worth their while. Innovation is a charming word, beguiling and rousing. But it is like other terms such as relevance, concern, sensitivity. One cannot be against these on principle. But they are almost always privately read as warning signs that there is trouble ahead for those who are fulfilling their accustomed routines. Moreover, since most innovation efforts fail sooner or later, wise masters of ongoing enterprises have learned to live and wait until such innovations speed themselves to their ultimate demise. While not many other earthly travails can be safely waited out, with any hope of relief, innovative programs do have that special quality of a high probability of failure, so that “this too shall pass” is a reasonably sound prediction about most of them. 

I read this just after I returned from SXSWedu, and it seemed particularly relevant to the (almost total lack of) dialogue between social entrepreneurs who want to use software to disrupt traditional classroom practices in ways that are supposed to benefit poor children, and the majority of teachers and administrators who might be enthusiastic about specific technologies, but who simply do not see the lack of technology as the key barrier preventing schools from better serving all students. Educators are more likely to point to teacher quality, or to the content of the curriculum, or to factors stemming from the home and family. And sure, technology may be able to address many of these challenges. Yet tech-hypers lose credibility when they ignore the fact that inequalities outside the realm of access to technology remain the primary drivers of disparate educational outcomes.

Evgeny Morozov's recent essay on technological "solutionism" is certainly applicable to the tendency to mistake the technocratic measurement of educational problems, using software, as the very same thing as coming up with solutions to those problems. It should be obvious--yet at SWSWedu, it rarely was--that the quantification of inequality, whether in student achievement scores or ratings of teacher effectiveness, is only a jumping off point for complex political, policy, and social debates over how to use such data, and whether and how to close achievement gaps. The collection of "Big Data" does not, in and of itself, guarantee the formulation of effective solutions to problems.

Radio Days

In a Nation podcast, Bryce Covert interviewed me about Obama's new early childhood proposals. Listen here.

And on Jane Williams' BloombergEDU show, I participated in a panel discussion of education writers. We each got to discuss two education issues to watch in 2013. I named preK (before Obama's annoucement!) and school diversity/integration. Listen here.

Big Data for Little Kids: "Assessment" in Obama's Pre-K Plan

Via Twitter, I'm seeing a lot of anxiety about the part of President Obama's new universal preschool proposal that calls for "comprehensive data and assessment systems" to track student progress and program quality. I know what you're thinking: Standardized tests for 4-year olds? And I agree, it is absurd to imagine toddlers filling in Scantron bubbles. But that isn't at all what the administration has ever meant when it talks about assessing pre-K quality. Instead, a big part of the Obama/Duncan vision is for statewide data systems that link students' early elementary school performance back to the preschool programs they attended, so those programs (not individual teachers or students) can be judged on whether they adequately prepare children for school.

As I reported for Slate in 2011, the potential best part of this Big Data for Little Kids push is that the accepted best practices for early childhood assessment include "testing" not only literacy and numeracy skills, but also the sorts of social, emotional, motor, and creativity skills that have gotten such short shrift since the onset of No Child Left Behind. For the youngest students, those skills include whether a child is confident asking questions, whether she can fasten her Velcro shoes and zip her jacket, whether she can hum a song and draw, and whether she plays cooperatively with peers. One widely-respected model for early-childhood assessment is the Work Sampling System, in which teachers observe students over the course of weeks or months, and then assign them a numeric rating in each of these categories. Again: Unlike assessment systems in use for older kids, these numbers are not meant to punish or reward individual teachers or students. Instead, the idea is to use the data to help teachers identify where students need to grow -- and to help schools, districts, and states figure out how to improve teacher training and program management at the system level.

Testing for older students, I think, has a lot to learn from the best practices in early-childhood assessment. One lesson is that we have to look at the whole child over time, not just at his reading and math scores during a specific hour. Another lesson is that, as psychometricians have been warning us, the most accurate use of a test is as a diagnostic tool for the person who is taking it, not as a human resources tool to punish or reward the test-taker's teacher or principal. 

Now. I don't want to be too optimistic, because early-childhood assessment can certainly get screwed up. In the Slate piece, I describe how Ohio managed to turn a good idea for longterm holistic assessment into a 10 to 15-minute, one-time, sit-down, scored literacy test for 5-year-olds. Most child development experts laugh at the idea that you can tell very much about any kindergarten student in a specific, short block of time like that, since 5-year olds don't typically have the self-control to put aside immediate concerns like hunger or boredom to focus on a Very Important Task. 

A colleague of mine is working on a large paper about early-childhood assessment, so there will be much more to report in the coming weeks and months. Stay tuned.

Behind Obama's SOTU Remarks on Vocational Education, Germany, and American High Schools

In his State of the Union address, President Obama promised to create a new federal funding stream to provide high school students with technical education, to help them prepare for the workforce. This is promising. As I've reported, during the first term, President Obama and Education Secretary Arne Duncan talked about vocational education almost exclusively in the context of post-secondary schools. That allowed the administration to dodge controversial, historically-freighted debates over K-12 "tracking," a practice that, throughout the twentieth century, pushed non-white students, girls, and the intergenerationally poor into lower-paid professions and away from college. But this political dodge also ignored some important research and innovation at the K-12 level.

For one thing, surveys of high school drop-outs show a leading cause of their leaving school is the belief that nothing they are learning in the classroom will help them get a job. What's more, a number of innovative new high schools across the country are pioneering a model for career and technical education that has little to do with the narrow vocational classes of yesteryear, in which kids were taught how to sew or operate a printing press -- skills that technological change can render less relevant over time. Instead, at the MET schools in Rhode Island, Linked Learning schools in California, and at Tech Valley High outside Albany, high school students complete externships in real workplaces, exploring fields as diverse as baking, engineering, biotechnology, and train conducting. Students have the opportunity to dip into more than one profession, because the idea is less to train for a specific job than to see how adults use their own educations in the workplace each day. That helps students stay motivated to earn a degree, and introduces them to the behaviors and practices specific to the working world.

A few notes of caution. First, the school Obama referenced in his address is P-Tech here in Brooklyn -- the Pathways in Technology Early College High School. Like a lot of celebrated educational arrangements in New York City (ex; the Harlem Children's Zone), P-Tech is special; it is a partnership between the New York City schools, IBM, and the City University system. To earn the combined high school diploma/associate's degree P-Tech offers, students must commit to a six-year course of study, which is time-consuming and expensive, requiring extra resources that most school districts simply don't have, especially absent the deep-pocketed corporate partners present in a city like New York. Obama's promise of more funding is desperately needed if we want to recreate schools like this one. But since the Republican Congress has demonstrated a specific interest in cutting job-training programs, I'd like to hear more about exactly where the president believes this new money will come from. (Same goes for his pipe dream of universal pre-K, by the way.)

There are certainly discomfiting curricular trade-offs at many vocationally-oriented schools, and we shouldn't overlook them. P-Tech doesn't offer foreign languages or Advanced Placement courses, for example. So I found it interesting that Obama praised the German secondary school system in his address, saying, "Right now, countries like Germany focus on graduating their high school students with the equivalent of a technical degree from one of our community colleges, so that they’re ready for a job." German "dual track" vocational education has been hotly debated among American school reformers for over a century. Why? Because this system is radically different from our own. It separates students after the 10th grade and places them on a vocational, academic, or mixed vocational/academic track. The result is that in Germany today, only about one-third of all upper secondary school students are enrolled in a course of study that leads directly to a liberal arts university. That reality deeply challenges American notions of "college readiness" for all, a major priority of the new Common Core -- shared national curriculum standards that the Obama administration also supports.

The truth, of course, is that despite our romanticization of college, only about one-third of Americans under 30 have earned a bachelor's degree. Where we actually do differ from Germany is what happens to the other two-thirds of young adults. In Germany, they typically hold an occupational certification by the age of 20, while in the United States, non-college grads are often left without marketable skills or qualifications.

More on vocational education in the US and Europe:

• The Future of Vocational Education
• The Apprentice
• Should All Kids Go to College?

"Unquestioning Loyalty"

This is the text of a loyalty oath New York City public school teachers were expected to take in 1917 and 1918. Over the next three decades, teachers across the country were periodically subjected to these sorts of jingoistic fevers. For Quakers and antiwar activists, a particular problem with the pledge was the vow of unquestioning obedience to American military policy, as well as the promise to inculcate such obedience in one's students:

"We, the teachers of the public schools of the City of New York, do solemnly pledge our unqualified loyalty to the President and Congress of the United States in this war with the imperial governments of Germany and Austria. 

 "We pledge ourselves actively to inculcate in our pupils by word and deed love of flag and unquestioning loyalty to the military policy of the government and to the measures and principles proclaimed by the President and Congress.

"We declare ourselves to be in sympathy with the purposes of the government and its efforts to make the world safe for democracy, and believe that our highest duty at this moment is to uphold the hands of the President and Congress in this crisis. 

"We believe that any teachers whose views prevent them from subscribing to such sentiments should not be permitted to teach the youth of our city.”

 Related: Texas state senators object to a curriculum that supposedly sympathizes with socialists, critiques founding fathers.

A Little Glimpse at My Book Research: Red Scare Attacks on Teachers

If there were endless hours in the day, I'd blog about the news much more regularly. But since my book project is pretty much all-consuming at this point, I'm not posting here as much as I'd like. I do want to let you know that over at my Tumblr, Popular History, I'm posting occasional tidbits I find as I research the political and social history of American public schools and teachers, including some fascinating newspaper and magazine reports, photos, and drawings.

Here's today's offering, which displays the paranoid rhetoric surrounding leftist educators during the WWI, WWII and Cold War Years. Though just a few hundred of New York City's 80,000 teachers at mid-century were social activists, those who lost their jobs during periodic red scare witch hunts were some of the district's most inspiring and dedicated teachers, including early advocates for school integration and a black history curriculum. Chapter five of the book will tell their stories. 

Michelle Rhee, Back in the News in a Big Way

I'm looking forward to this evening's episode of FRONTLINE, which will explore evidence of adult tampering with children's tests in certain Washington, DC public schools during the chancellorship of Michelle Rhee, who is perhaps the nation's most controversial school reformer. The allegations are not new. They were first revealed nearly two years ago by my colleague Greg Toppo and his crack reporting team at USA Today, which used a computer algorithim to comb through reams of data, and found a national pattern of manipulated standardized test scores in the wake of No Child Left Behind, which greatly increased test-score pressure on schools and districts. Tonight's documentary, however, reported by John Merrow, will delve more deeply into Rhee's efforts to evade a thorough investigation of statistically implausible test score gains. In a preview interview with the Education Writers' Association, Merrow judges Rhee harshly. "The record is pretty clear that D.C. schools are not better because she was there," he says. "They’re still at the bottom, with the lowest graduation rate in the country."

The FRONTLINE report is especially timely as it comes just a day after Rhee's national advocacy organization, Students First, released a "report card" grading states on how closely they align with Rhee's agenda of tying teacher evaluation and pay to student test scores; weakening teacher tenure; transitioning teachers from traditional pension to 401(k) plans; funding charter schools and private school vouchers; instituting mayoral and state control of schools; and expanding the charter school sector and holding it accountable for results. Doug Henwood notes that states with high grades on the report card, including Louisiana and Florida, have woefully poor student academic achievement, while Massachusetts, with the highest math and reading scores in the nation, earned a D+ from Students First. This is true, but as Matt Yglesias writes, most of Rhee's favored policies are too new to be conclusively judged; the proof will be in the pudding a decade from now, when we can track what effect, if any, school choice and test-score based accountability policies have had on gold-standard NAEP test scores over time. 

That said, Rhee's most influential effort, her push to tie teacher pay and job security to individual students' standardized test scores, is the one element of this agenda for which we already have some powerful, and disturbing evidence, thanks to the investigations of USA Today and FRONTLINE. As I've reported, psychometricians, the scientists who study testing, have been warning for decades that when policy-makers attach ever-higher stakes to tests--first accountability measures targeting schools and districts, and now job security threats for individual principals and teachers--the reliability of test scores is compromised, sometimes due to teaching-to-the-test, and sometimes due to outright cheating. This consistent finding has crucial implications for "value-added measurement," the method, promoted by economists, of using standaridzed test scores as the major proxy for teacher quality and student learning. If we are concerned, as Matt is, about so much economic modeling being backed by poor data, we must deal with the implications this has for education policy. In my 2012 year-in-review, I talk a bit more about what all this means for education research; I want to emphasize that raising concerns about value-added measurement does not mean one is reflexively "anti-testing," it simply means that one is questioning the wisdom of tying high stakes to tests.

Before I leave the subject of Michelle Rhee, it's worth noting that Students First has had a very difficult time coming up with a consistent position on the rights of teachers to collectively bargain, even just for pay.  (Most recently, it seems that they are against collective bargaining -- for everyone.) Now, as Joy Resmovits reports at the Huffington Post, many of the well-connected Democrats who worked for Rhee have left her team, and are being replaced by former staffers from Americans Elect, a hedge fund-backed effort to elect third-party independents. 

I'm not questioning Rhee's personal commitment to certain progressive aims. I was impressed, for example, with her genuine efforts to better integrate public schools in rapidly-gentrifying DC neighborhoods, by working hard to convince college-educated parents to enroll their kids in local schools. But as her career advances nationally, she is more and more allying with Republican, even Tea Party-type conservative governors and state legislators. I'm midway through writing a book on the history of American education, and from this vantage point, I'm very skeptical about relying upon an anti-government political movement--one almost totally indifferent to social and economic inequality--to invest in and improve the schools poor children attend.

2012: The Year in Review, Education and Beyond

Education story of the year: The Chicago teachers' strike. American teacher unionism was founded in Chicago in the late 1890s, as female, largely Catholic elementary school teachers resisted centralization policies--standardized testing, a uniform curriculum, numeric teacher evaluations--pursued by a male, Protestant bureaucracy. So it was fitting that the loudest cry of protest against contemporary standards-and-accountability school reform emerged in the Windy City this September, as teachers resisted professional evaluation tied to student test scores, closures of neighborhood schools, and the expansion of the charter school sector. You can read my history of Chicago teacher unionism here.

The strike has had a few interesting results. First, it raised the profile of Chicago Teachers' Union leader Karen Lewis, who is a less compromising and more leftist figure than Randi Weingarten, president of the national American Federation of Teachers. Second, it brought to the public's attention the tension bewteen increasing test-score pressure on teachers and schools while cutting budgetary support for art, music, counseling, school psychologists, and the many other crucial, yet more holistic services schools provide. Third, it resulted in a compromise contract with both progressive and regressive features. More funding for social support services, especially in high-poverty schools, is a good thing. Continuing to backload teacher salaries and bonsues, though, will not make the profession more appealing to ambitious young people or career-changers. Yet it is encouraging that CTU agreed, at least in theory, to professional evaluations that include evidence of student learning. Now the devil will be in working out the details, particularly on what role standardized test scores will play, and how to evaluate teachers of currently non-tested subjects and grades, like art, music, PE, and kindergarten.

 

Magazines of the Year: Tomorrow and Jacobin.
 

Education research-finding of the year: Teachers matter, and not just for academics. A study by economists Raj Chetty, Jonah Rockoff, and John Friedman found that teachers who consistently improve their students' standardized test scores also help children avoid teenage pregnancy, get to college, and earn higher incomes. But there is a crucial caveat to this finding: The study was conducted in a low-stakes setting, in other words, among teachers whose pay and evaluation were not tied to test scores. The researchers admitted in their paper that when tests do become high-stakes, there is an increased risk of score manipulation, which can occur either through teaching-to-the-test or outright cheating. In high-stakes settings, test scores become less reliable and, therefore, their link to better life outcomes for kids could be compromised.

 

Under-reported education trend of the year (and decade): Sociology used to be the academic discipline with the most influence over public education policy. Today that discipline is economics. On the upside, we now have more data about children's academic and life outcomes. On the downside, education policy-makers are paying less attention to aspects of children's lives that are more difficult or impossible to quantify, such as parental involvement and comfort with diversity.

 

New York restaurant of the year: Pok Pok. We only had the patience to wait in line for this place once, but it was so, so, so worth it. A close runner-up is Battersby, where I had two wonderful meals. Another year, another few reasons never to leave Brooklyn. 

 

Book of the year on How We Live Now. Twilight of the Elites, by Chris Hayes. This bracing book about "America after meritocracy" has implications for every area of policy-making, but is especially sharp at deconstructing myths we tell ourselves about education: that test scores measure aptitude and that elite schools serve the common good.

 

Education story to watch in 2013: The roll-out of the Common Core. Will the movement to implement shared national academic standards remain bipartisan, or will conservatives and Republicans increasingly turn against it? Will schools implement the Core faithfully, or will myths about the standards--like the false idea that they cut out fiction reading--persist?  

 

Albums of the year: "Break it Yourself," by Andrew Bird. "Sun," by Cat Power. "Channel Orange," by Frank Ocean. "The Idler Wheel..." by Fiona Apple. "Shields," by Grizzly Bear.

 

Education book of the year: My favorite was Saving the School, by Michael Brick.

 

A book to pounce on in 2013: The absolutely masterful Hope Against Hope, by Sarah Carr, the definitive account of education reform in post-Katrina New Orleans, told through the eyes of a student, a teacher, and a principal. A gripping narrative with deep historical and political ramifications. 

 

Cultural controversy of the year: The battle over the future of the New York Public Library's main branch, at 5th Avenue and 42nd Street. Should this world-class research institution ship several million books to New Jersey, and open space for a lending library? Should architect Norman Foster, known for his glass additions to historic buildings, be let loose on this Beaux Arts masterpiece? I work there almost every day, and I still can't decide how I feel about it. 

 

TV Show of the Year: "Girls." Feminists are funny. 

 

#longreads of the year: This past spring, the magazine that launched my career, The American Prospect, experienced a terrifying brush with death. I'm so glad donors and subscribers have helped The Prospect continue its work, because under editor Kit Rachlis, it has published some amazing writing. Monica Potts' "Pressing on the Upward Way" is a compassionate, beautifully-constructed portrait of rural poverty in Eastern Kentucky. Equally stirring was Gabriel Arana's "My So-Called Ex-Gay Life," which not only told Gabe's personal story of surviving "ex-gay therapy," but also broke news by revealing how the psychiatrist who pushed to define homosexuality as a mental illness, Robert Spitzer, has come to regret and retract his previous work.

Documentary of the Year: "Ai Wei Wei: Never Sorry," by Alison Klayman.

Personal Highlight of the Year: Pacific Street. With this guy.

Happy New Year, folks. 

Further Thoughts on the Politics of Unconditional Cash Transfers and the Notion of the "Deserving" Poor

This morning I told you about my new Atlantic article about GiveDirectly, a charity that distributes cash to poor families in Kenya, and allows them to spend the money on whatever they want. The recipients are chosen not because they have demonstrated any kind of favored behavior, like enrolling their children in school, but simply because they are among the poorest people in their rural villages, living in homes made of mud or thatch and frequently going without enough food. They are generally spending their new income on worthwhile causes, like food and weatherproof roofs.

Yet conditional cash transfer programs are more politcally palatable across the world than unconditional ones like GiveDirectly, whether they are pursued by philanthropies or governments. Why? Because, as this beautiful Reuters piece demonstrates, much public policy rests on the assumption that there are two kinds of poor people: the deserving poor and the undeserving. Deserving poor people work, even if the wages they earn are less than the costs of child or health care. They endure cumbersome bureaucratic processes to seek child support from the absent fathers of their children, even if those fathers are in jail, drug addicted, or otherwise unable to provide for their kids. They open college savings accounts, even if they need 100 percent of their monthly income just to cover the costs of housing and food. They attend classes on why it's important to get married.

In the United States, government tells a small group of poor people -- typically mothers of young children -- that if they fulfill such requirements, they can receive, for a discrete period of time, a small amount of supplemental monthly income. Childless adults and the longterm non-working, non-disabled poor are almost completely excluded from social welfare efforts in many states. 

No matter how much economic research we cite showing that unconditional cash income and savings improve the lives of poor people and their children, it will remain politically difficult to tell taxpayers that we aren't going to require anything from the recipients of social welfare. That's because most of us assume poor people need to learn how to best help themselves. The radical premise of GiveDirectly is that poor people already know, much better than their governments or a charity director, what they need. 

Which assumption is true? Should governments and non-profits use the promise of cash to attempt to train or educate the poor out of poverty? Or should a basic income be understood as a matter of human dignity?