Season 2, Episode 3: The Forgotten Experiment

Zig Engelmann gets the chance of a lifetime: Put Direct Instruction to the test in one of the largest education experiments ever conducted. The results could have changed the direction of American education. Instead, they’re contested — and ultimately cast aside.

Email us: soldastory@apmreports.org
Call us: ‪(612) 888-7323

---

Previously on Sold a Story .…

Adrienne Allen: The man came in with a cigarette hanging from his mouth.

Zig Engelmann: What if I had something as hard as this?

Kids: Four!

Zig Engelmann: That's it! Yo!

Elaine Bruner: I couldn't believe what I was seeing. And I thought — this is the future.

Walter Cronkite: One educator has called the approach totalitarian. Another — horrible.

Gary Davis: There's a wide range of people against it. It doesn't fit what they want education to be.

By 1967, President Lyndon Johnson was looking for more ways that schools could help children from poor families. His Head Start preschool program was already serving 3- and 4-year-olds. And he wanted to build on that with a huge new federal effort in elementary schools.

But something happened. Something got in the way.

Walter Cronkite: These ruins are in Saigon.

The Vietnam War.

Ernest House: And I said, “Where's all this money we were promised in education?” And he said, “It's on boats going to Vietnam.”

The president's education ambitions had to be scaled way back. And his plan for a big new program in elementary school got turned into something else.

An experiment. A study.

Cathy Watkins: It was like a Hail Mary. It was like a way to save the program.

The goal of the study was to answer that question on a lot of people's minds by the late 1960s — what actually works in education? How could school become a more powerful tool for helping kids get out of poverty?

(Music)

The study was going to try to find out.

Bob St. Pierre: And for the first time, research institutions, universities, others with theoretical and experimental ideas about education of children were asked to put them into practice in the public schools.

Watkins: It didn't just evaluate different curricular or different instructional methods, it evaluated different philosophies of education.

The study became known as Project Follow Through. The idea was to “follow through” on the promise of Head Start.

The study began in 1968. And lasted nearly a decade. It was one of the largest educational experiments in the history of the world. Nearly 60 years later, that's still the case.

And Project Follow Through ended up providing some answers. Some information that could have influenced the direction of American education.

But it didn't.

I'm Emily Hanford and this is Season 2 of Sold a Story, a podcast from APM Reports. We're at Episode 3.

You've probably never heard of Project Follow Through. Lots of people haven't. Including teachers.

Emily Hanford: Do you know about Project Follow Through?
Teacher: No.
Hanford: Do you know about Project Follow Through?
Teacher: I do not.

Hanford: Have you ever heard of Project Follow Through?
Teacher: No.
Teacher: No.

My producers and I spent two years figuring out what happened with Project Follow Through. We tracked down researchers who worked on the study.

Andrew Porter: It is like a blast from the past. You know, I haven't thought about Follow Through in decades.

We went to the National Archives ...

Emily Haavik: (Whispering) I just found some documents from 1970.

... where we dug through dozens of boxes of letters, memos, notes, reports.

Haavik: The missing data that was never released to the public. Maybe? Sure looks like it.

In this episode, I'm going to tell you what we learned about Project Follow Through. And why some people didn't want it to influence the direction of American education.

(Music ends)

Government officials were scrambling to turn President Johnson's plan for a major new education initiative into a study.

As they were searching around for which programs and educational philosophies to test, the head of the U.S. Office of Education went to a conference in New York. And heard Zig Engelmann giving a talk about Direct Instruction.

He went up to Zig after the talk and asked him on the spot if he wanted to be part of Project Follow Through.

Zig Engelmann: I confessed to him that I'd never heard of Follow Through, and he confessed to me that he'd never heard of Direct Instruction.

Zig immediately called his co-director back in Illinois.

Zig Engelmann: And I said, should we do it? And he says, yeah, let's do it.

But they had a problem. They didn't have a Direct Instruction program for elementary school. They would have to create one ...

(Music)

... fast.

Doug Carnine: We were inventing the airplane while we were flying it.

You met Doug Carnine in the last episode. He worked with Zig.

Doug Carnine: It was a madhouse of writing the curriculum, figuring out the video production system to video teachers, send in the videos, send back feedback.

They were trying to keep up that routine from the preschool where they'd teach, figure out the mistakes they made, and then revise the lessons. But now they had schools all over the country.

Marcy Stein: There were Native American communities.

Jerry Silbert: I went to New York. I went to Providence.

Linda Carnine: I went to Flippin, Arkansas, and the Ozarks.

Stein: Spanish speaking sites in East Las Vegas, New Mexico.

Linda Carnine: Brooklyn.

Silbert: Flint, Michigan.

Stein: Uvalde, Texas.

Silbert: Dayton.

Linda Carnine: Smithville, Tennessee.

(Music ends)

Allen: I flew all around the country, for years.

Adrienne Allen is the woman who watched Zig walk into the classroom with a cigarette hanging from his mouth. She ended up working with him on Project Follow Through.

Allen: You know you'd be at a site like in New York and then you'd get a call: “We need somebody to go to Idaho, so get a plane ticket and go there.”

You had to be ready for anything.

Silbert: They would fly me down in a single seater.

Jerry Silbert remembers flying to South Dakota.

Silbert: And we'd land on some grass in the field, and the pilot said, “Oh, I hope there's no cows there.” (Laughs)

Jerry was a graduate student when he started working on Project Follow Through.

Silbert: I was one of the older ones. I was 28.

Susie Andrist: I was 20, so .... (Laughs)

Susie Andrist was still in college.

Andrist: I didn't come in with a lot of credibility, shall we say?

It was a tense dynamic sometimes — these young people coming in, telling veteran teachers to do things differently.

Andrist: I had a couple of teachers who just were really negative.

Allen: I was hated a lot.

This is Adrienne Allen again.

Allen: Because I was a policeman. I went in and made sure that they were following the script.

There were teachers who did not like those Direct Instruction scripts. They wanted to do their own thing.

Allen: Some of them would lock the door, and they would pull down the shade so that I couldn't see in the classroom or go into the classroom.

She had supportive principals who would help her out. But some principals were not supportive. And there were other obstacles. This is Jerry Silbert again.

Silbert: I remember one site — we did a training for a week or two. And then the teachers got transferred. So, school starts and we're missing half the teachers who were trained.

Other programs being tested as part of Project Follow Through were facing similar challenges.

Larry Schweinhart: I remember in one classroom .…

Larry Schweinhart worked on a program called High/Scope.

Schweinhart: The teacher was doing business as usual and she looked up at me when I came in and she said, “Do you want me to do the High/Scope now?”

She wasn't doing the program. It's not clear why. But it's not that surprising that some teachers were reluctant. These were new, untested ways of teaching.

It was up to the program developers to get teachers on board. To figure out how to make the educational ideas they believed in work in the real world.

There were 22 different programs being put to the test as part of Project Follow Through.

Chuck Salzberg: They represented just about every theoretical structure that existed for early childhood education.

Chuck Salzberg says the programs were all over the map. Some were highly structured, like Direct Instruction. Others were a lot looser. Some put a lot of emphasis on getting parents involved. Or motivating kids. Chuck Salzberg worked on one of those.

Salzberg: The teachers are circulating, giving short instructions, doing what we call behavior specific praise: “Hey Sue, you're reading that very well. You're doing better and better.”

The teachers would give out tokens that the kids traded for rewards like a special toy at recess.

And many of the programs were based on the popular idea you heard about in the last episode. The idea that kids learn best through experience and discovery. One of those was the High/Scope program you heard Larry Schweinhart mention a moment ago.

Schweinhart: It really focuses on children being empowered to make their own decisions and to develop their own plans and activities rather than the adult simply telling children what to do.

(Music)

The various Follow Through programs didn't just have different approaches to teaching and learning. They had different goals, different things they were trying to accomplish.

Zig Engelmann's goal with Direct Instruction was to teach kids academic skills — language, reading, math. Other program developers had other goals. This is the creator of the High/Scope program.

David Weikart: We want the child of independence, of originality, of creativity.

Measuring all those things was going to be difficult. How do you know if a program has succeeded in helping a child become independent or creative? And even if you can reliably measure those things, how are you going to compare programs to each other if they have such different goals?

(Music ends)

St. Pierre: There was a huge amount of disagreement about what should be measured in the evaluation.

Bob St. Pierre worked on the government's evaluation of Project Follow Through.

St. Pierre: The issue of trying to get measures that treat everyone fairly. It's always an issue.

We could see the fights in the documents at the National Archives.

The government initially focused its data collection on basic academic skills. That worried the developers of some of the other programs. They thought it would give Direct Instruction an unfair advantage.

Schweinhart: It was easy to prove the effectiveness of Direct Instruction.

This is Larry Schweinhart again.

Schweinhart: But it was difficult to prove the effectiveness in these other areas that we considered more important.

In response to complaints, the government added other measures. There were tests that looked beyond basic skills to things like abstract reasoning and problem solving. And there were measures to assess things like self-confidence and attitudes about school and learning.

Not all the program developers were happy with the measures that were chosen. They were still concerned that what they were trying to accomplish would not or could not be accurately measured.

It was a high-stakes situation for them.

St. Pierre: This is their life's work.

This is Bob St. Pierre again. He worked on the government evaluation.

St. Pierre: You've got these evaluators coming in and they're going to come up with a report that might say something bad about their life's work. Or something good.

The U.S. Office of Education was planning to distribute information about the program models that were most successful. This was an opportunity to have a huge impact on American education.

Schweinhart: Everybody wanted to become the model that would become the dominant model.

Zig Engelmann: Follow Through was a horse race. To compare these models to see what works.

But the idea that Follow Through was a horse race that would have one winner — that idea was going to be contested.

(Music)

And here's the thing. At the beginning of Project Follow Through, many people — including government officials — were assuming there would be lots of winners.

Watkins: The idea was that every model would be so successful it would just blow the doors off of the competition — the competition being business as usual.

This is Cathy Watkins who's written about the history of Project Follow Through.

She says the assumption that all the Follow Through models were going to be better than what was already happening in schools — that assumption meant government officials were especially alarmed by what happened next.

The first year of data arrived on their desks. And it did not look good. In fact, it looked so bad that they decided — we can't let people see this.

Watkins: A decision was made that those data would never be released to the public.

We found that first-year data in the National Archives. It showed that students in the Follow Through models were not doing better than students in traditional classrooms. Some of them were doing worse. That was bad news for the government because the goal was to find better approaches.

But Zig Engelmann wasn't bothered by the early results. He thought it was going to take time to get Direct Instruction going, and he was feeling confident. So were his colleagues.

Davis: The program worked well after several years. And it worked well because the teachers got good.

Linda Carnine: They were saying, “You know, this program teaches kids I've never been able to teach before.”

Silbert: They'd be in the lowest group, and then they'd move up and up and up.

And Zig was telling them — your hard work will pay off.

Silbert: He said, “It's gonna be worth it.” You know, it was hard, but he said, “We'll be able to help lots of kids.”

Andrist: It was pretty exciting to think about how this might grow and how many more kids might get taught.

(Music ends)

The government collected data for nearly 10 years. It had test scores and other information about more than 350,000 students in thousands of classrooms all over the country. It released some of that data to the public and the program developers as the study was going on. But the ultimate verdict on the effectiveness of the different Follow Through models was going to come down to a final evaluation.

(Music)

Coming up, the government releases the evaluation. And there's a winner. But that's not the end of the story.

That's after a break.

**

(Music)

By 1977, the Project Follow Through study was wrapping up after nearly a decade. And the government was getting ready to release its final evaluation.

A Direct Instruction trainer named Marcy Stein was on her way home from a school in Flint, Michigan where she'd been coaching teachers. At the airport, she ran into one of her colleagues.

Stein: And he said, “Oh, I just came from the meeting.”

The meeting.

(Music)

The meeting was for the developers of all the Follow Through models. To give them a preview of the final evaluation.

Stein: I said, “Well, what happened?” And he said, “Well, you know, we did really well.”

He told her the other program developers were shocked.

Stein: He said the place went nuts. They couldn't be … that can't, that's gotta be wrong.

Zig Engelmann: We beat ‘em in everything.

The government's evaluation showed that Direct Instruction had outperformed all the other models.

It had done especially well compared to the other models when it came to math, reading and spelling skills. Exactly what Direct Instruction was trying to do.

But it had also done well when it came to things like abstract reasoning and problem-solving.

And perhaps the biggest surprise? Direct Instruction performed the best on those measures of self-confidence and how kids felt about school and learning. Even Zig was surprised.

Zig Engelmann: The DI model never professed to be concerned with self-image. And yet our kids had the best self-image. Well guess why? They knew damn well they were smart.

(Music ends)

Zig was triumphant.

Owen Engelmann: I remember that everything was electric.

Zig's son Owen was a teenager at the time.

Owen Engelmann: This is awesome. It's going to be disseminated everywhere.

Doug Carnine: The assumption was that once the data got out, then the Direct Instruction model would be used widely all over the country with disadvantaged children.

This is Doug Carnine again, Zig's colleague.

Doug Carnine: I mean, that was the whole purpose of doing it. It wasn't some kind of academic exercise.

But there were two groups of people that did not like what the final Follow Through evaluation said.

(Music)

The first group that didn't like it: the government itself. The U.S. Office of Education.

St. Pierre: It didn't say what any of us were hoping it would say.

Remember — they were hoping all the models were going to be better than what was already happening in schools. But Bob St. Pierre, who was one of the authors of the government evaluation, says that's not what the data showed.

The data showed that Direct Instruction was the only model that had positive results in every category that was measured. In all the other models, the kids did worse — on average — on at least one set of measures.

And that brings us to the other group of people who were unhappy with the results. The program developers who did not do well.

They complained to the Ford Foundation. It had been funding some of the unsuccessful models. And a program officer from the foundation made a phone call. To a guy named Ernest House.

(Music ends)

House: So she called me and says — you know, I'm getting a lot of complaints from a lot of the Follow Through sponsors saying they're being unfairly treated.

The program developers were also known as “sponsors.”

House: I don't know what to make of it, she said. So, what do you think?

Ernest House was one of the country's leading experts in evaluating social and educational programs. He agreed to meet with the sponsors.

House: And I went out to this kind of rural retreat they were at.

It was a summer camp in Michigan, run by the man who created the High/Scope program. It was a rustic spot — cabins, bunk beds, a dining hall.

House: And I asked them what they thought, sat down with them over coffee, I listened to them in their meetings.

Some of the program developers were still upset about the measures the government had used. Felt that what they were trying to accomplish with their models hadn't been accurately measured. And there were some issues with the study design that went back to the fact that Follow Through was originally supposed to be a big government program, not a research project.

House: So, after listening to them, I thought, I think they got a point here. I think they got a point.

Ernest House convened a group of experts to review the government evaluation.

House: We went through the whole thing.

They thought it was a messy study that could have been set up better. The measures weren't perfect. And they concluded some of the government's statistical analysis was flawed.

So, House and his group made some adjustments. And ran the numbers again.

Some of the models did better. And some of the models did worse.

But the winner didn't change. Direct Instruction still came out on top.

However, Ernest House and his panel came to a very different conclusion about what it all meant.

(Music)

They wrote a critique of the government's report. And took issue with how the government had defined success.

This is a little complicated. Bear with me.

The government evaluators had calculated the average performance of the various educational models. They took all the data from all the places where each model was used and figured out how effective each model was on average. It's kind of like rating restaurant chains by averaging customer reviews from every location across the country. Overall, some chains are better than others.

But that doesn't mean every location of a highly rated chain is good. Or that every location of a poorly rated chain is terrible.

Same thing happened in Project Follow Through.

According to the government's evaluation, the models that didn't succeed — on average — had some schools that did relatively well. The teachers, the coaches, the principals — they made it work.

And the models that did succeed — on average — didn't succeed everywhere. Even Direct Instruction had schools where things did not go well, where kids did not do better.

And that's what Ernest House and his group focused on in their critique of the government's evaluation. They focused on the variation.

House: You take one program, put it on the east side of Manhattan. You get one result.

This is Ernest House.

House: You put it in Urbana, another result. You put it in East St. Louis, yet another result. The very same program. That's the lesson from Follow Through. There's all this variation; therefore, you cannot draw the conclusion which is best.

(Music ends)

Project Follow Through had not revealed a program that worked equally well everywhere, every time it was implemented. And for that reason, Ernest House and his group concluded that there was no Project Follow Through winner. There couldn't be a victor.

Andrist: I remember Ziggy's rage.

Zig Engelmann: That was one bitter ass pill.

Zig and his colleagues fought back.

(Music)

They contacted politicians, wrote letters. Eventually, the U.S. commissioner of education responded. He acknowledged in a letter that Direct Instruction was the only program that “consistently produced positive outcomes.” But he wrote that because Follow Through had identified only one successful program, the U.S. Office of Education was “forced” — that's a quote — “forced” to abandon its plan to distribute information about any of the programs.

Zig Engelmann: We have this in a letter that he wrote — we can't call anybody a winner because there was only one winner. I mean, come on, man.

The government's logic was that — as the federal government — it could not promote just one educational program. Local control is a foundational principle of American education. Decisions about what and how to teach are supposed to be made locally, not by the federal government.

But Zig and his colleagues thought there was more to it than that.

Davis: We were the opposite of what the establishment thought education should be.

Gary Davis worked with Zig.

Davis: Like, it should not be structured. You shouldn't be teaching kids. The kid will learn in their own time.

Zig Engelmann: Their logic goes something like this.

This is Zig again.

Zig Engelmann: Direct Instruction is at odds with my model of the mind and learning, which is built around discovery. If I accept data showing that DI works, I will have to acknowledge that discovery and my model of learning is flawed.

(Music out)

There's no question there were influential people who didn't want the nation's schools using Direct Instruction.

One of them was a man named Gene Glass.

Haavik: And Dr. Glass, are you OK with me recording this and using it in the podcast?
Gene Glass: Yeah.

This is our producer Emily Haavik, interviewing Gene Glass.

Haavik: Can you just introduce yourself, say who you are and kind of what you do or did?
Glass: Gene Glass, I've worked on problems in education for, lord, since the early 1960s.

Gene Glass is a towering figure in the field of education research. He pioneered new methods for evaluating programs.

But before that, back in the 1960s, he was a young professor at the University of Illinois ... at the same time that Zig Engelmann was starting his preschool there.

And Gene Glass went to see Zig's preschool.

He told us he watched a lesson for about 10 minutes. And he didn't like it.

(Music)

It seemed to him like the kids were being tormented. That was his word.

Glass: I think performance and achievement and outcomes depend so much on things other than shut up, sit still.

This was his impression of Direct Instruction when — more than a decade after his visit to Zig's preschool — Ernest House got in touch with him. House invited Gene Glass to be one of the experts to review the government evaluation of Project Follow Through.

Gene Glass had nothing good to say about Direct Instruction when we interviewed him.

Glass: I grant them nothing. Nothing. I find no evidence in the entire Follow Through evaluation that they have anything special or worthy of wide distribution. It's a terrible way to organize teaching and learning. The best education starts with the interests, the motivations, the curiosity of the children. Sit down with the child, find out what the child is interested in. Let the child bring his own book, the book he loves. You know, see if he can read, see if he understands.

(Music ends)

After the critique by Gene Glass and Ernest House, the government did two things.

Number one: It started giving more money to the Follow Through models that had not done well. The rationale was that those models needed more support, more time to prove themselves.

And number two: The Office of Education shifted its focus from highlighting successful Follow Through models to highlighting successful Follow Through schools.

By doing this, it was possible for the government to showcase many of the Follow Through models because most of them had a school somewhere that was doing reasonably well.

House: Ziggy was always going around saying — oh, you guys hurt us, you did us in. We won the horse race, so on .…

This is Ernest House again. He acknowledges that the Follow Through critique he wrote with Gene Glass had a huge impact.

House: I do know our critique was very effective in stopping the government from saying Direct Instruction, that's the way to go.

We asked Bob St. Pierre about this — the government evaluator. He agreed the government's response to Project Follow Through should not have been — Direct Instruction is the answer, schools should drop what they're doing and adopt Zig's program.

But he thinks it was a mistake for federal officials to basically ignore the fact that Direct Instruction did better, on average, than all the other models. He says averages matter. They tell you something.

St. Pierre: If you don't believe in looking at average effects, what's your conclusion if that's what you think — that we can never learn anything?

(Music)

The message after Project Follow Through was — anything can work somewhere.

And that gave people permission to pursue what they liked, what they believed in, what they thought was best for kids.

But Project Follow Through had provided some clues about what works in education. The model that did the best — after nearly a decade with lots of kids in lots of schools in one of the largest education studies of all time — was a highly structured program that relied on carefully sequenced lessons and didn't expect students to discover things on their own.

It was an answer. But it was not an answer a lot of people wanted to hear.

(Music ends)

The end of Project Follow Through was devastating for the people involved with Direct Instruction.

Andrist: Probably the heartbreak of my career.

Susie Andrist was that 20-year-old college student. She decided she was going to leave education altogether.

Andrist: I was just sick of the whole scene.

She went to school to study computer programming. She says it seemed like a more rational profession.

Andrist: In education, it was just like people just did what they wanted to do, you know, regardless of the data.

But Zig wasn't giving up.

(Music)

His son Owen says what happened after Project Follow Through made his dad even more determined.

Owen Engelmann: He was like, going, “We're going to show those MFs. We're going to show ‘em. We're going to do it.” He's all in. He's all in.

Zig Engelmann didn't give up. He kept teaching kids and writing programs for the next 40 years.

And his ideas spread.

On the next episode, we take you to an elementary school in Texas that started using Direct Instruction back in the 1970s. Almost all the kids were Black and from low-income families. They eventually did so well, people thought they must be cheating.

Gayle Fallon: We've had them come in and say, we think they're cheating at Wesley. Can you find out how they're doing it?

That's next time, on Episode 4 of Sold a Story.

(Music)

When we were digging through all that material about Project Follow Through in the archives, one thing we could find almost nothing about was what it was like for the students. So we tracked down a bunch of former students and asked them. You can read what they had to say on our website, soldastory.org.

Sold a Story is a podcast from APM Reports. It's reported and produced by me, Emily Hanford, with Emily Haavik, Olivia Chilkoti, Carmela Guaglianone, Kate Martin and Christopher Peak. Our story editor is Curtis Gilbert. We had additional editing from Sasha Aslanian. Andy Kruse is our digital editor. Mixing, music and sound design by Chris Julin. Fact checking by Betsy Towner Levine. Our theme music is by the band Wonderly.

Final mastering of this episode by Maurizio D'Errico.

Our executive editor is Jane Helmke.

Special thanks to Emily Corwin and the APM Reports team. Thanks also to Cait Kelley, Dan Kraker and Margaret Goldberg.

Leadership support for Sold a Story comes from Hollyhock Foundation and Oak Foundation. Support also comes from Ibis Group, Esther A. & Joseph Klingenstein Fund, Kenneth Rainin Foundation, Wendy and Stephen Gaal and the listeners of American Public Media.