'Junk science': experts cast doubt on widely cited college free speech survey

Polling experts are raising red flags about a new survey that concluded that nearly 20% of American college students believe it’s appropriate to use violence to silence offensive speech.

The eye-popping results of the survey have been widely cited in conservative media outlets, including by the editorial board of the Wall Street Journal, and were written up by an opinion columnist for the Washington Post.

The way the survey results have been presented are “malpractice” and “junk science” and “it should never have appeared in the press”, according to Cliff Zukin, a past president of the American Association of Public Opinion Polling, which sets ethical and transparency standards for polling.

John Villasenor, a professor of electrical engineering at the University of California Los Angeles, defended his survey as an important window into what he had called a troubling atmosphere on American campuses in which “freedom of expression is deeply imperiled”. Villasenor, a cybersecurity expert, said this was the first public opinion survey he had conducted.

However, his survey was not administered to a randomly selected group of college students nationwide, what statisticians call a “probability sample”. Instead, it was administered to an opt-in online panel of people who self-identified as current college students.

“If it’s not a probability sample, it’s not a sample of anyone, it’s just 1500 college students who happen to respond,” Zukin said, calling it “junk science”.

“It’s an interesting piece of data,” Michael Traugott, a polling expert at the University of Michigan’s Center for Political Studies, said. “Whether it represents the proportion of all college students who believe this is unknown.”

Villasenor said his survey had been inspired by the “increasing trend towards censorship on college campuses in recent years”, which included both outright censorship and “self-censorship”.

Like many other academics, Villasenor has remained concerned that university campuses are not sufficiently open to “reasonable perspectives”.

He secured funding from the conservative Charles Koch Foundation to survey students this August about their views on free speech. Rather than write an academic paper, he posted some of his results online this week, arguing that given “the timeliness of the topic, I believe it is important to get some of the key results out into the public sphere immediately”.

“A surprisingly large fraction of students believe it is acceptable to act – including resorting to violence – to shut down expression they consider offensive,” he wrote. His survey also found that college students had deep misunderstandings about the scope of the first amendment’s free speech protections.

Some journalists and activists greeted his findings with shock, labeling them “chilling” and a call to arms. “Faculty have failed them,” a Yale professor tweeted of America’s “illiberal” college students.

Villasenor’s results had gone through no peer review process. The methodology section of his online post was vague, prompting several polling experts to question how reliable the survey’s conclusions might be.

Villasenor wrote in an email that he was reluctant to give a yes or no “sound bite” answer to the question of whether the students he surveyed were nationally representative of college students or not.

By some measures, Villasenor wrote, the 1,500 respondents to his survey had seemed to reflect the rough demographic makeup of American college students. By others, they might not.

Villasenor had calculated a margin of error for his survey results and included it in the public writeup of his report, even though the sample of students he had surveyed was not random. Public polling experts said this was inappropriate and a basic error. Zukin called it “very misleading” and “malpractice”.

By including a margin of error, the author appears to be “trying to overstate the quality of his survey”, said Chris Jackson, the vice-president of Ipsos Public Affairs, a public opinion firm.

Timothy Johnson, the current president of the American Association for Public Opinion Research, called it “really not appropriate”.

Villasenor said he disagreed that including a margin of error was any kind of “malpractice”, noting that he had included a caveat in his online post, warning that a margin of error was relevant “to the extent” that his survey respondents were actually representative of US college undergraduates.

Some journalists writing up his results, including at the Washington Post, made no mention of his caveat, Villasenor noted. He said they should have: “The caveat was there.”

His results “do say something useful about the national on-campus climate in relation to free expression,” he said, noting that the survey included respondents from 49 states.

His survey had posed the question about violence and speech to students in late August, in the days immediately after neo-Nazis and white supremacists marched through the university town of Charlottesville, Virginia, leading to the murder of one young woman.

This was “purely coincidental”, he said. “It wasn’t planned that way.”

Jackson, the Ipsos Public Affairs vice-president, said the post-Charlottesville moment would certainly have affected students’ responses to a question about whether it was appropriate for a student group to use violence to prevent the speech of a “very controversial speaker” who is “known for making offensive and hurtful statements”.

“If someone asks you that two days after Charlottesville, who do you think of immediately? You think of neo-Nazis,” Jackson said.

If the survey question had explicitly asked college students about neo-Nazis, and “19% of college students said it’s okay to use violence against neo-Nazis on campus ... I think people would find that more palatable.”

Last year, a different, more nationally representative survey of American college student opinions on free speech on campus found strikingly different results from Villasenor’s survey.

Villasenor’s survey asked students if it was more important for colleges to create an “open learning environment where students are exposed to all types of speech and viewpoints, even if it means allowing speech that is offensive or biased against certain groups of people” or “a positive learning environment for all students by prohibiting certain speech or expression of viewpoints that are offensive or biased against certain groups of people”.

He found that 53% of his respondents said they supported the “positive learning environment” that required “prohibiting certain speech”, a result Villasenor called proof that “students appear to prefer an environment in which their institution is expected to create an environment that shelters them from offensive views”.

The 2016 Gallup survey of more than 3,000 college students, who had been selected in a carefully randomized process from a nationally representative group of colleges, had asked students the same question. It found that 78% of students said colleges should create an “open learning environment”.