Intro
The LSAT often introduces the results of studies, surveys, and experiments as evidence in logical reasoning stimuli. Many times, this type of evidence can be totally valid. There's nothing inherently wrong with using studies, surveys, or experiments as support for a conclusion.
For example, maybe a tutor surveys all of their students about whether they prefer written or video lessons. If that tutor then decides which type of lessons to create based on their student's responses, that's probably a fairly logical way to make that decision.
Problems with Studies, Surveys, and Experiments
While using these tools as evidence is not inherently problematic, on the LSAT they will often be conducted in a flawed manner or their results will be misinterpreted. It's worth understanding the common logical flaws surrounding these topics so you can increase your speed and accuracy on test day!
Sampling Errors
One common type of flaw that pops up around studies and surveys involves sampling errors. A sample is a group taken from within the larger population that the study is trying to learn about. For example, if I wanted to run a survey to learn about the study habits of LSAT takers in 2025, I might randomly pick 1,000 names from LSAC's data about who took the test. Those 1,000 people would be my sample for the study.
For a conclusion from a study to be considered reliable, the study must have a representative sample. Therefore it must have a sample size that is not so small that it can't represent the larger group it makes a conclusion about, and it must have a sample that is randomly chosen to account for biases.
Sample Size Error
Consider the following:
"Nate wants to know what the most popular dog breed in North America is. So Nate asks the next 50 people he sees on the street what type of dog they have."
This is not at all a large enough sample to make a conclusion about North America's favorite dog breed. There are way more than 50 dog breeds that people have as pets, so this survey may not even end up including some of the most popular breeds. Beyond that, what if randomly 10 of the respondents own Bernese Mountain Dogs? First of all, he would be lucky, because those dogs rule. But unfortunately, if Nate treated that sample as representative then he might come to the conclusion that 20% of dog owners have Bernese Mountain Dogs. That number is way off from reality. So Nate would be committing an error related to the sample size of his study.
Other Unrepresentative Samples
Here's another example:
"Your friend Jackie wants to know if more people in North America prefer a beach vacation or a mountain vacation. So she walks over to the ski resort and starts surveying people as they get on the gondola (Jackie's dad is a partner at a law firm and she's currently on vacation at her house in Aspen)."
This is not a good group of people to study! Skiers will likely be more biased towards mountain vacations than the average individual. Even if Jackie surveys thousands of people that day, the sample probably would still not yield reliable results as it would skew towards mountain vacation lovers. Visit the sampling issues tab within the Identify the Flaw lesson for real LSAT examples!