Intro
The LSAT often introduces the results of studies, surveys, and experiments as evidence in logical reasoning stimuli. Many times, this type of evidence can be totally valid. There's nothing inherently wrong with using studies, surveys, or experiments as support for a conclusion.
For example, maybe a tutor surveys all of their students about whether they prefer written or video lessons. If that tutor then decides which type of lessons to create based on their student's responses, that's probably a fairly logical way to make that decision.
Problems with Studies, Surveys, and Experiments
While using these tools as evidence is not inherently problematic, on the LSAT they will often be conducted in a flawed manner or their results will be misinterpreted. It's worth understanding the common logical flaws surrounding these topics so you can increase your speed and accuracy on test day!
Sampling Errors
One common type of flaw that pops up around studies and surveys involves sampling errors. A sample is a group taken from within the larger population that the study is trying to learn about. For example, if I wanted to run a survey to learn about the study habits of LSAT takers in 2025, I might randomly pick 1,000 names from LSAC's data about who took the test. Those 1,000 people would be my sample for the study.
For a conclusion from a study to be considered reliable, the study must have a representative sample. Therefore it must have a sample size that is large enough to represent the larger group it makes a conclusion about while accounting for random error, and it must have a sample that is randomly chosen to account for biases.
Sample Size Error
Consider the following:
"Nate wants to know what the most popular dog breed in North America is. So Nate grabs an outdoor seat at his favorite coffee shop and records the breed of the next 50 dogs that walk by."
This is not at all a large enough sample to make a conclusion about North America's favorite dog breed. There are way more than 50 dog breeds that people have as pets, so this survey may not even end up including some of the most popular breeds.
Beyond that, imagine if randomly one of the people on the street that morning is Bernie the Bernese Mountain Dog breeder. If Bernie is out walking all 20 of his Bernese Mountain Dogs what would that mean for Nate? First of all, Nate would be lucky in a sense, because those dogs rule. But unfortunately, if Nate treated his 50 dog sample as representative then he would come to the conclusion that 40% of dogs in North America are Bernese Mountain Dogs. That number is way off from reality.
Since Nate's sample size was so small it was susceptible to random error. If Nate's sample size had been in the thousands then Bernie's 20 dogs wouldn't have skewed the results so much. Nate committed an error by having such a small sample size for his study.
Unrepresentative or Nonrandom Samples
Here's another example:
"Your friend Jackie wants to know if more people in North America prefer a beach vacation or a mountain vacation. So she walks over to the ski resort and starts surveying people as they get on the gondola (Jackie's dad is a partner at a law firm and she's currently on vacation at her house in Aspen)."
This is not a good group of people to study! Skiers will likely be more biased towards mountain vacations than the average individual. Even if Jackie surveys thousands of people that day, the sample probably would still not yield reliable results as it would likely skew towards mountain vacation lovers.
So not only do samples have to be large, they can't be obviously biased. This is an error that you should definitely be familiar with in preparation for the LSAT. Keep in mind that many studies on the LSAT will not be problematic at all. But familiarize yourself with these flaws so that you are comfortable evaluating the research and determining whether or not it's valid.
Visit the sampling issues tab within the Identify the Flaw lesson for real LSAT examples!
On the LSAT
The LSAT uses phrases like "a study of X found..." or "researchers surveyed Y people and concluded..." to indicate that a study took place. Treat those as a signal to double check that the study was conducted properly.
Sampling errors may appear in Flaw questions, where your job is to identify what went wrong with the argument's reasoning. When you see a stimulus that introduces a study or survey results you should ask yourself: was the sample large enough, and was it randomly selected?
On Weaken questions, you can attack a study-based argument by pointing out that the sample was too small or unrepresentative. An answer choice that introduces a bias in how the sample was collected could very well undermine the conclusion.
On Strengthen questions, the correct answer may show that a sample in a study was randomly selected, that it was sufficiently large, or that the group surveyed was actually representative of the broader population.