Post 3 – Reading the Research
The first two posts in this series introduce its purpose and address where to find the research. As a whole, this series intends to empower educators to take control of their professional development with research. Often times, the prescribed whole-school PD doesn’t meet the needs of all classes and students. I believe teachers should invest in seeking out your own development, specific to your class rosters. Who knows your student’s classroom needs better than you? I believe, to improve the education of our students, teachers need to know the following:
- where to find relevant research,
- how to read the research,
- and how to apply the research.
I’m super excited about this post in the series. I have enlisted the assistance of two educators and researchers I hold in high regard. Mr. Charles Schallhorn (@MtnHousePsych) is a social science teacher at Mountain House High School. He is in the classroom daily and, among other things, strives to create a classroom that best educates his students. He also created a wonderful video series designed to increase understanding of the AP Psychology exam and important topics in psychology. Mr. Tim van der Zee (@Research_Tim) is a Ph. D. student and researcher/writer. He believes a skeptical scientist is a better scientist. Both gentlemen are crucial to what I believe is the most important post in this series. They generously donated their time, answering every question I asked to assist with this post. Thank you so much, Charles and Tim.
This post answers the following questions:
- How do I read a research article?
- What makes an article’s findings reliable and valid?
While all of the posts in the series are important, I believe this to be the most important with a more dense accumulation of usable knowledge. Certainly knowing where to access the research is vital, but I believe understanding how to read a technical journal article to be a bigger stumbling block for those interested in furthering their professional development with research. I know both contributors to be very knowledgeable in this arena and Charles and Tim provide somewhat different methods of reading an article. I don’t believe either is necessarily right or wrong. They both have their merits as their reasons for accessing articles and usage of material are sometimes different.
How do I read a research article?
What makes an article’s findings reliable and valid?
Mr. Schallhorn’s method is quite straightforward and simple. He mainly searches for information related to the classroom; that improve the learning environment and increases retention of material. He begins with the title of the article. For more information on how to find research, please visit the previous post in this series.
“I love it when looking at titles seeing something related but with a different approach to the topic.”
After the title catches his eye, Mr. Schallhorn next scans the first few paragraphs to see if the title accurately reflects the information presented and is applicable to his interests. While skimming the article, he also looks for keywords/phrases and headings to alert him to pertinent information. After reading the article and pulling out the usable material, Mr. Schallhorn weighs how best to present the information in class with his students or apply the knowledge to his teaching practice. This application of research in the classroom will be the focus of post 4 in this series.
Mr. van der Zee’s approach to reading an article is quite different. He places great importance on the methods section of the article.
“Almost all the evidential value of a study is conditional on the quality of the method. The introduction section, the discussed theories, the hypotheses, etc. only become relevant in the context of a strongly designed study.”
Within the methods section, he looks at the comparisons being made by the author(s). Groups and/or stimuli may be compared or a single group may be compared at different points. These comparisons dictate what is actually being studied. Proper research typically relies on multiple comparisons that are relevant to the research question. An important point about comparisons to pay close attention to is the fact that many are not ‘pure’ comparisons. Usually there is more than just one difference being compared.
“For example, if two groups of students are given a different course, but are also being taught by a different teacher it is no longer possible to distinguish the effects of the course from that of the teacher.”
According to Mr. van der Zee, a good question to ask yourself when reading the method section of the article is “If I repeat this study over and over again, would the same things happen?” If the answer is no, you should really question the reliability and validity of the results.
Mr. van der Zee also focuses on the sample size used in the study. While it is difficult to say how large a sample size should be, if you want to estimate a correlation in a population, he believes you usually need to have over 250 participants to get a reliable estimation. (1) With smaller sample sizes, the correlation estimate will differ substantially from study to study. Also, it is important to understand that you cannot justify a sample size because of having found a significant difference or because an earlier study used the same sample size.
A last area of focus for Mr. van der Zee is the amount of undisclosed freedom the authors possess in how they analyze the data found. He poses these questions when assessing this freedom:
How many variables did they or could the have added as covariates?
In how many different ways could they have operationalized the outcome variables?
How many potential outcome variables are there, or could there have been?
The more potential freedom the authors have, the less reliable a study is, because it becomes more probable they found differences due to random variation, selective reporting, and/or a publication system which favors certain results.
Lastly, Mr. van der Zee provided an article by Cathleen O’Grady (@cathleenogrady) highlighting the sticky topic of statistically significant results. Ms. O’Grady does a great job of providing relevant examples and makes a somewhat confusing topic much easier to understand.
In the next post of the series, I will examine the application of results found in research articles. Obviously, if we find the research and read the research but do nothing with it, we’ve wasted our time. This is the last piece of the puzzle in taking control of your professional development with research.
- Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize?. Journal of Research in Personality, 47(5), 609-612.