Research Article Evaluation (Final Paper)
Research Methods I
New York University
I. Research Question-
Do stereotypic and counter-stereotypic textbook images affect high school students’ science comprehension and state anxiety?
II. Hypotheses and Null Hypotheses- a) = Hypothesis b) = Null Hypothesis
1. a) Female students exposed to counter-stereotypic images will perform better on the comprehension assessment and report less anxiety than female students exposed to stereotypic images. b) Neither stereotypic nor counter-stereotypic images will affect a female student’s performance on the comprehension test or their level of anxiety.
2. a) Male students who viewed stereotypic images will have higher comprehension and less state anxiety than male students viewing counter-stereotypic images. b) Neither stereotypic nor counter-stereotypic images will affect a male student’s performance on the comprehension test or their level of anxiety.
3. a) Both female and male students exposed to mixed gender images will perform more equivalently on the comprehension assessment and report experiencing comparable levels of anxiety. b) Mixed gender images will not affect a male or female student’s performance on the comprehension assessment or affect their anxiety level.
III. Variables- A) Subject Variables (Participant Characteristic): 1. Gender and had 2 levels: Male or Female. B) Independent Variables and Manipulation: 1. Chemistry Lesson and had 3 levels: a) The counter-stereotypic condition, photos consisted of three lone female scientists. b) The stereotypic condition, images consisted of three lone male scientists. c) The mixed gender condition, photos consisted of one lone female scientist, one lone male scientist, and one image of both a male and a female scientist working together in equally active roles.
C) Dependent Variables and How Measured:
1. A comprehension assessment consisting of 12 multiple choice and fill-in-the-blank questions was created by the researchers to assess participants’ comprehension and retention of the textbook lesson. Internal scale reliability was acceptable, with Cronbach’s alpha = .76.
2. The 20-item Spielberger State Anxiety Inventory was used to assess participants’ level of anxiety following the testing situation. Participants were asked to rate on a 4-point scale (1 = not at all, 4 = very much so) how they were feeling at the moment they were answering the questionnaire. Internal scale consistency was high, with Cronbach’s alpha = .89.
3. An academic and career survey was created by the researchers to investigate the current academic and future career interests of the participants. Participants were asked to rate on a 5-point scale (1 = not at all, 5 = extremely) the degree to which they liked several academic subjects and the likelihood that they would pursue a career related to those academic fields. This survey was not tested for reliability or validity.
4. For a Manipulation check, a multiple choice question was designed to assess participants’ ability to identify the images in the text. Participants were specifically asked to identify the content of the first picture shown and were given four response options (male scientist, female scientist, male and female scientists, chemistry equipment). Questions were also included as an attempt to determine the level of difficulty of the lesson and comprehension test as perceived by the participants as well as the amount of effort participants exerted. This measure was not checked for reliability or validity.
IV. Study Design- The experimenters utilized a 2 x 3 pre-test post-test randomized groups expericorr factorial design; the 2 being Gender with two levels: male or female; the 3 being chemistry lesson with three levels: stereotypical, counter-stereotypical, or mixed gender; pre-test post-test because the academic and career survey was administered before the manipulation of the I.V. chemistry lesson and after the manipulation; also, though determined by the experimenters to be a factorial design with 2 independent variables, it is actually an expericorr design due to its integration of both a true IV being chemistry lesson and a subject variable being gender which cannot be manipulated but represents the characteristics of the participants.
1. Manipulation Check: A 2 (sex) × 3 (image condition) ANOVA was used to determine participant ability to correctly identify image content. Across the three conditions, 84% of the participants correctly recalled the content of the images they viewed. Independent t-tests revealed that participants who recalled the textbook images correctly did not differ significantly on any of the dependent measures from those who did not recall the images correctly. Students were asked to rate how hard they tried on a 7-point scale, with 1 being not at all and 7 being tried my best. A 2 (sex) × 3 (image condition) ANOVA on effort revealed no significant differences between groups (M = 5.35, SD = 1.61).
2. Comprehension Assessment, Means were computed for male and female students and comprehension scores ranged from 0 to 12 (M = 8.24, SD = 2.76). To determine any differences in comprehension scores between conditions, a 2 (sex) × 3 (image condition) between subjects ANCOVA was computed with a significant interaction of condition and sex being found, F (2, 71) = 3.78, MS = 23.98, p = .027. Female students scored significantly higher in the counter-stereotypic condition (M = 9.38) than in the stereotypic condition (M = 7.42). Male students demonstrated higher comprehension in the stereotypic condition (M = 9.00) than in the counter-stereotypic condition (M = 7.70)
3. State Anxiety, A 2 (sex) × 3 (image Comprehension condition) between subjects ANCOVA, covarying initial science interest, failed to reveal significant main effects or a significant interaction (p > .05). independent t - test found no significant differences between the levels of anxiety reported by males and females in the mixed condition (p = .21, d = .52). Results of the three regressions (anxiety on textbook image condition, comprehension score on image condition, and comprehension score on image condition and anxiety) indicate that state anxiety did not act as a mediator.
4. Academic and Career Interests, A 2 (sex) × 2 (pre-post) × 3 (image condition) mixed group ANOVA with sex and image condition as the between subjects variables and academic interest (pre and post-test) as the within subjects variable was computed. A significant main effect of time was found, F(1, 72) = 8.36, MS = 1.17, p = .005, d = .19, with students reporting greater interest prior (M = 3.62, SD = 1.06) to the lesson than afterwards (M = 3.42, SD = 1.09). Although students reported decreased academic interest in science after reading the lesson and taking the subsequent test, a 2 (sex) × 2 (pre-post) × 3 (condition) mixed group ANOVA revealed no significant main effects or interactions for their desire to pursue a career in science ( p > .05). To summarize, female students viewing counter-stereotypic images were found to have significantly higher comprehension of the science lesson than female students viewing stereotypic images. The means were in the opposite direction for male students; although not statistically significant, a large effect size was found. When viewing mixed gender images, no difference was found in the comprehension of male and female students. Female students reported more anxiety than male students in the mixed gender condition, but this difference was not statistically significant. Students reported a decreased interest in science as an academic subject, as well as math and English, following exposure to the lesson and comprehension test, but did not alter their interest in pursuing a career in science.
V. Limitations and Strengths- Strengths to internal validity included avoiding biased assignment of participants because the packets were distributed randomly; avoiding attrition, history affects, and maturation because the study was conducted once and lasted only one period; and avoiding experimenter expectancy effects by allowing the students normal teachers to administer the test and not the researchers themselves. Limitations to internal validity included: pre-test sensitization may have occurred since the academic and career survey was administered twice; individual differences of participants could have varied; environmental factors could have varied since there were two grade levels with most likely two different classrooms; transient states of the students could of varied; differential treatment could have been a factor due to two different teachers administering the test; and measurement error because some measures were not checked for reliability and validity.
VI. Revisions- A) Sampling: 81 students were utilized with 29 males and 54 females primarily (98%) identifying themselves as Caucasian. To strengthen the external validity and generalizability of this study I would have utilized a larger sample of students encompassing an equal amount of 9th and 10th graders with an equal ratio of females to males to prevent a possible biased distribution of gender affecting my results. Also, I would have utilized a more representative sample of students to reflect the ethnic and cultural differences of the 9th and 10th grade populations of : either the city, state, country, or worldwide population. B) Study Design: To strengthen internal validity, I would have utilized the same classroom setting for both grades avoiding environmental differences; utilized the same teacher for both grades avoiding differential treatment; conducted the study around noon so it’s not too early in the day or not too late in the day minimizing transient states; and tested the academic and career survey and the manipulation check for reliability and validity to avoid measurement error.