Focus Article: The Immediate Feedback Assessment Technique
“Teaching Perspectives”, St. Thomas University’s Teaching Newsletter recently published an article written by James Whitehead (Director of Teaching and Learning) about the IF-AT. The article follows (you can read the entire newsletter HERE).
“Picture this: Groups of students actively engaged in discussion, thought, critically analysing concepts, chatting and arguing points amicably with one another. One of the groups explodes in an animated frenzy with cheers while another, a little less exuberantly, bubbles into laughter and groans, then huddles back down into discussion. This might sound like some fantasy of how we might hope our small group activities might work, but is an actual description of a group of Indonesian students taking a Statistics course. This scene was shown in a video at last semester’s Effective Teaching Institute at UNB Saint John by Dr Larry Michaelson of the Central Missouri State University. As you’ll know by now, having read Janice Harvey’s and Kathleen McConnell’s articles in the previous few pages, Dr. Michaelson is one of the world gurus and key developers of the Team-Based Learning (TBL) method of instruction.
So, what was happening in that statistics course that had the students so enthusiastic and engaged? Surely chemical assistance would be required to have fun doing statistics! Believe it or not, they weren’t just doing stats, but were actually doing a group statistics test!
The test is termed an `Immediate Feeback-Assessment Technique` (IF-AT) by team-based learning afficianadoes. As Ray Williams (Education) shared with us last year in his presentation on research on assessment techniques, the learning value of an assessment tool is greatly enhanced if the student is provided with feedback, especially immediate feedback. Without immediate feedback on a quiz question, the students progress to the next question generally assuming they have chosen correctly. With the IF-AT they immediately see if their primary choice is incorrect and then get to re-evaluate their second response. Several studies indicate that students generally `commit` to their selection in a quiz. Without immediate corrective feedback, students often learn the wrong information based on what they answer on a test! The IF-AT system corrects this problem, as the immediate corrective action means that students always leave the test with the CORRECT answer in mind.
Both Janice Harvey and Kathleen McConnell mention the Readiness Assurance Tests (or RATs) that form part of the team-based learning technique. The students are assessed on readings that are required to pursue a topic in class. These RATs use the IF-AT technique and provide immediate feedback on whether the students adequately understand the material, and if not, they have an opportunity to immediately clarify any misunderstandings.
The IF-AT system is part of the Team-Based learning technique. In other words the quizzes are not individualised, but each student is part of a team of 5-7 students that work together to discuss the options and the rationale for the choices. This is a great illustration of the social construction of knowledge. However, the IF-AT tests could be set as individualised quizzes and still provide formative feedback, though the benefits of collaboration would be lost.
How the IF-AT Works
The IF-AT quizzes are formatted as multiple choice questions. I can already hear groans, but MC quizzes are not the epitome of evil if they are designed well. To design them well does require some skill, and clearly they are most easily applied to quizzing content knowledge, but they can perhaps be most effectively used to generate considered discussion in small groups when questions are not simply based on the recollection of data, but if instead require and inspire higher order levels of thinking, including attributing, analysing, critiquing. For example, you might provide a case study description, a chart, an image, a poem and then ask a series of questions that mines the material for progressively deeper levels of understanding/processing. See below for an example developed by Dr Brent MacLaine of UPEI, analysing:
The Cool Web By Robert Graves (1895-1985)
Children are dumb to say how hot the day is,
How hot the scent is of the summer rose,
How dreadful the black wastes of evening sky,
How dreadful the tall soldiers drumming by.
But we have speech, to chill the angry day,
And speech, to dull the rose’s cruel scent.
We spell away the overhanging night,
We spell away the soldiers and the fright.
There’s a cool web of language winds us in,
Retreat from too much joy or too much fear:
We grow sea-green at last and coldly die
In brininess and volubility.
But if we let our tongues lose self-possession,
Throwing off language and its watery clasp
Before our death, instead of when death comes,
Facing the wide glare of the children’s day,
Facing the rose, the dark sky and the drums,
We shall go mad and die that way.
1. The poetic form of “The Cool Web” is best characterized as
A. free verse with a concluding rhymed couplet.
B. ballad stanzas with irregular rhymes.
C. blank verse with unusually irregular rhythm.
D. partially rhymed quatrains with a concluding sestet.
2. To emphasize the adult loss of childhood experience, the speaker of the poem
A. relies on frequent breaks in the middle of the line.
B. establishes a tone of caution, nostalgia and forgetfulness.
C. uses imagery of drowning.
D. alludes to the classical myth of endless return.
3. The “web of language” is cool because, according to the poem, language
A. is the means by which heated conflict may be resolved.
B. lessens the likelihood of achieving spiritual vision.
C. makes our register of the world less intense.
D. entangles us in misunderstandings.
If you need some help designing good multiple choice questions, watch the recorded session by Dr David DiBattista on LTD’s Moodle page in the Archived Workshops section. According to information that accompanies the IF-AT forms, students re-tested a week later with the same or similar questions show a 20% increase in the number of correct responses, on average. Students retested with traditional multiple choice quizzes show a slight decrease in the number of correct responses on re-testing. Research indicates that with the IF-AT, more than 60% of the initial mistakes are answered correctly on follow-up tests. Without immediate feedback, students repeat errors 70-85% of the time.
Another significant difference between traditional multiple choice quizzes and the ones being taken by the Indonesian students described above, is that every group gets more than one opportunity to get the question right. For those of you who use MC quizzes, how many times have you heard “I could narrow it down to option a) or c), but then I chose the wrong one”, or “I changed my choice at the last minute to the wrong one”. Shouldn’t a student get some credit for narrowing the correct response to two from four or five, rather than get none? Isn’t there some learning that could happen if the first choice is found to be incorrect, and yet there is still potential to get some marks if the next selection is correct – especially if that second chance is coupled with group discussion? This is the beauty of the IF-AT technique.
The Use of Scratch Cards
So, how do IF-AT tests work? They use scratch cards. Everyone is familiar with scratch card lottery tickets: incrementally reveal the scratchoff numbers by removing the silver paint with a coin to confirm that yes, you have lost your $2 purchase price! In the case of the IF-AT quizzes, the student (or group of students, if used as a group quiz/activity) can scratch off their preferred answer to see if they got it right. If they did, they will reveal a star, and they can claim the full marks (chosen by the instructor, say 4 points). Cheers ensue. If they get it wrong they have a second chance, after a group groan of course. This is the opportunity that most multiple choice quizzes do not afford, an immediate opportunity to re-evaluate the question, look more deeply at other options, to perhaps hear from the quiet student in the group who may have been unheard among more vociferous, but misguided others. Now is their opportunity to pitch in and and be heard. Perhaps at the next question the group will be a little more reluctant to cursorily accept noisy Dave’s primary selection, when quiet Katie may actually have a more considered opinion to share. The group has a vested interest in hearing from everyone prior to making a selection and it is a wonderful mechanism for increased participation in discussion, critical reading and thinking.
Having chosen incorrectly the first time, but correctly the second time, the group gets awarded a reduced amount of marks, say 2 points instead of 4. If they require three attempts they get 1 mark, and if they scratch all four they get none. Even a regular multiple choice quiz will present odds of 1 in 4 of selecting the correct answer without any knowledge or consideration.
Isn’t marking such a quiz onerous? Traditional MC quizzes are simple right or wrong and easy to mark. Larry Michaelson suggests that the groups mark their own quizzes, or pass the test cards to another group and they award the marks for each question based on rules that are posted to the front of the class on an overhead or Powerpoint slide. Points are tallied up and returned to the instructor.
These quizzes can be implemented in a number of ways: after the presentation of some content, to establish if the students understand some key issues, or alternatively; to promote some discussion about a complex issue at the start of a class so that students can frame new content in the light of their deliberations. They could also be used at the start of class to judge an aspect of a reading assignment that was required for the class. If a student has not done the reading, and does not properly participate in the discussion, the group has the right and obligation to inform the instructor. Students can be asked to bow out of the quiz if they have not done the reading, perhaps a couple of times a semester without penalty. Students who freeload are easily identified as everyone is called upon to provide a considered response to aid the decision of the group. Generally, since not being prepared is obvious to group members, their peers, students are less inclined to let their groups down than if they were simply accountable to the professor.”
Fattening up Assessment with the Immediate Feedback Assessment Technique (IF-AT)
Dan Stroud recently wrote an article for the December edition of “The Assessable Roo”, University of Missouri-Kansas City’s monthly newsletter. The article discussed a training seminar held at the university. The article follows (you can read the full newsletter HERE):
‘At a meeting in early November, the University Assessment Committee (UAC) was treated to some educational training and entertainment by Barbara Glesner-Fines, Associate Dean for Faculty and Rubey M. Helen Professor of Law at the UMKC School of Law. She introduced a game platform that can be used to promote assessment in the classroom.
The system, called the Immediate Feedback Assessment Technique (IF-AT) is the brainchild of Epstein Educational Enterprises, Inc. IF-ATs are multiple choice cards that reveal the correct choice by scratching off the answer. The benefits of using this include more active learning and creative collaborative learning.
Glesner-Fines offers a more specific advantage the small scratch-off forms seem to offer. “IF-ATs are helpful to formative assessment because they provide immediate feedback (the IF part of their name) and they help distinguish among students who understand the material a little and those who understand it not at all,” she said. “They are especially helpful in team-based work as they allow dissenting voices to be heard and minimize the zero-sum game entality of choosing an answer. The physical layout (scratching off the answer) evokes lottery tickets and lends an air of playfulness to the multiple choice quiz process.”
More information on this innovative assessment and teaching idea can be found at https://www.if-at.com/home/about/benefits.aspx.’
IF-AT Sightings on the Web
photo used under a creative commons license,
flickr user gerlos
Michelle McConville of Kurtztown University recently presented about the IF-AT at the 2012 PASSHE Psychology Conference - Millersville University
IF-AT was featured in section 4.4. of this online article about the Assessment of a Variety of Behavioral Disabilities
The IF-AT was featured on page 9 of St Thomas University’s newsletter “Teaching Perspectives“
Deborah Page and Ruth Benander from UC’s Raymond Walters College recently presented “Information Gap Tasks: Immediate Feedback Assessment for Foreign Language Learning” at the ACTLF Conference
Jim Van Rhee (MS, PA-C) of the Physician Assistant Program at Northwestern University recently discussed the IF-AT in his presentation about “Problem Based Learning”.
Christine Bruce (MHSA, PA-C), Program Director at Penn State Hershey included the IF-AT in her presentation “Team-Based Learning: A Strategy for Transforming Teaching and Learning in PA Education“
The IF-AT was featured in “Reflections on Helping Students Learn: Thriving in Academe”
The Effect of the IF-AT on Student Retention
Jim Henderson and Joe Kappock of University of Purdue’s Biochemistry Department recently did an assessment to determine the effect of the use of the IF-AT had on student retention of material taught in one of their biochemistry courses.
Henderson and Kappock teach a large enrollment biochemistry course in which students are divided into five different sections. Three sections of students were given a multiple choice test using IF-AT scratch cards, two sections were given a multiple choice test in standard format.
The results? Henderson reported that the IF-AT testing method did help his students retain the information that was introduced in the course. Based on improvement on a second test compared to a first test, Henderson saw more than a two-fold increase in test scores for the students that took the IF-AT format test in comparison to the students that took the standard format multiple choice tests.
Henderson and Kappock plan on sharing the information from their assessment with the rest of the biochemistry department and will encourage their colleagues to implement IF-AT testing in their courses.
Collaborative Research - Immediate Feedback Assessment in Chemistry Courses
The grant for the study was awarded by the National Science Foundation. The National Science Foundation is an independent federal agency that was founded in 1950 by Congress. The foundation funds about 20% of all federally supported research conducted by higher education institutions in the United States. The National Science Foundation funds research in many fields such as mathematics, computer science, and social science.
According to the National Science Foundation, the collaborative project “is gathering evidence on the role timing and type of feedback has on student learning when multiple choice exams are used in large-enrollment general chemistry and organic chemistry lecture courses. This systematic chemical education research project is using both traditional Scantron and Immediate Feedback Assessment Technique forms (also known as “Answer Until Correct” in which answer spaces are covered with a coating that is scratched off like a lottery ticket). The variables being investigated include immediate versus delayed feedback, corrective versus non-corrective feedback, test retaking, deliberate increases in cognitive complexity demands, and student confidence in their answers. The results are providing evidence about student cognitive and metacognitive growth and how different sub-populations vary within the testing regimes. Volunteer student populations from one doctoral institution, three comprehensive institutions, and two community colleges (Anoka-Ramsey Community College and South Suburban College) are participating. The intellectual merit of this project includes valid and reliable General Chemistry 1 and Organic Chemistry 1 multiple-choice tests that encompass a spectrum of thinking skills, including higher order cognitive skills. In addition, the project is documenting how multiple choice exams are used both for real-time student learning as well as assessing a student’s knowledge. The broader impacts of this project are the knowledge that is being created about how the structure of multiple choice questions, coupled with type and timing of feedback, influences student learning for the whole, as well as for different sub-groups. Given the wide-spread use of multiple-choice exams, this project also is establishing best practices for the construction of online practice and homework problems such that they now can now be formulated better to maximize student learning.”
Were there any news articles that you saw this week that really grabbed your attention? Leave a comment with a link. If the article stood out to you, it will likely be interesting to our other readers!
Study: A Student Evaluation of the IF-AT
- 85.8% of the students either “agreed” or “strongly agreed” that they preferred the IF-AT to traditional multiple choice format tests
- 92.9% of the students either “agreed” or “strongly agreed” that they would like to see the IF-AT should be used in other courses at the University of Windsor
The students highly recommended continued use of the IF-AT. They also hoped to see a trial run of the IF-AT throughout the university.
Update! New Research and Articles
We’ve just updated our Research/Articles section of our website! Here are the latest published research and articles regarding the IF-AT!
This article introduces an experiential exercise that enhances students’ ability to identify ethical issues and to respond to them in ways that consider the relationship between organizational factors and ethical action. Students identify a required number of ethical incidents in their workplaces during a specified period. Students submit a written description for each incident, drawing from moral philosophical frameworks and/or other ethical concepts to label the issue as one that either exemplifies a “best practice” or “raises concern.” For “best practice” examples, students consider the implications of the practice on the organization and its stakeholders and whether and how the practice could be improved. For examples that “raise concern,” students explain what the ethically appropriate action would be, indicate whether they would take that action, report any reservations they have about taking that right action, and consider how to behave ethically in a way that would bring about desired outcomes without incurring negative outcomes. Then, a subset of submissions is selected for an in-class discussion. Using examples from students’ own experiences engages them and underscores for them the relevance of business ethics issues. Instructions for facilitating classroom discussion and variations for adapting the exercise are provided.
Immediate Feedback and Learning In Athletic Training Education
Immediate feedback has been shown to improve student learning more efficiently than delayed feedback in lower-level general education courses. No research exists examining the effects of immediate feedback on learning in higher-level athletic training coursework.
The objective of this study was to determine if using the Immediate Feedback Assessment Technique (IF-AT) improves retention of information taught in upper-level athletic training courses better than traditional multiple choice (TMC) exams. The second purpose of this research was to determine students’ perceptions of the IF-AT.
Multiple-choice Question Tests: A Convenient, Flexible and Effective Learning Tool? A Case Study
The research presented in this paper is part of a project investigating assessment practices, funded by the Scottish Funding Council. Using established principles of good assessment and feedback, the use of online formative and summative multiple choice tests (MCT’s) was piloted to support independent and self-directed learning and improve performance in an efficient manner for both students and staff. The paper reviews previous studies that have examined the relevance of MCT’s and presents an evaluation of the students grades and the results of a questionnaire designed to capture their perceptions about the effectiveness of MCT’s. Our findings identify improvements on students’ marks and positive responses from students who found MCT’s to be useful at supporting their learning of basic concepts and building confidence and self-esteem. We also argue that MCT’s work more effectively when used in conjunction with other assessment methods.
Relationship Between Study Island and Student Achievement, The
Many schools in America have issues with raising overall achievement as well as the achievement of subgroup populations on state tests required by No Child Left Behind. This quantitative study determined whether an online program called Study Island significantly effected overall and subgroup achievement on the Missouri Assessment Program (MAP) tests in communication arts and mathematics at the elementary and middle school levels. The results will inform school officials in this district and similar districts on whether Study Island can meet the needs of their teachers and students.
The students in the study began using Study Island in preparation for the 2009 MAP state tests. Therefore, the average scale scores before using Study Island (2008 MAP) and after using Study Island (2009 MAP) formed the basis for the data analysis. The z tests and t tests (95% confidence interval) performed on random samples from the total population and seven subgroup populations provided the results for this study. The subgroup populations for the district in this study included Asian/Pacific Islander, Black, Hispanic, White, Free and Reduced Lunch, Individualized Education Program, and Limited English Proficiency. A significant difference existed in the 2008 and 2009 MAP overall population and each subgroup at the elementary level in communication arts and mathematics. Conversely, at the middle school level, no significant difference existed in the 2008 and 2009 MAP overall population and each subgroup, with the exception of the mathematics subgroups Asian/Pacific Islander, Hispanic, Free and Reduced Lunch, and Individualized Education Program.
This study will not provide evidence that Study Island was the sole factor that effected student achievement. However, when reviewing the amount of time spent on Study Island and the number of questions answered by the schools in this study, evidence exists that the use of Study Island represented a significant change in the practice of teachers as well as opportunities for students when comparing 2007-2008 data to 2008-2009 data.
Team-Based Learning in Political Science Research Methods Courses
This paper describes the use of Team-Based Learning (TBL), a cooperative learning approach, in a course on political science research methods. The author explains how he adapted the pedagogy to his class needs, points to pitfalls and errors to avoid, and evaluates the learning outcomes and attitudinal effects that may be attributable to the TBL approach. He finds that the TBL students did better on a 10-question quiz about research methods than students in a non-TBL comparison class. In addition, he finds some evidence that the TBL students had different attitudes towards group work than students in the comparison class. Due to small numbers of observations, it is not possible to control for potential alternative explanations of the data, though, and hence the conclusions have to be consumed responsibly.
The IF-AT at the BCAPT General Annual Meeting
Recently, the British Columbia Associations of Physics Teachers (BCAPT) held their 2012 general annual meeting at the University of the Fraser Valley in Abbotsford, British Columbia. BCAPT is an organization of British Columbia physics teachers from various high schools, colleges, institutes, and universities. The mission of BCAPT is to improve how physics is taught in British Columbia.
One of the featured presenters at the annual meeting was IF-AT fan Joss Ives. Ives presented on the use of the IF-AT in group quizzes (for more information about how Ives uses group quizzes, be sure to read our article on Group Quizzes). Included in Ives presentation were the benefits he observed upon introducing the IF-AT group quizzes in his physics classes. Some of the benefits that he pointed out include:
- IF-AT group quizzes resulted in the highest level of student engagement he ever observed in any of his courses.
- An overwhelming positive response to the testing format.
- Students find the group quizzes make a large contribution to their learning experience.
- His students learn the correct response to test questions immediately, when they are most receptive.
Joss Ives has been extremely generous is allowing us to share his complete presentation with all of our readers! It’s available in several formats, for your convenience!
· Group Quizzes as an Assessment that Supports Learning – PDF
· Group Quizzes as an Assessment that Supports Learning – Powerpoint
· Group Quizzes as an Assessment that Supports Learning – Slideshare
Be sure to check out Learnification, Joss Ives blog. It’s a valuable resource for those using the IF-AT. You can also follow Ives on twitter
Group Quizzes - Working as a Team
Many instructors find that their students benefit greatly when group quizzes are integrated into their course curriculum. According to the University of Minnesota’s Center for Teaching and Learning, group quizzes are advantageous due to their potential to promote subject comprehension, to improve test-taking skills, and to practice and reinforce cooperative learning.
Professor Joss Ives at the University of Fraser Valley regularly uses IF-AT based group quizzes in his physics classes. Every week, his students write an individual quiz. They then re-write the same quiz within a group. Marks are determined by combining the score of the individual portion (worth 75%) and the group portion (worth 25%). In the instance that the student would obtain a higher score through the individual quiz alone, the individual quiz becomes worth 100% of their final mark.
In a recent class survey, 87% of Ives’ students felt that group quizzes contributed positively to their learning. The benefits that Ives reported his students experience during group quizzes include:
- students are given a chance to participate
- students develop collaboration skills
- students enjoy the quizzes, leading to an increase in overall enjoyment of the course
- students are promoted to utilize higher-level thinking
- students experience a possible increase in retention of information
- students experience an improvement in overall learning
- students display a higher level of engagement
- students display a decrease in test anxiety
What do his students thing about IF-AT group quizzes? Here’s some feedback that was posted on Ives’ blog:
“I heart group quizzes!!! I always learn, and it’s nice to argue out your thinking with the group. It really helps me understand. Plus, scratching the boxes is super fun.”
“I feel finding out the answers immediately after the quiz helps clarify where I went wrong instead of letting me think my answer was right over the weekend. It also works the same for right answers.”
“When I think I know what’s going on I can explain it and realize yes, I really do know what I’m talking about…and sometimes vice-versa.”
“It’s a good time to discuss and it’s the perfect time to learn, ‘cause right after the quiz, the ideas and thoughts stick to your mind. It’s gonna leave a great impression.”
For more information about how Professor Ives uses the IF-AT and group quizzes in his physics classes, be sure to check out his blog! Interested in using the IF-AT for group quizzes in your classroom? Contact us and we’ll help you get started!